The Misguided Expectations of Human Overseers in AI in Healthcare
DOI:
https://doi.org/10.15346/hc.v12i1.149Abstract
This commentary proposes an idea based on the outcomes of collaborative workshops and ethnographic inquiry within hospital settings, exploring the dynamic interplay between medical practitioners (clinicians and nurses) and artificial intelligence (AI). The research reveals a poignant finding: the prevailing emphasis on ethical AI places undue strain on physicians, obligating them to engage in continuous 'digital literacy' training. This imposition not only exacerbates the existing burdens of healthcare professionals but also fosters a misguided sense of security, given their non-specialist status in software programming and AI comprehension. The investigation underscores the intricate challenges and ethical quandaries inherent in the human-AI partnership within the domain of healthcare. Furthermore, the notion of physicians as the 'human overseer,' regarded as a requisite component of 'ethical AI' per legislative mandates, is revealed to be somewhat fallacious, shifting a complex ethical dilemma towards individual responsibility, as not all clinicians in this loop possess the capacity to rebut AI outcomes or grasp the complexities of AI algorithms.References
Amoore, L. (2020). Cloud ethics: Algorithms and the attributes of ourselves and others. Duke University Press. Artificial Intelligence Act. (n.d.). Article 14. Retrieved June 13, 2025, from https://artificialintelligenceact.eu/article/14/
Brown, P. (2021). On vulnerability: a critical introduction. Routledge.
Brown, P. R., & Meyer, S. B. (2015). Dependency, trust and choice? Examining agency and ‘forced options’ within secondary-healthcare contexts. Current Sociology, 63(5), 729-745. https://doi.org/10.1177/0011392115590
Callon, M., & Law, J. (1995). Agency and the hybrid collectif. South Atlantic Quarterly, 94(2), 481-507. https://doi.org/10.1215/00382876-94-2-481
De Togni, G., Erikainen, S., Chan, S., & Cunningham-Burley, S. (2021). What makes AI ‘intelligent’and ‘caring’? Exploring affect and relationality across three sites of intelligence and care. Social Science & Medicine, 277, 113874. https://doi.org/10.1016/j.socscimed.2021.113874
Enarsson, T., Enqvist, L., & Naarttijärvi, M. (2021). Approaching the human in the loop – legal perspectives on hybrid human/algorithmic decision-making in three contexts. Information & Communications Technology Law, 31(1), 123–153. https://doi.org/10.1080/13600834.2021.1958860
Enqvist, L. (2023). ‘Human oversight’in the EU artificial intelligence act: what, when and by whom?. Law, Innovation and Technology, 15(2), 508-535. https://doi.org/10.1080/17579961.2023.2245683
Eubanks, V. (2018). Automating inequality: How high-tech tools profile, police, and punish the poor. St. Martin's Press.
Ghassemi, M., Oakden-Rayner, L., & Beam, A. L. (2021). The false hope of current approaches to explainable artificial intelligence in health care. The Lancet Digital Health, 3(11), e745-e750.
Hannah-Moffat, K. (2013). Actuarial sentencing: An “unsettled” proposition. Justice quarterly, 30(2), 270-296. https://doi.org/10.1080/07418825.2012.682603
Hayles, N. K. (2022). Ethics for cognitive assemblages: Who’s in charge here?. In S. Herbrechter (Ed.), Palgrave handbook of critical posthumanism (pp. 1195-1223). Cham: Springer International Publishing.
Monahan, J., & Skeem, J. L. (2016). Risk assessment in criminal sentencing. Annual review of clinical psychology, 12(1), 489-513. https://doi.org/10.1146/annurev-clinpsy-021815-092945
Passchier, R. (2021). Artificiële intelligentie en de rechtsstaat: over verschuivende overheidsmacht, Big Tech en de noodzaak van constitutioneel onderhoud. Boom Publishers.
Peeters, R. (2020). The agency of algorithms: Understanding human-algorithm interaction in administrative decision-making. Information Polity, 25(4), 507-522. https://doi.org/10.3233/IP-200253
Pickersgill, M. (2020). Uncertainty work as ontological negotiation: adjudicating access to therapy in clinical psychology. Sociology of Health & Illness, 42, 84-98. https://doi.org/10.1111/1467-9566.13029
Savolainen, L., & Ruckenstein, M. (2024). Dimensions of autonomy in human–algorithm relations. New Media & Society, 26(6), 3472-3490. https://doi.org/10.1177/14614448221100802
van Voorst, R. (2024). The medical tech facilitator: an emerging position in Dutch public healthcare and their tinkering practices. Medicine Anthropology Theory, 11(2), 1-23. https://doi.org/10.17157/mat.11.2.7794
van Voorst, R. (2025). Redefining intelligence: collaborative tinkering of healthcare professionals and algorithms as hybrid entity in public healthcare decision-making. AI & SOCIETY, 40, 3237-3248. https://doi.org/10.1007/s00146-024-02177-7
Wagner, B. (2019), Liable, but Not in Control? Ensuring Meaningful Human Agency in Automated Decision-Making Systems. Policy & Internet, 11(1), 104-122. https://doi.org/10.1002/poi3.198
Zanzotto, F. M. (2019). Human-in-the-loop artificial intelligence. Journal of Artificial Intelligence Research, 64, 243- https://doi.org/10.1613/jair.1.11345
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Roanne van Voorst
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors who publish with this journal agree to the following terms:- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).
