Hospitals are using a transcription tool powered by an OpenAI model that can recognize missing objects

A few months ago, my doctor demonstrated an AI transcription tool he was using to record and summarize his patient meetings. In my case, the summary was correct, but the researchers cited by ABC News they discovered that’s not always the case with OpenAI’s Whisper, which powers a tool used by many hospitals — sometimes it just fixes things completely.

Whisper is used by a company called Nabla to find a medical tool that estimates it has recorded 7 million medical conversations, according to the report. ABC News. More than 30,000 doctors and 40 health plans use it, the outlet wrote. Nabla reportedly knows that Whisper has hallucinations, and is “fixing the problem.”

A group of researchers from Cornell University, the University of Washington, and others found in a study that Whisper revealed deception in about 1 percent of transcriptions, creating whole sentences with sometimes violent emotions or nonsensical phrases during recorded silence. The researchers, who collected audio samples from TalkBank’s AphasiaBank as part of the study, note silences are more common when someone with a language disorder called aphasia is speaking.

One of the researchers, Allison Koenecke of Cornel University, posted examples like the one below in a post about the study.

The researchers found that negative comments included fictional medical conditions or phrases you’d expect from a YouTube video, such as “Thanks for watching!” (OpenAI reportedly used to record over a million hours of YouTube videos to train GPT-4.)

The research was presented in June at the Association for Computing Machinery FAccT conference in Brazil. It is not clear whether it has been peer-reviewed.

OpenAI spokesperson Taya Christianson sent a statement via email The Verge:

We take this issue seriously and continue to work on improvements, including reducing hallucinations. In Whisper’s use of our API platform, our usage policies prohibit use in certain critical decision-making situations, and our open source usage model card includes recommendations against use in high-risk domains. We thank the researchers for sharing their findings.