ChatGPT Will *Always* Make Stuff Up—What Can We Do About It?

Demystifying generative AI #4

Midjourney

The hallucination problem (also called confabulation, my preferred term, fabrication, or simply making up stuff) refers to the tendency of language models (LMs) to generate text that deviates from what’s objectively true (e.g., ChatGPT saying 2 + 2 = 5 or GPT-3 implying Steve Jobs was still alive in 2020).

Although confabulation is pervasive—and a no-go when factuality is required—it doesn't matter in some cases. ChatGPT is great for tasks where truthfulness isn't relevant (e.g., idea generation) or mistakes can be assessed and corrected (e.g., reformatting). When boundless creativity is central (e.g., world-building) confabulation is even welcome.

High-stakes categories like finance and healthcare, however, require precision and reliability. Thankfully, the latest research suggests we're on the right track: GPT-4 was a jump from GPT-3, which already improved over GPT-2… the trend is clear; AI gets more factual over time. Extending this assumption we get that LMs will eventually reach a point where they confabulate less than humans—and not much later they won't confabulate at all.

But maybe not. So far, success has been partial. Instruction tuning, RLHF, Constitutional AI (RLAIF), prompt techniques like Chain-of-Thought and Tree of Thoughts, etc. alleviate but don’t eliminate confabulation. It's neither only a matter of data, compute, or size. Confabulation is the symptom of a design choice only fully modifiable with a design overhaul.