3 Prompt Engineering Lessons You Must Know
Knowledge, practice, and adaptation
I continue today my series of essays on prompt engineering with a slightly more practical one.
I first described prompt engineering from a historical perspective and argued that it's one of the most important skills of our time regarding consumer tech. I then explored a three-way framework to approach prompt engineering as a new thing, rather than taking at face value friendly but incomplete analogies (e.g. equating prompt engineering with code or natural languages).
In this article, I'll cover three fundamental lessons I've tested (or gathered from others) that you can apply quasi-directly in your prompt engineering practice. I want to lean on the usefulness side a bit more here so each section ends with a hands-on takeaway.
For some of you—those most versed on prompt engineering—this article will be borderline self-evident but it might be illuminating for many. Without further ado, let's get to the first lesson.
Sampling can prove the presence of knowledge but not its absence
The resource that probably most influenced my early views on language models and how we should interact with them was this (very long) essay by Gwern on GPT-3 creative fiction. If you’re into modern AI theory explained with practical examples (and have a couple of hours to spare), I thoroughly recommend you to read it—most of it applies well to ChatGPT or Bing chat.
Part of that essay is a review of criticisms against GPT-3 by experts—including Drs. Gary Marcus and Emily M. Bender, with whom I agree a high percentage of the time—and an exhaustive list of counterpoints.
One of the main arguments Gwern makes to defend GPT-3's latent potential is that prompt engineering (what he calls “prompts as programming”) isn't a good tool to criticize AI systems: it’s an intrinsically ill-equipped procedure to unveil language models’ upper limits, because “sampling can prove the presence of knowledge but not the absence.”
“GPT-3 may ‘fail’ if a prompt is poorly-written, does not include enough examples, or bad sampling settings are used. I have demonstrated this many times when someone shows a ‘failure’ of GPT-3—the failure was their own. The question is not whether a given prompt works, but whether any prompt works.”
The idea is straightforward: to prove that language models can't do something, you need to exhaust all the possibilities that would yield a positive result, which is, in practice, unfeasible.
It doesn't matter how many insights you gather by interacting with GPT-3 or ChatGPT. You can always be one more (better optimized) interaction away from unveiling a new capability you were unaware of. It follows that you can’t ever be completely sure of what’s possible to make these models do by means of good-enough prompt engineering (this becomes truer, if possible, as new, more powerful models are released).
This doesn't mean language models can do anything if we find the right prompt (they still have hard limits, e.g., ChatGPT won’t self-improve recursively). And it's also not a particularly great defense against the claim that language models fail from easy prompts (some of which people will intuitively use), which might create problematic scenarios.
But it’s a worthy critique of the methods commonly used to reveal the limitations of generative AI—if you look for flaws in AI models but don't consider your own in exploring it, then your equation is hopelessly incomplete to find the upper bounds of AI’s ability.
To arrive at robust conclusions about language models’ limitations we’d need more sophisticated tools than vanilla prompt engineering. I’m not referring to adversarial prompt injection attacks (e.g., jailbreaks) but to interpretability methods (one of the core research venues of OpenAI rival, Anthropic). Those are out of reach for the immense majority of users—that includes everyone except the builders themselves.
Practical takeaway
As a rule of thumb, when using language (or text-to-image) models never assume you've found proof that it's impossible to make them do something you want (unless you have gathered sufficient evidence in favor of your hypothesis using rigorous testing, which most of you won't do but a small fraction might). Instead, keep trying. It may just be a matter of approaching the problem from a different perspective (i.e. a different prompt), which takes us to the second lesson.