The Jeopardy Phenomenon

There’s the thing where if you’re reading an article in the newspaper, and it’s about stuff you don’t know a ton about, it all seems well and good. Then you read another article in the same paper and it’s about something you know intimately (your job, your neighborhood, your hobby, etc) there is a good chance you’ll be like hey! that’s not quite right!

I think of that as the Jeopardy Phenomenon. On the TV game show Jeopardy, if you don’t know the answer to a question, it can feel very much like jeez this quiz show is really hard! But then if a category or question comes up around a topic you know a bit about, the question (or “answer” in reverse Jeopardy parlance) can feel very basic and simple. Like if the “answer” is about popular fantasy card games, the “question” is not going to be Android Netrunner, it’s going to be Magic: The Gathering. (and you’ll roll your eyes a little bit, because it’s like duh)

I think AI has the Jeopardy Phenomenon too.

If you use it to generate code that is outside your expertise, you are likely to think it’s all well and good, especially if it seems to work at first pop. But if you’re intimately familiar with the technology or the code around the code it’s generating, there is a good chance you’ll be like hey! that’s not quite right!