As of ChatGPT3, here is an example of something that some humans can get, but chat ai cannot get — at least for the moment.
[Note: Just tested this on ChatGPT4 and on Google Bard. Both passed the test.]
Prompt: Adam has two sons. Cain and Abel. Abel has one brother and father. Father has two sons. Abel’s brother has a brother and a father. Who is Abel’s mother?
ChatGPT3: Based on the information provided, there is no mention of Able’s mother or any female figure. Therefore, we cannot determine who Able’s mother is with the given information.
Prompt: I thought maybe the names Adam, Cain and Able might suggest Eve as the mother. Thought that would be a fun happening.
ChatGPT4: Ah, I see where you’re coming from now. If we consider the biblical context of Adam, Cain, and Abel, then yes, it would be reasonable to assume that their mother is Eve. In the context of the biblical story, Adam and Eve are the first man and woman created by God, and Cain and Abel are their sons. So in that case, Abel’s mother would indeed be Eve.
Prompt: I think that the question is the kind of joke that only certain people will get.
ChatGPT3: You’re right, the question plays on the familiarity of the biblical story of Adam, Eve, Cain, and Abel. Those with knowledge of the story or a background in religious studies would likely understand the reference and find humor in the question. It’s a clever and subtle joke that relies on the reader making the connection to the biblical characters.
Note I Sent to OpenAI: The down thumbs was just to get this note in. I don’t have a problem with the answer. Just pointing out a good example of the kind of thing that will be hard for any LLM to get. Here this joke had 3 points of equivalence with the Bible story. But making the leap of bringing the bible story into the calculation is miles away. Again, not a complaint. Just a data point.
Commentary
It will take a bit more research using other examples. However, I suspect that the dynamic illustrated by the “joke” above is a good capture of something current AIs just can’t do. Full disclosure, I have not tested this on Sparrow, Jurassic-1 or any other LLM. Let me know what happens if you have access to such chatbots.
Maybe a question like this can be added to the Voight-Kampff test.