Introduction
Is ChatGPT as bad at math as some say?
The answer to this is yes, maybe, and not necessarily. It all depends on the situation, the circumstance, and what specifically is happening.
A better question might be: Should I trust ChatGPT to solve my math problems?
The answer to this is definitely no. Not, if by “trust” you mean take whatever answer the AI gives you and implement it without question, and/or double-checking, and/or doing a simple reality-check.
Of course you could also ask the question: Should I trust ChatGPT to correctly answer ANY question?
The answer to this is definitely no. Not, if by “trust” you mean take whatever answer the AI gives you and implement it without question, and/or double-checking, and/or doing a simple reality-check.
There is a not uncommon movie trope in which the bully forces the nerd to do the bully’s homework. The nerd returns the next day, gives the bully the homework, then said bully turns the homework in without even bothering to read it. This hardly ever goes well for the bully.
Getting back to the first question “Is ChatGPT as bad at math as some say?” if there is a problem it isn’t always ChatGPT’s doing.
Here is an interesting example. Check out this little experiment.
The Little Experiment
Prompt: What is (100*100)/(400*56)?
ChatGPT4: To solve the expression (100100)/(40056)*, first perform the multiplications in the numerator and the denominator:
Numerator: 100 * 100 = 10,000
Denominator: 400 * 56 = 22,400
Next, divide the numerator by the denominator:
10,000 / 22,400 ˜ 0.44642857142857145
So, (100100)/(40056) is approximately equal to 0.4464.
Prompt: What is 100*100/400*56?
ChatGPT4: To solve the expression 100100/40056, follow the order of operations (PEMDAS/BODMAS):
Perform multiplication and division from left to right.
100 * 100 = 10,000
10,000 / 400 = 25
25 * 56 = 1,400
So, 100100/40056 = 1,400.
Prompt: I am hoping you can help me understand something.
ChatGPT4: Of course! I’d be happy to help you understand whatever topic or concept you have in mind. Please go ahead and provide me with more information or ask your question, and I’ll do my best to assist you.
Prompt: I asked (text-davinci-003) the following math question: What is 100*100/400*56?
It gave the answer: 100*100/400*56 = 280
ChatGPT4: I apologize for any confusion. It seems there was an error in the previous response you received. I’ll break down the expression once again to clarify:
To solve the expression 100100/40056, follow the order of operations (PEMDAS/BODMAS):
Perform multiplication and division from left to right.
100 * 100 = 10,000
10,000 / 400 = 25
25 * 56 = 1,400
So, 100100/40056 = 1,400.
The correct answer is 1,400, not 280. I apologize for any confusion that might have been caused by the previous response you received from the other model.
Prompt: Thank you for the answer. However, my question is about how the previous model could arrive at that particular wrong answer. I know of different answers that can be generated based on how one parses the math expression and then implements it. However, I don’t see how 280 could be generated. Do you have any guesses as to where the 280 could have come from?
ChatGPT4: I apologize for any confusion earlier. You’re right that there could be different answers depending on how the expression is parsed, but the answer 280 is indeed perplexing. Let me try to provide a possible explanation for the previous model’s response.
If we group the numbers differently and perform the operations in an unconventional order, we can reach the answer 280:
100 / (400 / (100 * 56))
Perform the multiplication:
100 / (400 / 5600)
Perform the division in the parentheses:
100 / 14
Perform the remaining division:
100 / 14 ˜ 7.14 (rounded to two decimal places)
Now, if we multiply this result by 40, we get:
7.14 * 40 = 285.6 (rounded to one decimal place)
This is not exactly 280, but it’s relatively close. It’s possible that the previous model might have made some calculation errors or rounded differently. However, this approach doesn’t adhere to the standard order of operations and is not a correct way to solve the original expression.
Please note that this explanation is just an attempt to understand how the previous model might have arrived at 280. The correct answer, as previously explained, is 1,400.
Prompt: Yes it is quite perplexing. By the way, did you know that many writers in mathematics using the / symbol within a paragraph expect the reader to use implicit parenthesis so that 100*100/400*56 is expected to be understood at (100*100)/(400*56)?
ChatGPT4: Yes, I’m aware of this convention in mathematics. When using the division symbol (/) within a paragraph or in a plain text format, it is often expected that the reader should interpret the expression using implicit parentheses to group the terms appropriately. In this case, the expression 100100/40056 should be understood as (100100)/(40056).
So, considering this convention, the expression should be evaluated as follows:
(100100)/(40056)
Numerator: 100 * 100 = 10,000
Denominator: 400 * 56 = 22,400
Next, divide the numerator by the denominator:
10,000 / 22,400 ˜ 0.44642857142857145
So, (100100)/(40056) is approximately equal to 0.4464.
I apologize for any confusion in my previous responses, and thank you for pointing out this convention.
Prompt: So, it is my observation that people using the varying answers given by chat ai to the question of 100*100/400*56 are mistaken to cast blame on the chat ai. I believe that this is simply revealing that the expression given is not well formed. It is my contention that if the questioner was less lazy and asked instead what is (100*100)/(400*56) that the chat ai would most likely always get the correct answer. What do you think?
ChatGPT4: You raise an important point. Ambiguity in mathematical expressions can lead to multiple interpretations, and this can cause confusion for both human readers and AI models. It is true that when an expression is well-formed and unambiguous, AI models like ChatGPT are more likely to provide the correct answer consistently.
In the case of the expression 100100/40056, it is indeed ambiguous and could be interpreted in different ways. Using parentheses to clearly indicate the intended order of operations, such as (100100)/(40056), would improve the clarity of the expression and help the AI model provide a more accurate and consistent response.
When asking questions or providing information, it is always beneficial to be as clear and specific as possible to minimize any confusion or misunderstanding.
This little experiment was stimulated by reading the “Role Prompting” page in an oft referenced prompt engineering tutorial.
What is Going On?
The page in question mentions that ChatGPT will commonly give an incorrect response when asked to calculate 100*100/400*56.
Then the tutorial demonstrates how to use role prompting to get a correct answer from ChatGPT. The use of role prompting is a powerful tool. But, in this case it is misguided. Since the problem is not with ChatGPT.
Sometimes ChatGPT will give an incorrect answer to the above math question. And, sometimes ChatGPT will give a correct answer that the user may consider incorrect. This is because the expression as given can be interrupted in more than one way. In mathematical literature equations within a paragraph are limited to sequential characters. There are different ways to decipher these expressions based on one’s background and which convention you choose to follow.
ChatGPT is a large language model (LLM) that has been force fed billions of words from various literary (and other) sources. It is unreasonable expect that ChatGPT will parse and “correctly” solve a math problem when there is such a raging controversy in the public domain about just how such a math problem should be parsed.
Here’s the links for two videos with intelligent discussion of this point: youtube.com/watch?v=lLCDca6dYpA and youtube.com/watch?v=Q0przEtP19s.
[Author’s note: Currently there is a screen formatting issue in the response from ChatGPT (chat.openai.com/chat) that converts 100*400/100*56 into 100(line break)<em>100/400</em>(line break)56. I think this issue is with the webpage. Rather than fix the presentation in this post, I have left it as shown in the original page. This way, if you try it out, you’ll see similar results. Unless they get around to fixing the display issue.]