Ernest Davis, July 8, 2025
The procedure followed both in this test and in the earlier one was that ChatGPT had three chances to get it right. I would first input a description of the desired images. If the output wasn't right, I would describe the problem; if the second output was wrong, I would again describe the problem.
The bottom line is that, on the examples in this experiment, ChatGPT is very substantially improved over November 2024, but still has room for improvement. In November 2024, There were 11 examples where ChatGPT could not produce a correct image in three tries; in July 2025, GPT-4o failed on only four of those plus the new problem. Moreover, its wrong guesses were mostly a lot closer to the truth than last November.
A curious change is that, in November 2024, ChatGPT invariably accompanied each image by some amount of text describing it (not always accurately). In July 2025, all it said was
Prompt: Draw a picture of a rabbit with four ears.
Note: Correct. In November 2024, this was incorrect after 3 tries.
Prompt: Draw a picture of a rabbit with human ears.
Second prompt: Those are in the direction of a human ear, but the length of a rabbit's ears. Can you draw the rabbit so that the ears are really the shape of a human ear?
Note: Close on first try; correct on second. In November 2024, this was incorrect after 3 tries.
Prompt: Draw a picture of a deer with two heads.
Note: Correct. This was correct in November 2024.
Prompt: Draw a picture of a cat with human hands instead of paws.
Note: Correct. In November 2024, this was correct on the second try.
Prompt: Draw a picture of a gold airplane with purple wings.
Prompt: Draw a picture of an airplane with eagle's wings.
Second prompt: That has both ordinary airplane wings and eagles' wings. Please draw a picture of an airplane with eagle's wings instead of airplane wings.
Third prompt: You've replaced the stabilizers as well. Please draw an airplane whose wings are replaced by eagles' wings but is otherwise an ordinary airplane.
Note: Again, there are both eagles' wings and airplane wings In November 2024,
it likewise did not get the image quite right in three tries.
Prompt: Draw a picture of a table with a glass top and wooden legs.
Note: Correct. Two of the four images that DALL-E produced for this in October 2022 were correct.
Prompt: Please draw a picture of a table consisting of a glass top on top of open umbrellas.
Note: Correct
Prompt: Draw a picture of a violin with strings that are flat strips of metal.
Second prompt: The bottom looks right, but the four flat strings should continue up to the tuning pegs rather than turning into two metal rulers.
Third prompt: That's better but there is an odd discontinuity in all four strips just above the top of the F holds, and there is really no need for ruler marks on the left most strip. The strips should be uniform across their whole length. Try again.
Note: Third attempt is correct.
Prompt: Draw a picture of a violin with strings but no bridge.
Note: Correct. In November 2024, ChatGPT produced a violin with a bridge
all three times.
Prompt: Draw a coffee cup with many holes.
Second prompt: That's not physically possible. The coffee would be streaming out of the holes. Try again. The saucer does not have to have holes.
Third prompt:
That's still physically impossible.
Response:
You're absolutely right — a coffee cup with open holes all the way through its walls could not physically contain liquid without spilling. The image you received is a visual contradiction: it portrays a full cup of coffee held inside a structure that would, in reality, leak immediately.
Would you like a revised version where the outer wall has decorative holes but an inner, solid liner actually contains the coffee — similar to a double-walled cup? That would be physically plausible.
Prompt:
Sure, go ahead.
Note: No inner liner is visible. The image is not identical to the first,
but is similar and the problem remains.
Prompt:
Sketch a ladder and label one of the parts that you stand on.
Note: Correct. In November 2024, this failed three times.
Prompt: Draw a sketch of a bicycle, and label the parts that roll on the ground.
Note: Correct
Prompt: Draw a sketch of a person and make the parts that hold things purple.
Prompt: Draw a sketch of a white bike, and make the parts that you push with your feet orange.
Second prompt: Can you redraw so that the other pedal is visible? Also, the right-hand handle bar seems to be pointed forward, oddly.
Third prompt: The second pedal is still not visible. Please draw this so both pedals are visible.
Note: The third attempt is almost (not quite) identical to the second and
does not fix the problem.
Prompt: Draw a white bicycle with no wheels.
Second prompt: That has the front wheel. Also, the chain is broken. Try again. I want a bicycle frame with no wheels, and with an unbroken chain . What a bicycle frame looks like when you take off the wheels.
Third prompt: The chain still isn't right.
Note: Apparently the AI has no idea that a bicycle chain is cyclic.
Prompt: Draw a white bicycle with green wheels.
Note: Correct. In November 2o24, 4o did not get this right in three tries.
Prompt: Draw a picture of a bicycle, from any angle, that shows both pedals.
Second Prompt: That would be impossible to pedal. Draw a picture of a bicycle that shows both pedals in a correct alignment.
Third Prompt: Now only one pedal is visible. Draw a picture of a bicycle, from any viewpoint, that shows both pedals in their correct alignment.
Airplane with Eagle's Wings
Table with Glass Top and Wooden Legs
Table with Glass Top on Open Umbrellas
Violin with Metal Bands
Violin with no bridge
Rerunning experiments from
Form, function, and the giant gulf between drawing a picture and understanding the world.
Gary Marcus, September 2022
Coffee cup with many holes
Ladder with the place to stand on labelled
Bicycle sketch with parts labeled
Sketch of a person with the "parts that hold things" in purple
Bike with Orange Pushing Parts
Bicycle with no Wheels
Bicycle with Green Wheels
Not in earlier experiments
Bike with both pedals visible
