Is the model actually getting dumber, or are we just reading tea leaves from single samples?

Automated excerpt

Lately a lot of people I know have been throwing the pelican test at models (have it draw an SVG of a pelican riding a bicycle). If the model's spatial understanding is even slightly off, you get something pretty abstract. But after running it a few dozen times, I think the biggest trap with the pelican test is judging by a single output.

Selected automatically from source text; not independently written or fact-checked. Read the original for full context.