Benchmarks versus everyday use of new models

In November and December, three new models came out in quick succession, Gemini 3, Claude Opus 4.5 and GPT-5.2, and pushed benchmark scores even higher. Gemini 3 stood out in particular, topping the class on 19 different benchmarks.

And yet it’s getting hard to tell what we actually gain in everyday use. It’s a bit like each new iPhone: on paper, the specs look promising. Once you hold it, the real difference is usually pretty thin.

The excellent Andrej Karpathy suspects a form of reward hacking during the training of the new models, which may have been specifically over-optimized for the benchmark tests.
The no less excellent Simon Willison suggests a pragmatic way to measure concrete LLM progress: keep a backlog of your own use cases that AI can’t handle yet. Retest them with each new release. The day one of them passes, you have a concrete sign of progress.

I had one of those moments this week, with identifying trees from photos. Until now, results were pretty random. This time, Gemini 3 correctly identified a common walnut (Juglans regia) from photos taken in a garden. It explained its reasoning in detail, and even told me about the tree’s health.

If you’re into botany, now might be the time to vibe-code a new plant identification app, without training a specialized algorithm.

But careful, not all multimodal LLMs are there yet on this use case: GPT-5.2 still insists it’s an apple tree 🍎

![Gemini 3 identifying a common walnut from photos of the bark, silhouette and twigs](image-1.jpg "Gemini 3’s answer, in French: common walnut (Juglans regia)“)