If you have been messing with AI at all the past couple of years, you are no doubt familiar with Simon Willison's Pelican Riding a Bicycle benchmark. I think it's a bit saturated at this point and likely has become part of the training sets of frontier models.

What Brought This On?

Recently, there have been a number of fun diversions like asking AI models to produce low-poly interactive scenes from favorite movies such as iconic moments from Ghostbusters or Back to the Future, with surprisingly good results. Just a week or so ago, the Japanese pagoda benchmark made the rounds. The days of simple one-shot tests are over. Large models can answer simple questions. Small models are at a disadvantage due to their size, as they can't possibly memorize everything in their small weights. I wanted to see if there was a difference between frontier models and the open weights models, and in particular the small-to-medium sized ones you can run locally on modest hardware. I also wanted to see what happens inside one specific harness configured exactly the same way regardless of what provider or endpoint is used. For me, this was as much a test of Beast as it was for AI models. The best way to do this was to run the benchmarks myself. So I decided on a prompt and tried it out on a bunch of the cheap or free LLMs. I'm calling it the Japanese Tea Garden Bench, for reasons that will hopefully become obvious once I write this up properly.

What Is This Testing?

The idea is to exercise multiple elements of a model's understanding at once. Web programming, which models should be really good at by now given this is largely their training data diet. User interface design, which in my experience models are frequently NOT especially good at. More nebulous are style, taste, aesthetic, understanding of shape, color, form, and spatial awareness. Put those together into one task and you get a much more interesting signal than "can it reverse a binary tree". This is not the first benchmark like this, but this one is mine.

Example output from the Japanese Tea Garden Bench: an interactive web scene of a Japanese tea garden rendered by an AI model
One model's take on the tea garden prompt.
A second example output from the Japanese Tea Garden Bench showing another AI model's Japanese tea garden scene
Which one do you think this was?

Final Thoughts

I figure this is at least a visually appealing benchmarks worth browsing for kicks. I also figure, I've spent the money so you don't have to... might as well share.

Check it out, and if your favorite model isn't on the site, you can always ask me to run it. Free credits are delightfully accepted.