I wanted to run an ablation. A what? It's a term borrowed from neuroscience, where you slice a piece out of a brain and see what changes. I don't want to do that on a real brain, though, I wanted to make a tea garden at low reasoning, then see what changes when I raise the reasoning level to medium, high, and max reasoning. Or turn off reasoning entirely. Change quantization weights. Tweak this or that. These are experiments that you'd think would be easy, but they're not.

Sure, I could do what everyone else does, and just run it once and look proudly at the utterly random output and pretend I can draw a conclusion from it. People do. Or I could do the statistician thing and run each experiment 21 times and do some analysis to plot some dots and fit some curves. But I'm a programmer, so I'm going to do what a programmer does: fix it.

First let's understand what's wrong. Try this. Go into any ChatGPT or Claude or Gemini window and open it in Incognito mode, where it doesn't know anything outside of that window. Ask it to write you a poem about walruses. Open a new Incognito mode tab. Do it again. Odds are very good you will not get the same poem about walruses twice. That's because the sampler is non-deterministic. Aha! But there's a "seed" you can send in, isn't there, that makes everything deterministic? Yes. Good, you've heard of it. Setting the seed means you tell the sampler to start with a specific random number, which causes each successive random number to be a predictable, deterministic sequence.

So I tried it, in the harness I've built called Beast. What I saw was exactly the same text coming out. For about half a page. Then it desynchronized. Whaaaa?! About that time, the LLM looked at the files in the current working folder and although the files listed were identical both times, because they ran at different times, one folder was newer than the other. The timestamp was different. Then it clicked. These things are unbelievably sensitive to every letter in the context. Even one small difference, a 3 being a 4, is enough to completely change the outputs.

Set the Seed, They Said

The seed isn't the source of randomness, it's one of thousands. The entire context is the source of randomness.

To get to an output that is the exact same twice has been very challenging. Even though I wrote this harness from scratch, a lot of small changes had to be made. For example, I had to write a FUSE plugin for my agent sandbox so that all file times and dates are set to a date in the future 2 weeks away, so I can run the same experiment several days in a row and get the same outputs. You never know what bash command will run and output some file's timestamp into the context, and all new files must have matching incrementing timestamps run to run! I had to build a screenshot tool that was completely deterministic, and prevent all headless browser screenshot commands from functioning and instead tell the model to use the browser_screenshot tool. This is because browser timings are based on real time passing, not fixed time steps, and a screenshot taken at 5 seconds is really taken at 5.1 or 4.89 or some other slightly different time, and if even ONE PIXEL is different, the image tokens cause the context stream to diverge. I know, I ran into this.

The agent reads everything. Every time I fixed something, I'd run a prompt. Copy the logs. Restart the inference engine to clear the prompt cache. Run it again. Compare the logs and find out where something differed. Fix it. Do it again.

All told, I made about 25 small changes, removing random GUIDs from filenames. Changing tool calls to carefully use globally incrementing numbers instead of random values. Stripping runtime timings from common tool outputs with regex. WebGL printing out memory addresses that change every run. Python randomizing string hashes per process. I even had to rewrite tool call and subagents to only run in series, rather than in parallel, because the inference engine itself changes its outputs when batching processes more than one request at a time, creating a race condition. Yes, I ran into that too.

Well, now I have a reliable harness. My S.O. says I like to find windows in brick walls by smashing my forehead into each brick. I think that's a compliment?

And Finally It Wasn't My Fault

After all that, two runs matched byte for byte right up to the first image the model looked at. And even a few words past it. Then it split. This engine's image handler is slightly non-deterministic. /slaps forehead

Which brings me to the engines. These things are being built right now, in public, and I'm rummaging around in the dark corners of where not everything has been well-tested. But the authors are remarkably responsive. uzu wouldn't take a seed the way every other engine does, so I reported it. It was fixed in less than a day. Open Source is wonderful. Try getting that out of a vendor. The vision tower bug is in a different engine, and it's been reported already--I expect it'll be sorted pretty soon.

The big hosted APIs are simpler: neither OpenAI's Responses API nor Anthropic's Messages API has a seed at all. I don't know if the versions I can talk to over OpenRouter will accept a seed or not.

Conclusions

• Doing anything substantial requires a significant investment in the harness.
• Most harnesses will not work for this out of the box, and there'll be a ton of surprises if you try to force it.
• You might think that a single bit in a context would be a self-healing nothing-burger, but it's a huge butterfly effect.
• I plan to do some ablations soon. Stay tuned and follow me on X.com