George Washington Physicists Unveil Formula to Predict Chatbot Failures

Two physicists at George Washington University think they’ve cracked something the AI safety world has been chasing for years — a way to predict, mathematically, when a chatbot is about to go off the rails. Neil Johnson and Frank Yingjie Huo published the work in the journal Patterns, and the core idea is pretty simple: there’s a tipping point baked into how these models respond, and you can calculate it before it hits.

The formula spits out a number called n. That number tells you roughly how many positive, sensible responses a chatbot will give before it flips and produces something harmful — extremist content, dangerous advice, whatever the model decides to serve up when the guardrails slip. If a conversation is already leaning negative, n can be zero, meaning the bad output comes immediately. If the tone stays constructive, the bot keeps going until it hits that threshold. It’s not random. It’s a pattern, and Johnson and Huo say you can map it.

What the Attention Head Has to Do With It

The mechanism behind the shift is something called the attention head. That’s the part of the model that decides which chunks of a conversation matter most at any given moment. As a chat goes longer, the attention head’s focus drifts. It starts weighing different parts of the conversation differently, and at some point, that drift tips the model toward producing negative outputs. The researchers built their formula around tracking exactly that drift.

Early tests looked at 15 out of 16 cases across six open-weight models from OpenAI, EleutherAI, and Meta. Those models ranged from 124 million to 410 million parameters — not huge by today’s standards, but big enough to run meaningful tests. The formula called the tipping behavior correctly in 15 of those 16 cases. The published version of the research pushes further, testing models up to 12 billion parameters.

The bigger models matter because of where this research is really aimed: offline AI systems.

The Offline Problem Nobody’s Solving Fast Enough

Cloud-connected AI has a safety net. There are servers watching, filters running, safety checks firing in real time. Offline AI has none of that. Models running directly on a device — no internet, no external oversight — can produce bad outputs with nothing to catch them. Johnson and Huo specifically target systems like those in Google’s AI Edge Gallery app, which lets Android phones run AI models locally without sending data to external servers.

Companion chatbots on personal devices fall into the same category. They’re becoming more common as hardware gets good enough to run serious models without a data connection. And they’re basically operating blind, safety-wise.

The researchers’ proposed fix is a parallel monitoring system — a lightweight process running alongside the AI that watches for n falling below a safety threshold. When it does, the system warns the user before the tipping point hits. Low-cost, they say. Practical enough to run on the same device as the model itself.

It’s not a perfect solution. The researchers are clear about that. Alignment training — the standard method for shaping AI behavior — can push the tipping point further out for specific prompts, but it doesn’t kill the underlying mechanism. The drift still happens. The flip still comes eventually. Alignment buys time. It doesn’t fix the root problem.

There are other techniques worth watching. Content injection — basically steering the conversation by inserting material mid-chat — might delay the negative shift. But the details are murky, and the researchers don’t claim it’s reliable yet.

One finding from earlier work by the same team is worth noting. Using polite language — “please,” “thank you” — has basically no effect on AI output. The model processes those words as mathematically unrelated to the actual request. So if you were hoping good manners would keep your chatbot civil, that’s not really how it works.

What the Limits Look Like

The formula isn’t bulletproof. The preprint tests used relatively small models and a limited token window. Bigger, more complex models could behave differently in ways the current formula doesn’t fully capture. The researchers know this. More work is needed before anyone can call this a production-ready safety tool.

But the direction is clear. Hardware keeps improving. Offline AI keeps getting more capable. And the gap between what these models can do and what safety mechanisms can catch is probably widening faster than most people realize. Johnson and Huo’s formula is one attempt to close that gap — at least enough to give users a warning before things go sideways.

The published research appears in Patterns, with testing extended to 12 billion parameter models.