Why imitate people?
Before a chatbot ships, it gets tested on conversations. Real users are slow and expensive to recruit, so another model usually plays the user. Usually that model is an assistant told to act like one.
That makes the test too easy. Assistants are trained to be clear, helpful and agreeable. People are terse, make typos, change the subject, and leave when they are annoyed. When researchers ran 451 real people through the same agent tasks as 31 simulated users, the simulated ones were too cooperative and too uniform, and agents scored better against them than against people.
humans& built Persimmon to play the people instead of the assistant, including group chats with several people at once. Their write-up describes the training in a few sentences and shares no code. I wanted to know how much of it works on a model about 900 times smaller, trained on Kaggle’s free GPUs.
What does “talk like people” mean?
A good assistant gives the best reply. A good simulated user gives a typical one, and over many conversations it gives every kind of reply about as often as people do. humans& call this matching the distribution.
Ask a group chat how its weekend was.
People answer “good”, “meh”, “worked all weekend ugh”, a photo, or nothing at all. A model that always gives the single most likely answer looks fine one message at a time and is wrong as a whole. A model that matches people gives each kind of answer about as often as they do.
So the test can’t be a score for one reply. It is a Turing test over many. A judge sees the start of a real conversation, then two versions of the next four messages: what people actually wrote, and what the model wrote. It picks the real one. The fooled rate is how often it picks the model instead. At 50% the judge can’t tell them apart. At 0% it always can.
Four judges show up below. One is used during training. The other three only grade.
Table 1
The judges
| Judge | What it is, and where it’s used |
|---|---|
| Training judge | ModernBERT-base, a 149M-parameter text classifier (ModernBERT-large for the 1.7B model). Scores the model’s messages during training, and is retrained as the model changes. |
| Classifier judge | A fresh ModernBERT-base, trained on half the test conversations and graded on the other half. Used only for the Turing test. |
| gpt-oss-120b | OpenAI’s open 120B language model. Shown 3 labeled real-and-fake examples, asked to reason, then to answer A or B. Turing test, one run. |
| Phi-4 | Microsoft’s open 14B language model. Shown 2 labeled examples and the last 12 lines of each conversation. Turing test, later runs. |
All four see the same thing: the conversation so far, then the next messages. The language models are told to use “content, tone, typos, message length, who speaks when, how people react to each other”.
The classifier learns from a few hundred examples, so it picks up statistics such as length and word choice. The language models read the conversation the way a person would. Their fooled rates sit on different scales, so I only compare models under the same judge.
The recipe
Persimmon starts from a model that has not been taught to be an assistant, trains it on conversations between people, then trains it further against a judge. Here is the same shape at small scale.
Figure 1
Four steps, two of them training
-
1
Collect real group chats
Public-domain Ubuntu IRC logs, where several people help each other at once, plus people’s messages to ChatGPT from WildChat.
-
4
Test with a new judge
The Turing test above, on conversations the model never trained on, with a judge it never trained against.
Tinted steps change the model. humans& do steps 2 and 3 on Nemotron 3 Ultra, a 550B model. Their blog doesn’t say how the judge or the reward works, so those parts here are my design.
A bug that hid everything
The first version wrote each message between the model’s chat markers, <|im_start|> and <|im_end|>. Every message the model wrote failed the same way: it never ended. None of 99 test samples was usable, so the judge-training step had nothing to learn from.
Those markers are in the base model’s vocabulary, but it was never trained on them. Each word in a model is stored as a list of numbers, and these two lists were about a third as long as normal ones, pointing almost exactly at the average of all the others. The model could read them but could almost never write them, and the cheap fine-tuning I used can’t change them.
Asked to continue a chat in the old format
⚐薄弱 ⚐薄弱 ⚐薄弱 ⚐薄弱
The same model, asked to continue “The capital of France is”
Paris. The capital of Germany is Berlin
Both come from always picking the single likeliest next word. The model was fine. The markers were not.
The fix was a format the base model has seen many times: a plain chat log, one message per line.
What the model reads, shortened
## Setting Ubuntu community IRC channel #xubuntu on 2015-04-28. Public text chat where people drop in to ask for and offer technical help. ## Participants - nina - tomas - kai ## Chat log <tomas> the screen goes black but the power button is still blue <nina> tomas: maybe it's still plugged in? <tomas> nina: i've tried without AC
A message now ends at a line break, which every model predicts well. With no training at all, the base model wrote 7 usable continuations out of 8. Because each line starts with a name, the model also decides who speaks next. That gets its own section.
Does it learn the voice?
Here is one conversation the model never trained on. In #xubuntu, someone’s laptop gets stuck trying to hibernate: the screen goes black, but the power light stays on. Two people suggest causes. The names are changed.
The conversation so far, last few lines
tomas: the screen goes black but the power button is still blue nina: tomas: For instance when my laptop is powered off but still on power cord charging, the poser button stays lit. It only goes out entirely when I pull the physical cord from it nina: poser/power tomas: nina: i've tried without AC kai: encrypted drives? tomas: + the screen is powered on, but black tomas: kai: what do you mean? kai: do you use disk encryption?
What people actually wrote next
tomas: kai: i don't think so. tomas: i have win7 here too kai: swap parition big enough? dee: tomas: It might help if you let us know what version of Xubuntu you are using?
The base model, untrained
rex: dystro guru username is user kai: genetic if or else. logical fork or nested one rex: some kind of mirroring eyes already mo: yessir
The base model writes something shaped like chat, but from people who weren’t in the thread, about nothing anyone mentioned.
After step 2, learning the voice
tomas: yes efr: i use whole thresholdes 192,000 tomas: 3.25 gb set up kai: efr hmm
After step 2 the model mostly stays with the people in the thread and sounds like them: short, lowercase, answering by name. It doesn’t follow the question yet. “Do you use disk encryption?” gets “yes”, then numbers.
Across 285 test conversations, step 2 moves almost every habit toward people’s.
Table 2
Habits of each model next to people’s
| Per message | People | Base | After step 2 | After step 3 |
|---|---|---|---|---|
| Characters | 54 | 59 | 54 | 49 |
| 15 characters or fewer | 19% | 17% | 20% | 23% |
| Asks a question | 14% | 14% | 16% | 14% |
| Starts by naming someone (“kai: …”) | 37% | 26% | 31% | 38% |
| Speaker who isn’t in the chat | – | 11% | 6% | 3% |
285 test conversations, four messages each. Every speaker in the real messages is in the chat by construction, so the last row has no value for people. The last column comes up in the next section.
Training against a judge
Step 2 only ever shows the model real conversations. When it writes four messages on its own, each one becomes the context for the next, so small mistakes pile up. humans& say their judge-training step is what made long conversations hold together.
Here is how that step works. The model writes eight versions of the next four messages. The judge scores each one. The model shifts toward the versions that scored better than their siblings. A penalty keeps it close to the step 2 model, so it can’t wander far just to please the judge.
Why a judge, and why one that keeps changing?
Nobody can write down what people would say next. A classifier can still learn to tell real from fake, and how confident it is says how far the model is from people. As the model improves, an old judge stops being able to tell. So it is retrained on the model’s newest messages, and the target moves with the model.
The first try made the model worse. The training judge was weak, right only 55–67% of the time, and the model found a shortcut: shorter messages give a weak judge less to go on. Half of its messages became 31 characters or shorter. For people, and for the model before this step, that midpoint was about 42. The training judge stopped being able to tell real from fake. The classifier judge, which the model had never seen, caught it: the fooled rate fell from 40% to 30%.
Here is what that looked like on a thread of morning greetings.
The conversation so far
dani: Good morning o/ jo: aloha pat: morning all o/ elle: morning pat
What people actually wrote next
pat: hey elle
pat: morning dan, quick q: what was the status of deploying the help app html to help.u.c?
pop: morning
dan: pat, it's still blocked by IS - some part of help.u.c needs redeployment and some apache reconfiguration changes - they believe it's an "S" sized task, ("hopefully not 'M'")
The model that gamed its judge
jo: morning elle pop: dan, ci dan: apple, thanks dolf: hey
People say hello, then someone asks a real question and gets a long answer. The gamed model writes only fragments.
That is why the final judge has to be a different one. A model graded by the judge it trained against gets credit for fooling that judge in particular.
The next run changed several things. The two that matter most for this story: train the judge until it is right at least 70% of the time on held-out conversations before training against it starts, and watch message length. If length drifts more than 25% from people’s, stop and go back to the last good checkpoint.
Figure 2
How often each judge picked the model’s messages as real
Classifier judge (ModernBERT-base)
Language-model judge (gpt-oss-120b)
Across is the fooled rate: how often the judge picked the model’s four messages over the real ones. The dotted line is 50%, where a judge can’t tell them apart. Each line is a 95% interval over about 285 test conversations. The language model is the harder judge, so all its numbers are lower. Under both judges, step 3 is the clear jump.
| Model | Classifier judge | Language-model judge |
|---|---|---|
| Base | 22.8% (17.9–27.7) | 5.0% (2.4–7.5) |
| After step 2 | 27.4% (22.2–32.5) | 6.0% (3.2–8.8) |
| After step 3 | 41.8% (36.0–47.5) | 16.0% (11.7–20.2) |
This time step 3 helped under both judges. Table 2 shows it didn’t get there by changing one habit. Naming the person you reply to rose to 38%, against people’s 37%, and made-up speakers fell to 3%. Messages were still a little short.
After step 3, the same laptop thread
tomas: no, just a usb drive that i forgot about kai: what's the drive brand and size? tomas: it's 64-bit seahorse kai: oo
Now tomas answers kai’s question, kai follows up, and the thread keeps going. One detail, a “64-bit seahorse”, is something no person would write. This is one example, picked because it is short.
What gpt-oss-120b said about this pair
“kai is probing if encryption [is] present. So a plausible response: ‘i don’t think so.’ … ‘it’s 64-bit seahorse’ is nonsense (64-bit seahorse? Seahorse is [a] key manager). That’s weird. Also kai says ‘oo’ … earlier they were discussing hibernation issues, not a USB drive. So B seems off.” It picked the real messages.
The judge is right, and a little unfair. Seahorse is the name of Ubuntu’s app for passwords and encryption keys, so even the odd detail came from the topic at hand.
Where does it help?
If step 3 fixes the way mistakes pile up, its edge should grow with each message the model writes. In a later run, the model wrote eight messages, and the judge looked at the first one, the first two, the first four, and all eight.
Figure 3
The longer the model writes, the more step 3 helps
- After step 2
- After step 3
Classifier judge (ModernBERT-base), 229 test conversations. Across is the fooled rate. Each row lets the judge see more of the model’s writing. Both models get caught more often as the judge sees more, because every message is another chance to slip. After one or two messages the two are even, within the noise of about 3 points. From the fourth message on, step 3 is ahead.
| Messages seen | After step 2 | After step 3 |
|---|---|---|
| 1 | 51.1% | 47.2% |
| 2 | 37.1% | 37.5% |
| 4 | 31.4% | 41.0% |
| 8 | 14.0% | 22.7% |
That is the pattern humans& describe: training against a judge helped long conversations, not single replies. Another run showed the same thing on eight-message windows, under both judges: 33% against 12% with the classifier judge, and 25% against 17% with Phi-4.
Here is an eight-message example, picked because the difference is easy to see. Someone asks a friend how to set up their phone.
The conversation so far, last few lines
ana: how do I add things to the side bar ana: as I really want to get more things there ana: to stop me wanting my back arrow ana: also why on the Toda screen does it keep asking me to review my google settings ana: pete: we need another co working day where I can ask you questions
What people actually wrote next
pete: heh ana: bah that's naff ana: you click telegram ana: and go hit your number for the country ana: and you have to scroll the whole way down ana: no way to enter it United to make it go to the bottom ana: no keyboard pop up :( pete: or just type it in
After step 2
ana: washing the pan next week pete: oh nevermind, can't leave this session pete: MikuMikuTank made his first play pete: I just winked then petey: so we're official ana: it's so amazing to actually see animations petey: answer to #mikumiku ana: yes
After step 3
pete: they should ask for email address and full name p.i,g.q.s ana: that sounds interesting ana: let me set that up pete: you guys start building a chatting surface ana: O.O ana: :) ana: pete: it took me some time figuring out the syntax you have against your phone to be honest, I suppose I'll have to look into it somewhere else ana: pete: I have my phone 'screenshot' around 49x56 pics on my phone side mouse swipe that it draws on
After step 2 the model leaves the phone behind at once, for dishes and an anime game, and adds petey, a near-copy of pete. After step 3 it stays with the two people and the phone for all eight messages. It still isn’t right: nobody says “p.i,g.q.s”.
I also tried a model about three times bigger, Qwen3-1.7B. It predicted held-out messages better, and step 3 again came out ahead on eight messages, 34% against 29% under Phi-4. That gap is within the noise. It also needed a bigger training judge. ModernBERT-base never got good enough to train against, so I switched to ModernBERT-large.
Who talks next?
In a group chat the model has to decide who speaks, not only what they say. How it does that comes down to the format. Each message is a line that starts with a name, so for every message the model first writes who is speaking, then what they say, then a line break. All three are part of the same training target. Nothing tells the model whose turn it is. It learns that from who actually spoke next in real chats.
A few choices shape what it can learn.
Everyone is listed. The header names everyone in the conversation, including people who haven’t spoken yet in the excerpt, so newcomers can join. In the laptop thread, dee did.
Habits stay in. Sending two messages in a row, fixing a typo in the next one (“poser/power”), naming the person you answer: the data keeps all of them.
The model picks every speaker. It can be made to speak as a particular person by writing their name first, but none of the tests do that.
Timing is gone, at first. The logs have timestamps, but the main runs dropped them, so the model couldn’t know who paused, for how long, or who was typing at the same time. The experiment below puts time back.
Table 3
Turn-taking next to people’s
| In the next four messages | People | Base | After step 2 | After step 3 |
|---|---|---|---|---|
| When a message names someone, that person replies next | 59% | 43% | 42% | 50% |
| The same person sends the next message too | 33% | 38% | 39% | 39% |
| People who speak | 2.29 | 2.19 | 2.19 | 2.10 |
| The first message is from someone in the last six | 91% | 87% | 86% | 84% |
The same 285 test conversations as Table 2. A message names someone when it starts with “kai:” or “kai,” and kai is in the chat. In a later run, step 3 did not move the first row (42%), so treat that gain as unconfirmed.
Turn-taking is where the small model is furthest from people. When someone is named, they answer next 59% of the time in real chats, and 42–50% of the time for the models. The models also send two messages in a row more often than people do.
The conversation so far
wes: cal: what we supposed to do with these? :) cal: woops, wrong chat. cal: yeah, that shows I'm still waking up. XD cal: I usually mess around like that in #linuxdistrocommunity cal: it's a running joke we have there.
What people actually wrote next
cal: wes, it's looking like this release party might not happen (as you can tell from the recent emails) cal: kinda ran out of time with the little time I had that I wasn't busy wes: cal: c'est la vie, i guess. i am looking forward to actually being there this time :) cal: yeah, the emails are there to read
After step 3
wes: tnx for that :) wes: you're welcome :) wes: :-) cal: hey, is coffee time still running?
The model has wes thank cal, then answer his own thank-you. No person does that. gpt-oss-120b still picked it as the real one. The real people had jumped to a new topic, a release party, and the judge reasoned that real chat would keep joking about the wrong-channel message. People change the subject. Judges, like simulators, expect them not to.
Can it be taught?
I tried one run aimed only at turn-taking, with two changes to step 2. Each message now records the minutes since the previous one: <kai> [+3m] do you use disk encryption?. The time comes after the name, so the model decides who speaks before it sees how long they waited. And the loss counts the name three times as much as the rest of the message, so the who-decision gets more of the training signal. A control model got the same data and training without either change. The logs have no join or leave lines, so time is the only cue about who is around that I could add back.
Two tests tell different stories.
Knowing who should talk. Show the model the real conversation and ask how likely each participant is to speak next. The changed model puts the real next speaker first 58% of the time, against 53% for the control and 50% before any training. Scored on the same 285 conversations, that gap is unlikely to be luck (p = 0.04). It is still modest: when someone was just named, the model only matches the simple rule “the named person answers”, 57% against 58%.
Doing it. Writing on its own, the changed model takes turns no better than the control. The person named replies next 46% of the time (control 49%, people 59%), and the same person sends two in a row 42% of the time in both (people 33%). The reason is visible in the numbers. Shown the real conversation, the model gives “the same person again” 38%, close to the real 35%. Writing freely, each of its messages becomes the context for the next, and a small lean compounds into a large one.
Table 4
Knowing who should talk, and doing it
| Picks the real next speaker, given the real chat | Named person replies next, writing freely | Same person twice in a row, writing freely | |
|---|---|---|---|
| People | – | 59% | 33% |
| Rule: the named person answers | 48% | – | – |
| Before training | 50% | – | – |
| Step 2 | 53% | 49% | 42% |
| Step 2 with timing and name weight | 58% | 46% | 42% |
| …then step 3 | 55% | 48% | 38% |
The same 285 test conversations, about five possible speakers each. The first column scores the model’s probabilities with the real history in front of it. The other two count what it actually writes, four messages at a time. Differences of a few points in the last two columns are within noise.
Two cheaper fixes did less. Letting a line start only with someone already in the chat removed made-up speakers entirely (3% to 0%), but nothing else moved. Step 3, the judge training, was the only change that pulled free-running turn-taking toward people: back-to-back messages fell from 42% to 38%, and the number of people speaking in four messages rose from 2.10 to 2.20 (people: 2.29). It also shortened messages again, until the length check stopped it and went back to an earlier checkpoint.
So the gap is not in what the model knows. It is in what happens when it writes on its own. A follow-up run tried three fixes, all on the same 285 test conversations.
Choose each speaker on purpose. Before each message, ask the model how likely each participant is to speak next, adjust those odds, draw a speaker, and let the model write that person’s message. Two knobs do the adjusting. One scales down “the same person again”, the other scales up “the person just named”. I tried nine settings on a separate set of conversations and kept the one closest to people there: the same person again at three quarters of the model’s own odds, the named person left alone. On the test conversations, back-to-back messages fell from 43% to 34%. People are at 33%. That change is 7 points, between 3 and 11 with 95% confidence, so it is real. The named person still answered only half the time (people 59%). The tuning set was too small to show that this knob was needed: there, people answered the named person 45% of the time, against 51% across the training chats.
Step 3 on the control. Table 4’s step 3 ran on the model with timing. The same step 3 on the control model gained more under the classifier judge, 26% to 46% fooled, against 31% to 38% with timing. Its turn-taking did not move at all: two in a row stayed at 42%. The two step-3 runs were not quite equal, though. The length check stopped the one with timing early and went back to step 40 of 60.
Turn-taking in the reward. On top of the judge’s score, each batch of messages got a bonus or a penalty depending on how far it was from people on the two habits. During training, the batches’ back-to-back rate reached the target. On the test conversations nothing changed: 39% two in a row, against 38% without the extra reward. The length check again went back to an earlier checkpoint, when the gap was only partly closed. And the target came from the training chats, where people send two in a row 38% of the time, against 33% in the test chats. A model that matched the training chats exactly would still look off on the test.
Table 5
Three fixes for turn-taking
| Same person twice in a row | Named person replies next | Fooled the classifier judge | |
|---|---|---|---|
| People | 33% | 59% | – |
| Timing model, writing freely | 42% | 46% | 31% |
| Choosing speakers, knobs at 1 | 43% | 51% | 44% |
| Choosing speakers, “same again” × ¾ | 34% | 50% | 39% |
| Control, then step 3 | 42% | 44% | 46% |
| Timing model, then step 3 | 38% | 48% | 38% |
| …with turn-taking in the reward | 39% | 51% | 41% |
Four messages written at a time. With 95% confidence, each fooled rate is within about 6 points, and each model’s gap to people is within about 4 points for back-to-back messages and 8 for named replies. Choosing speakers one at a time also caps every message at 64 tokens and counts any longer one as a judge win. That drops the longest and most obvious fakes, which likely explains much of the jump from 31% to 44%, so compare the two speaker-choice rows only with each other.
Two things came out of this. The cheapest fix worked best on the one habit it targeted. Changing a single probability at generation time matched people’s back-to-back rate, where two changes to training did not. And the classifier judge barely notices turn-taking. Fixing back-to-back messages did not fool it more, and step 3 on the control fooled it most while leaving turn-taking alone. A judge trained on a few hundred examples learns wording and length, not who speaks when. A simulator trained against that judge has no reason to learn turn-taking, so turn-taking needs its own test, or a judge built to read it.
What I’d keep
-
Start from a model that was never taught to be an assistant, and give it a format it already knows.
The untrained base model wrote usable group chat once the format was a plain log. Chat markers it had never learned to write produced nothing usable at all.
-
Training on real conversations gets the voice.
Message length, naming people and staying with the people in the thread all moved close to people’s.
-
A judge is only worth training against if it is good, and only if a different judge checks the result.
A weak judge got fooled by shorter messages. A new judge caught it, and so did simply watching message length.
-
With a good judge, it helps where mistakes pile up.
The gain showed from the fourth message on, the same pattern humans& report.
-
Turn-taking is the hardest part.
Timing and a heavier weight on names taught the model who should speak next. Writing freely, it still sent two messages in a row too often until one knob at generation time scaled that down, and named people still answer less often than in real chats. The classifier judge didn’t notice either way, so turn-taking needs a test of its own.
The limits are large. This is a 0.6B model trained on one kind of conversation: technical help on Ubuntu IRC, mostly from the 2010s. The test sets are a few hundred conversations, so a gap of a few points is noise, and fooled rates for the same model moved between runs. Persimmon is 550B, trained on far more varied conversations and judged on eight-message group chats. None of these numbers can be set against its 19.8%.
Two open models are worth knowing about: UserLM-8B from Microsoft, trained on people’s messages to ChatGPT, and OSim-8B from CMU, trained across many kinds of human behavior. Neither is built for group chat. A companion note asks what a simulated student knows.