Supervisor 2.1

Since 2.0 shipped, every flag Supervisor made in Discord came with a 👍 and a 👎 button. Thousands of you pressed them. Those votes are the single most valuable thing we have, because they are not our opinion about whether a decision was right, they are yours, on your own server, about your own members.

We read them. The pattern was uncomfortable and completely fair: Supervisor was too quick to flag. Not missing harmful content, the opposite. It was firing on messages that were obviously fine to a human, and every one of those costs a moderator time and makes the bot feel like a liability.

2.1 is a full retrain of all three models built directly on that feedback.

What changed

The headline number, on our internal evaluation set, comparing the Arbiter model in 2.0 against the Arbiter model in 2.1. Both are the same base architecture, so this is a like for like comparison:

Every one of the sixteen labels improved. The largest gains landed on the ones that were weakest before:

The same numbers as a table:

Violence and Medical/Injury were the two worst labels in 2.0, both under 0.70. They are now the two best.

The false positive problem specifically

F1 rewards catching things, so it is not the number to look at if what you care about is the bot leaving innocent people alone. For that, the metric is how well the model recognises a message as safe.

Across the sixteen labels, that score went from 0.975 to 0.986. That sounds like a small move until you look at it as errors rather than successes: the rate of getting a harmless message wrong dropped from 2.5 percent to 1.4 percent.

That is 44 percent fewer mistakes on safe content.

That is a real improvement and it is the one the feedback was asking for. It is not a claim that false positives are solved. If Supervisor flags something it should not have, press 👎. That is exactly the signal that produced this release, and it is what will produce the next one.

Short messages

The most common complaint we received was single words getting flagged. Messages like "join" or "wait" were being classified as spam, which is a bad look in any server where people talk normally.

This is a known weakness of models trained mostly on longer text: a very short message carries almost no context, and the model reaches for whatever weak signal it can find. 2.1 was trained with substantially more short-message data, and the Spam label specifically went from 0.772 to 0.959.

The tiers

All three models were retrained. Observer and Sentinel now use smaller, faster base models than 2.0 did, which is what lets them stay cheap, so their numbers are not directly comparable to the Arbiter figures above. Average F1 on the evaluation set:

If you are on Observer or Sentinel and you moderate a busy server, Arbiter is where the accuracy is.

Being straight about what this is

These numbers come from our own evaluation set, which we built and label ourselves. That is the right tool for tracking whether a retrain improved things, because it is the same yardstick across both versions. It is not an independent audit, and you should read any vendor quoting their own benchmark, including us, with that in mind.

What we can point at that is not self-scored is the feedback itself. The 👍 and 👎 votes are yours, they are recorded, and they are what we are steering by. That is a slower signal than a benchmark, and a more honest one.

Rolling out

2.1 is live now for every tier, on every plan, at no extra cost. There is nothing to update. If Supervisor is already in your server, it is already running the new models.

Try it on something that used to get wrongly flagged in the live demo, or add Supervisor to your server.

And keep pressing the buttons.