The Profanity Filter, and the Scunthorpe Problem It Had to Solve

A lightweight profanity filter went live today, checking display names at signup and whenever someone changes theirs in account settings. Display names are the one genuinely free-text field on this entire site that's shown publicly to every visitor — on leaderboards, on public profiles, on friends lists — with no moderation queue standing behind it, which is exactly the kind of surface that needs this kind of check running synchronously before anything gets saved.

This wasn't the first time site security has needed a real look

Back when accounts first launched in August, a security review turned up several genuine vulnerabilities that needed fixing before the account system could be trusted with real user data. This is a smaller, narrower issue by comparison, but it comes from the same underlying discipline: any field a stranger can put content into and have shown back to other strangers deserves a deliberate look at what could go wrong with it, rather than assuming good faith is a sufficient safeguard on its own. Display names sat exposed without this check for weeks longer than I'd have liked, and closing that gap today felt overdue rather than proactive.

A plain word-boundary check misses how usernames actually get typed

Most filtering approaches check for blocked words as complete, space-separated tokens, which works fine for filtering sentences and almost completely fails for usernames. Display names are usually typed as one unbroken string — something like "xXfoo123Xx" — with no natural word boundaries for a boundary-based check to even look for. A filter built around whole-word matching would miss the overwhelming majority of real attempts to sneak something past it, since there's rarely a clean word boundary anywhere in a typical username to anchor a match against in the first place.

Leetspeak and repeated letters needed normalization before matching, not during

Common evasion tactics — substituting numbers for letters, repeating characters to break up a pattern — are trivial for a human to read through but defeat a naive substring search looking for an exact blocked word. The filter normalizes common leetspeak substitutions and collapses repeated letters before running the substring check, which means the actual matching logic stays simple — plain substring search — while the normalization step in front of it does the real work of making evasion attempts collapse back down to a form the simple check can actually catch. Doing the hard work in normalization rather than in an increasingly complex matching algorithm kept the whole system easier to reason about and maintain.

The Scunthorpe problem meant normalization alone would over-block innocent names

Substring matching on a normalized string runs directly into a well-known problem, famously named after an English town whose own name innocently contains a substring most filters block on sight: plenty of ordinary, innocent words happen to contain a blocked fragment somewhere inside them, purely as a spelling coincidence. A filter that blocks any string containing a banned fragment, with no further nuance, ends up rejecting perfectly reasonable names for reasons that would look bizarre to anyone hitting the rejection with no context for why. I addressed this by stripping known-safe words out of the normalized text before running the substring check, so a legitimate word that happens to contain a flagged fragment doesn't trigger a rejection just because of where a problematic substring happens to sit inside a real, innocent word.

Running synchronously at write time was the only honest option here

Checking a display name asynchronously, after it's already saved and potentially already visible somewhere, would mean a window where an inappropriate name is live on the site before any moderation catches it — and with no moderation queue behind this feature at all, that window could be indefinite rather than brief. Running the check synchronously in the signup and display-name-update flow, rejecting a bad name before it's ever written to the database, was the only approach that actually prevents the problem rather than reacting to it after the fact. That's a meaningfully stricter constraint on the signup flow's performance than an async check would impose, and I think it's the correct trade for a field with zero moderation coverage behind it.

Why display names specifically, and not other free-text fields

There are other places on the site where someone types free text — a feature request, a bug report — and those don't get the same automated check. The difference is exposure: a feature request lands in a private inbox I read myself before anything happens with it, while a display name is immediately and permanently visible to every other visitor the moment it's set, on leaderboards and profiles neither I nor anyone else reviews before they go live. Fields that already route through a human before becoming visible don't need the same automated gate a field with zero review layer absolutely does.

This won't catch everything, and I want to say that plainly

A leetspeak-and-substring filter is a genuinely useful first line of defense, not a comprehensive moderation system. Determined evasion — creative misspellings, references a static word list simply doesn't anticipate — will still get through some of the time. I built this as the honest, achievable version of automated protection for a field this exposed, not as a claim that display names are now fully moderated. Closing the remaining gap means either a much larger detection system or actual human moderation, and this is the reasonable, immediate step available before either of those becomes necessary.

What I actually want from this feature is a meaningful reduction in the obvious, casual cases — the kind of thing someone types without much real effort to evade a filter — rather than a claim of total coverage against someone specifically determined to get something past it. That's a realistic bar for a synchronous, lightweight check to clear, and I'd rather ship something honest about that scope than something that implies more protection than it can actually deliver.

This is also a small enough piece of code that maintaining it going forward should stay manageable — a normalization table and a safe-word exclusion list, both easy to extend as new evasion patterns or false positives turn up. I'd rather have something this size that I can actually keep tuned over time than a much larger, more elaborate system that's harder to reason about and slower to adjust when it inevitably needs a correction, which every filter like this eventually does once enough real names have actually passed through it and shown me something the original list didn't anticipate.