Duplicate Finder: Automating a Tedious Eyeball Check

Paste a list, one item per line, and get back every line that appears more than once. That's the entire tool, and it exists because this is the kind of check people do constantly by eye in a spreadsheet or an exported list, squinting at rows trying to spot a repeat, when it's a problem a computer solves instantly and without the risk of a tired eye missing one.

Case sensitivity had to be a real choice, not a hidden default

Whether "Apple" and "apple" count as the same entry or different ones depends entirely on what the list actually represents, and there's no universally correct default. A list of email addresses should almost certainly be treated case-insensitively, since email delivery generally doesn't care about case. A list of case-sensitive identifiers or codes might need exact matching to be meaningful at all. Rather than silently picking one behavior, I exposed it as a visible toggle, because guessing wrong on this specific setting doesn't just give a slightly-off count — it can hide real duplicates or invent fake ones, which defeats the whole purpose of the tool.

Whitespace is where "duplicate" quietly gets ambiguous

A line with a trailing space and an identical line without one look the same to a human skimming a list, but they're different strings to a naive exact-match comparison. Pasted data is full of this kind of invisible inconsistency — copied from a spreadsheet cell, a stray space survives; typed by hand, it usually doesn't. I trim each line before comparing specifically because the visible content is what a person actually means by "duplicate," not the exact byte sequence, and a tool that reports two visually identical lines as unique because of an invisible trailing space would be technically correct and practically useless.

Large lists needed to stay responsive, not just eventually correct

A naive way to find duplicates is comparing every line against every other line, which works fine on a short list and gets noticeably slow as a list grows into the thousands of entries, since that approach's cost grows roughly with the square of how many lines there are. Building the duplicate check around a proper lookup structure instead — tracking how many times each line has been seen in a single pass rather than repeatedly re-scanning the whole list for every entry — keeps the tool responsive even on a genuinely large pasted list, like a full mailing list export, rather than freezing the tab for a noticeable moment on exactly the kind of large input this tool is most useful for in the first place.

The output format needed to answer "how many," not just "which ones"

Each repeated line shows up once in the results with its total count attached, rather than just a bare list of which lines repeated. Knowing that "apple" appears three times is a meaningfully different, more useful answer than just knowing "apple" is a duplicate at all — three occurrences might point to a real data problem worth investigating, while two might be an expected, harmless repeat. Showing the count turns a yes/no duplicate flag into an actual diagnostic signal about how bad the duplication in a given list actually is.

Empty lines needed an explicit decision, not a silent default

A pasted list very often has stray blank lines in it — a trailing newline at the end, an accidental double line-break in the middle from a messy copy. If blank lines are treated as regular entries, a list with three blank lines gets reported as having a "duplicate" that's really just an artifact of formatting, not a meaningful repeat anyone actually cares about. I made blank lines get skipped entirely before the duplicate check runs, since counting whitespace as a real list item would clutter the results with a false positive that has nothing to do with the actual data someone's trying to check.

Why this stays a paste-in tool with no file upload

I considered adding file upload support so people could point this at a CSV or text file directly instead of copying its contents out first, and decided against it for this first version. Paste-and-go keeps the entire interaction to two steps with nothing to configure about file formats or encoding, and it matches how people already have the data in hand most of the time — copied straight out of a spreadsheet or an email client, not saved to disk first specifically to check it for duplicates. A file-upload flow would add real complexity for a convenience most visitors wouldn't actually need for the quick, disposable checks this tool is built around.

This is a small tool with a small job, and I don't think that's a weakness. Some problems genuinely are simple once you strip away the tedium of doing them by hand — spotting a repeated line in a long list is exactly this kind of problem. The value here isn't clever engineering, it's just reliably automating a check that's boring and error-prone to do by eye, which is a perfectly good reason for a tool to exist even when there's nothing flashy to point at in the result. If a devlog post about a tool this small feels disproportionate, that's fair — but the small decisions inside it, case sensitivity, blank-line handling, staying fast on a large list, are exactly what separates a tool that quietly works from one that quietly gives wrong answers nobody ever double-checks. That's a fine trade for the amount of time this saves someone squinting at a spreadsheet, over and over, for as long as this site stays up, and that's a genuinely good return on one afternoon's work, tucked into a launch day with a dozen other tools that each earn their own more detailed writeup in their own time, none of them any less worth building carefully just because they're small, and none of them any less deserving of an honest account of what went into them than the biggest game on the site.