Cleaning 27,718 Directory Listings Without Deleting a Real Business

When we moved an older business directory onto dueclix.com, we imported 27,718 listings. The old platform had already blocked 25,934 of them — mostly spam registrations. The job was to decide which of the rest were real businesses that deserved to be public, without removing a single one that was.

This is a short account of how that went, including the rule we nearly shipped and the one that shipped with a gap in it.

Rule zero: unpublish, never delete

Every listing came across as a draft. Nothing was deleted — not the 25,934 the old site had blocked, and not anything we later judged to be spam. A draft is invisible to the public, absent from the sitemap and fully restorable.

That decision shaped everything after it. When a filter can only unpublish, a wrong call costs a restore rather than a lost customer record, and we could afford to review results instead of trying to be perfect the first time.

The heuristic we rejected

Spam business names often look like keyboard noise, so the first idea was a vowel ratio: names with too few vowels are probably random strings. It is cheap and intuitive, and it was the first rule we tried.

Before switching it on, we looked at what it would unpublish. The list included genuine companies — businesses whose names are acronyms, abbreviations or brand coinages. A vowel ratio cannot tell a random string from a real company that happens to be called something short and consonant-heavy.

The rule that shipped — and its blind spot

The first pass that went live looked for four or more transitions between upper and lower case inside a name. Random mixed-case strings — something shaped like QwRtZxPLmnBv — flip case constantly. Most real names, even stylised ones, flip far less.

It unpublished 38 listings. What it could not catch was junk written entirely in capitals. A string of capital letters has no case transitions at all, so it scores zero however random it is. Five such listings stayed public after the first pass.

They were caught afterwards, reviewed one by one and unpublished. One further listing was our own test entry. Those numbers are small, but the gap is structural: any single-signal rule has a shape it is blind to.

The rule we use from here on

The standard we have adopted for any future pass is that no single signal unpublishes a listing. A suspicious name alone is not enough; it has to be corroborated by something independent of the name — for example, a listing with no website and no description as well.

• Require corroboration. Two weak, independent signals are far safer than one strong-looking one.
• Preview before applying. Read the list a filter would remove, not just its size.
• Keep it reversible. Unpublish to draft; never delete during cleanup.
• Review what survives. The blind spot of a rule shows up in what it leaves behind, not in what it catches.

None of this is sophisticated, and that is the point. The risk in cleaning a directory is not that spam gets through for a day. It is that a real business quietly disappears because a clever rule misread its name — the same reason we argue for mapping a process before automating it.