Invisible Whitespace & Zero-Width Space Cleaner

Modern text processing relies heavily on Unicode, an expansive character encoding standard comprising over 149,000 discrete symbols. While Unicode enables worldwide multilingual software engineering, it also introduces dozens of invisible formatting markers, zero-width joiners, non-breaking spacers, and bidirectional control codes. When these non-printing characters find their way into source code, database queries, authentication credentials, or REST API payloads, they create silent failures that elude traditional visual debugging.

Because zero-width characters occupy zero visual width in standard user interfaces and text editors, a string such as "admin\u200B" appears completely identical to "admin" on screen. However, at the byte and binary level, their hash digests, memory representations, and string lengths differ completely. A strict equality check (userRole === "admin") evaluates to false, causing inexplicable logic bypasses, broken routing parameters, and corrupted JSON schemas.

The reference table below catalogs the most prevalent invisible Unicode code points, their standard typographic intentions, and their typical software engineering failure modes:

In November 2021, Cambridge University researchers disclosed a critical vulnerability class known as Trojan Source (CVE-2021-42574). This attack vector exploits the Unicode Bidirectional (BiDi) Algorithm. Because compilers parse source tokens in linear logical order while code editors render characters according to directional overrides, attackers can construct source code that appears entirely harmless in human code review while compiling into malicious binary instructions.

Implement the following battle-tested snippets in your backend microservices to clean user input and sanitize external text streams automatically:

export function cleanText(str) {
  return str
    // Strip zero-width & BOM
    .replace(/[\u200B-\u200D\uFEFF\u2060]/g, "")
    // Normalize NBSP
    .replace(/[\u00A0\u202F]/g, " ")
    // Strip BiDi controls
    .replace(/[\u202A-\u202E]/g, "");
}

import re

def sanitize_unicode(text: str) -> str:
    # Remove zero-width and BOM
    pattern = r'[\u200b-\u200d\ufeff\u2060]'
    cleaned = re.sub(pattern, '', text)
    # Replace non-breaking spaces
    return cleaned.replace('\u00a0', ' ')

-- Clean column data in PostgreSQL
UPDATE user_records
SET email = REGEXP_REPLACE(
    email,
    '[\x{200B}\x{200C}\x{200D}\x{FEFF}]',
    '',
    'g'
)
WHERE email ~ '[\x{200B}\x{200C}\x{200D}\x{FEFF}]';