Non-ASCII Characters
A non-ASCII character is any character outside the 128 codes of standard ASCII (0–127). That includes accented letters, curly quotes, the em dash, emoji, and every non-Latin script. They are perfectly valid in Unicode and UTF-8, but they cause trouble when a system or tool expects plain ASCII. This guide explains what counts as non-ASCII, how to find and remove them, and the gotchas that bite programmers.
What counts as a non-ASCII character?
ASCII covers codes 0 to 127: the unaccented English letters, digits, common punctuation, the space, and control codes. Anything with a code above 127 is non-ASCII.
Common examples include accented letters (é, ñ, ü), curly "smart" quotes (“ ” ‘ ’) that word processors insert automatically, the en dash (–) and em dash (—), the degree sign (°), the bullet (•), non-breaking spaces, currency symbols like € and £, letters from other alphabets (ж, 你), and emoji. Many arrive by accident when text is pasted from Word, Google Docs, or a web page.
How to find non-ASCII characters
The reliable way is to search for anything outside the ASCII range using a regular expression. Most editors and languages support a Unicode-aware character-class match:
# grep — show lines containing any non-ASCII byte
grep -P -n "[^\x00-\x7F]" file.txt
# Python — list positions of non-ASCII characters
s = open("file.txt", encoding="utf-8").read()
bad = [(i, c) for i, c in enumerate(s) if ord(c) > 127]
print(bad)
# JavaScript — test whether a string is pure ASCII
const isAscii = str => /^[\x00-\x7F]*$/.test(str);
How to remove or replace them
You have two options: strip the characters, or replace them with an ASCII equivalent. Stripping is blunt but simple. Replacing is friendlier — turning a curly quote into a straight one, or é into e — and is often done with Unicode normalization.
# Python — strip everything non-ASCII
clean = s.encode("ascii", "ignore").decode()
# Python — transliterate accents to plain letters (café -> cafe)
import unicodedata
nfkd = unicodedata.normalize("NFKD", s)
clean = nfkd.encode("ascii", "ignore").decode()
# JavaScript — drop non-ASCII
const clean = str.replace(/[^\x00-\x7F]/g, "");
Common gotchas in programming
Non-ASCII characters cause bugs that are hard to spot because the characters look almost normal. The classic offenders are smart quotes pasted into code (“” instead of "), which break string literals, and the non-breaking space (U+00A0) masquerading as a regular space, which breaks parsing and indentation.
Other traps: a byte-order mark (BOM, U+FEFF) hiding at the start of a file; homoglyphs like the Cyrillic а that looks identical to Latin a but is a different character; and mojibake — garbled text such as é appearing where é should be — which happens when UTF-8 bytes are read with the wrong encoding. When something "looks right but won't work", suspect an invisible or look-alike non-ASCII character and inspect the raw bytes.