How to Fix Garbled Characters in a CSV (é Instead of é)

You open a CSV and every accented character has been replaced by nonsense:

name,city
René,Montréal
Zoë,Zürich
It’s fine,São Paulo

This is called mojibake, and the good news is that it is almost always exactly reversible — no guessing, no manual find-and-replace list. Once you understand what happened at the byte level, the fix is mechanical.

What actually went wrong

Text is stored as bytes. An encoding is the agreement about which bytes mean which characters.

In UTF-8, the character é is stored as two bytes: 0xC3 0xA9.

In Latin-1 and Windows-1252, every byte is exactly one character. So when a program reads a UTF-8 file but believes it is Windows-1252, it sees those two bytes and renders them as two separate characters:

| Byte | UTF-8 intends | Windows-1252 shows | |---|---|---| | 0xC3 | (first half of é) | Ã | | 0xA9 | (second half of é) | © |

Except most fonts render 0xA9 as ©, and what you usually see is é. Same idea.

That's the whole mechanism. Nothing was lost — the bytes are all still there. They were simply interpreted with the wrong agreement.

The lookup table you keep finding on other sites

Because the mapping is deterministic, the same corruptions appear over and over:

| You see | It should be | |---|---| | é | é | | è | è | | ü | ü | | ñ | ñ | | Ã¥ | å | | ç | ç | | ’ | ’ (curly apostrophe) | | “ | “ (opening quote) | | â€ | ” (closing quote) | | â€" | — (em dash) | | … | … (ellipsis) |

Most guides stop here and tell you to find-and-replace each pair. Don't. That list is incomplete by construction — it only covers whatever the author remembered. There are hundreds of possible sequences, and you will miss some.

The correct fix: reverse the byte interpretation

Rather than substituting characters, take each character, get the byte it represents in Windows-1252, and decode the resulting byte sequence as UTF-8. That reverses the exact operation that broke it.

Python:

broken = "René"
fixed = broken.encode("cp1252").decode("utf-8")
# 'René'

To repair a whole file:

import csv

with open("broken.csv", encoding="utf-8", newline="") as f:
    rows = list(csv.reader(f))

def fix(s):
    try:
        return s.encode("cp1252").decode("utf-8")
    except (UnicodeEncodeError, UnicodeDecodeError):
        return s  # already fine, or not recoverable — leave it alone

fixed = [[fix(cell) for cell in row] for row in rows]

with open("fixed.csv", "w", encoding="utf-8", newline="") as f:
    csv.writer(f).writerows(fixed)

The try/except matters. If a string is already correct, encoding it as cp1252 will usually fail — and that failure is your signal to leave it alone. Never apply this blindly to text that isn't broken.

Command line (iconv), when the whole file is uniformly affected:

iconv -f UTF-8 -t WINDOWS-1252 broken.csv > fixed.csv

That looks backwards, and it is — deliberately. You are asking iconv to write out the bytes that the broken text represents.

Why Windows-1252 and not Latin-1

You will see both names used interchangeably. They differ in one range that matters enormously in practice.

Bytes 0x80–0x9F are unused control codes in true Latin-1. Windows-1252 puts printable characters there — and they are exactly the ones that show up in mangled text:

| Byte | Windows-1252 | |---|---| | 0x80 | € | | 0x91 0x92 | ‘ ’ | | 0x93 0x94 | “ ” | | 0x97 | — |

This is why ’ appears instead of ’. The middle character is €, the Windows-1252 rendering of byte 0x80.

If you decode as strict Latin-1, smart quotes and em dashes will not recover. Use cp1252.

The other invisible culprit: the byte-order mark

A different problem with the same symptom of "the file looks fine but nothing works":

"id","name"

looks normal, but your code insists there is no column called id. Print the header and you get 'id'.

That is a byte-order mark — three invisible bytes (EF BB BF) some programs, Excel especially, write at the start of a UTF-8 file. It attaches itself to your first column name and breaks every lookup by name.

In Python, read with encoding="utf-8-sig" and it is stripped automatically:

open("file.csv", encoding="utf-8-sig")

How to stop it happening again

Fixing the file is half the job. The corruption usually happens at a predictable moment:

Don't double-click a CSV to open it in Excel. Excel guesses the encoding from your system locale and frequently guesses wrong. Instead use Data → From Text/CSV, and set File Origin to 65001: Unicode (UTF-8).

When exporting from Excel, choose CSV UTF-8 (Comma delimited), not plain CSV. Plain CSV writes in your system's legacy encoding and will mangle every accented character on the way out.

When exporting from a database, set the client encoding explicitly:

SET NAMES 'utf8mb4';        -- MySQL
\encoding UTF8              -- PostgreSQL psql

In your code, always specify the encoding. open("file.csv") uses a platform-dependent default — which is why a script works on your Mac and corrupts data on a Windows server.

Checklist

• Confirm it is mojibake, not a font problem — look for Ã, â€, or Â.
• Reverse it by re-encoding to cp1252 and decoding as UTF-8, not by substituting characters from a list.
• Leave text alone when the round-trip fails; that means it was fine.
• Check the first header for a stray byte-order mark.
• Fix the export step, or you will be doing this again next month.