How to Remove Duplicate Rows in Excel and CSV Files

Duplicate rows creep into spreadsheets from everywhere: two exports merged together, a form that let people submit twice, a CRM sync that ran when it shouldn't have. They inflate your counts, skew your averages, and cause double-billing if the file feeds anything downstream.

Removing them is easy. Removing them correctly is where people get stuck — because the rows you think are duplicates often aren't identical, and the tools only catch exact matches.

The fastest method: Excel's Remove Duplicates

If your data is already in Excel:

1. Click any cell inside your data range.
2. Go to the Data tab → Remove Duplicates.
3. Check the columns that define a duplicate. Leave all checked to require an exact match across the whole row; check only Email to treat any repeated email as a duplicate.
4. Click OK. Excel reports how many it removed.

Excel keeps the first occurrence and deletes the rest. If you need to keep the most recent record instead, sort by date descending first.

The catch: Excel compares raw cell values. jane@example.com and Jane@Example.com (note the capital and trailing space) are two different strings to Excel, so both survive. More on fixing that below.

Google Sheets: UNIQUE or Remove Duplicates

Sheets gives you two options.

Menu method — select your range, then Data → Data cleanup → Remove duplicates. Same behavior and same limitations as Excel.

Formula method — leaves your original data untouched, which is safer:

=UNIQUE(A2:E1000)

Put that in an empty column and you get a deduplicated copy. To dedupe by a single column while keeping whole rows, combine it with FILTER and COUNTIF.

Python, if the file is huge

For files over a few hundred thousand rows, pandas handles it comfortably:

import pandas as pd

df = pd.read_csv("contacts.csv")
df = df.drop_duplicates()                      # exact match, whole row
df = df.drop_duplicates(subset=["email"])      # by one column
df = df.drop_duplicates(subset=["email"], keep="last")  # keep newest
df.to_csv("contacts-clean.csv", index=False)

Normalize before deduplicating, or you'll hit the same problem:

df["email"] = df["email"].str.strip().str.lower()
df = df.drop_duplicates(subset=["email"])

Why duplicates survive your first attempt

This is the part most guides skip, and it's the actual reason your row count didn't drop as much as expected.

Every dedupe tool compares text. Two rows describing the same person are only removed if their text matches exactly. In real exports it rarely does:

| Row A | Row B | Same person? | Matches? | |---|---|---|---| | jane@example.com | Jane@Example.com | Yes | No — case differs | | Jane Smith | Jane Smith | Yes | No — double space | | (415) 555-1111 | 4155551111 | Yes | No — formatting | | 2024-03-14 | 03/14/2024 | Yes | No — date format |

So the reliable sequence is normalize first, then deduplicate:

1. Trim whitespace from every cell.
2. Lowercase email columns.
3. Normalize phone numbers to one format.
4. Standardize dates to one format.
5. Then remove duplicates.

Run in that order, the near-matches above collapse into single rows. Run dedupe first and they all survive.

Choosing which columns define a duplicate

"Duplicate" depends on your data, and getting this wrong is worse than leaving duplicates in.

• Contact lists — dedupe on email. Two people can share a name; they rarely share an inbox.
• Transactions — dedupe on transaction ID, never on amount and date. Two legitimate $20 charges on the same day are not a duplicate.
• Product catalogs — dedupe on SKU.
• Survey responses — dedupe on respondent ID, or accept that repeat submissions are real data.

When in doubt, dedupe on the narrowest column that uniquely identifies a record, and always keep a copy of the original file.

A quick checklist

• Back up the original before you touch it.
• Decide which column defines a duplicate.
• Normalize formatting in that column first.
• Deduplicate.
• Compare the before and after row counts — if the drop looks wrong in either direction, investigate before shipping the file.