dedupe

Deduplication

Removing duplicate rows from a dataset so each real-world entity appears once. The rule for "what counts as a duplicate" is the hard part.

When you'd see it: Almost any pipeline that loads from multiple sources or replays the same source. Customer records ingested twice. Events recorded by both client and server. Logs from redundant systems. A dedupe pass somewhere in the pipeline keeps downstream numbers honest.

Why it matters: Without dedupe, every metric double-counts. Revenue, headcount, active users, customer count — all wrong by some uncontrolled amount. Dedupe rules also encode editorial decisions: when two records disagree, which wins? Without an explicit rule, the answer is "whichever loaded last," which is rarely what you want.

Common mistakes: Deduping on exact field match when duplicates have slight variation (whitespace, capitalization, trailing punctuation). Fuzzy matching with a defined tolerance is usually closer to the truth, but it has to be specified, not implied.

Related ETL, ELT