How to Handle Duplicate, Missing, and Conflicting Catalog Records
A way to tell duplicate, missing, and conflicting catalog records apart, and to choose a survivor without inventing a match.
- 01 Classify the failure
- 02 Compare identifiers
- 03 Choose a survivor
- 04 Record the rule
Three failures, three different next steps
Duplicate, missing, and conflicting are not three words for a messy file. They ask for different actions. Treating them as one “data quality issue” is how a team merges records that should have stayed apart, or fills a blank with a value from the wrong row.
Use these definitions and do not soften them in the moment.
- A duplicate is two or more records that the identifier rule says are the same thing.
- A missing record or value is an absence the rule says should be present. Nothing has been matched yet.
- A conflict is two values that the rule says cannot both be true for one field of one record.
A blank title is missing. Two titles on one identifier is a conflict if the field allows only one. Two identifiers that a person suspects are the same product are not a duplicate until the identifier rule, or a documented match rule, says so. Suspicion is not a match.
This diagram scrolls sideways on a narrow screen.
Classify before you compare the text
Read the identifier first. If the rule says the identifier is the only match key, then different identifiers are different records even when the titles look alike. Similar titles are a review queue, not an automatic merge.
If you do have a documented match rule beyond the identifier, write the rule where the reviewer can see it: which fields are compared, what degree of difference is still a match, and who may confirm the match. Do not hide the rule in a script with no description. A future reviewer cannot appeal a decision they cannot read.
This publication does not publish a similarity percentage. Any threshold is local, and a number copied from another catalog would be a fake measurement. If you use a threshold, it belongs in your rule, with a named owner, not in an article as a universal cutoff.
Missing values are classified before anyone searches for a donor record. Required and optional fields decide whether the absence blocks release. A missing optional subtitle is not a duplicate problem and not a conflict. It is an empty optional field. Leave it empty.
Compare only what the rule names
Put the candidate records side by side with the fields the rule names, not with every column in the export. Extra columns invite a merge based on a field nobody agreed to trust.
For each field, mark one of four results: same, different and allowed, different and forbidden, or missing on one side. “Different and allowed” covers fields that are allowed to vary by channel or by language. “Different and forbidden” is a conflict. “Missing on one side” stays a missing value unless the match rule says the other side may fill it.
Do not average, concatenate, or pick the longer string as a default. Those habits feel tidy and create values that no source ever sent. If a person chooses a value, the change record should say a person chose it and which rule allowed the choice.
The source-of-truth rules decide which copy is allowed to commit. Quality control does not invent a second winner because this file arrived more recently. Recency is a winner only when the update rule says so.
This table scrolls sideways on a narrow screen.
| Type | What you have | What not to do | Next step |
|---|---|---|---|
| Duplicate | The match rule says these records are one thing | Keep both in production under two identifiers | Choose a survivor and point the other identifiers at it or retire them |
| Missing | A required value or record is absent | Copy a nearby row to fill the gap | Leave it blocked, or fill it from a source the update rule allows |
| Conflict | Two forbidden values for one field | Concatenate them or keep the longer one | Hold the record and apply the disagreement default |
| Look-alike | Titles resemble each other and the match rule does not hit | Merge on a hunch | Leave both records, and send the pair to review only if a person owns that queue |
Choose a survivor without rewriting the other record out of history
When the match rule confirms a duplicate, one record should remain the production record. The others should not vanish. Keep their identifiers resolvable: either they redirect to the survivor in whatever sense your catalog supports, or they remain as retired records that name the survivor. Deleting the identifier removes the only explanation a later file will have when it sends the old code again.
W3C Best Practice 27 says that when removing data from the web, preserve the identifier and provide information about the archived resource. That recommendation is about public web data and URI dereferencing. Use it here only if the duplicate’s identifier was public. Even then, the practice does not choose which record survives. Your match rule and update rule do that. The practice only argues against dropping the public identifier on the floor.
For an internal-only identifier, write your own retirement note. “Preserved” does not have to mean “still for sale.” It means a later process can see that this code was merged, retired, or rejected.
The survivor’s fields still pass the ordinary checks. A merge that creates a required-field gap has not finished. Send that gap back as a missing value. Do not paper over it with a fragment from the retired record unless the update rule allows that fragment to commit.
Record the rule that made the decision
Every merge, fill, or hold should leave a short record: identifiers involved, failure type, rule version, action, and person or job function if a person acted. Without that note, the next file will reopen the same pair and a different person will make the other choice.
The intake checklist should stop a row when this page’s type is duplicate or conflict. Onboarding is the wrong place to improvise the survivor. Quality control is also the wrong place to accept a brand-new source file that has not named its sender. Hand that back to intake.
What this page refuses to invent
It refuses a universal match score, a claim about how often catalogs contain duplicates, and a story about a company that cleaned its file in a weekend. Those are the kinds of statements that need a measurement this publication does not have. The method above is usable without them.
It also refuses to call a missing optional field a data incident. Severity follows the field rule. If everything empty is an incident, the incident queue becomes the catalog.
When to merge, fill, or wait
When is a look-alike a duplicate?
Only when the written match rule says so, or when a named reviewer applies that rule and records the decision. A shared word in the title is not a rule.
When needed: Before any identifier is retired.
When may a missing value be copied from another record?
When the update rule allows that source to commit the field. Otherwise the absence stays blocked or empty, depending on whether the field is required.
When needed: After classification, not during a hurried import.
When should you leave a conflict in place?
You should not leave it in the production field. Hold the record or apply the pre-written default. Showing both forbidden values as if they were one value is still a conflict.
When needed: As soon as the field rule says both values cannot be true.