Data quality: the silent killer of AI projects

Why the model gets the blame and the data gets a pass

When an AI project disappoints, the post-mortem almost always points at the model. Wrong choice of foundation model. Prompt wasn't tuned. Should've fine-tuned. We've sat in those meetings, and the conversation is comfortable because it's a technology problem with a technology fix. Swap the model, adjust a parameter, move on.

The pattern we see across the systems we ship is different. The model is usually fine. What's broken is the data underneath it, the customer records with three spellings of the same company, the product catalog that hasn't been reconciled since the last migration, the support tickets where half the 'resolution' field is blank. The model does exactly what it was asked to do with the inputs it was given. The inputs were the problem.

This is uncomfortable to say out loud on a project because data work isn't glamorous and nobody gets promoted for deduplicating a CRM. But if you skip it, you're building a fast, expensive machine for producing confident wrong answers.

• Model fixes are visible and satisfying; data fixes are invisible and tedious, so teams default to the former.
• A great model on bad data produces worse outcomes than a mediocre model on clean data, because the polish makes the errors look trustworthy.
• Most vendors sell models and tooling, not data hygiene, so the incentive to talk about data quality is low.
• Data problems compound silently; you rarely see them until a user catches an answer that's obviously wrong.

The five dimensions of data quality worth measuring

Data quality is vague until you break it into things you can actually check. We use five dimensions on every engagement, and we score them per use case rather than across the whole warehouse, because a field that's fine for billing can be useless for an AI assistant.

The point of naming them is triage. Once you can say 'completeness is our weak spot, the customer tier field is empty on 40% of records,' you've turned a fuzzy worry into a task someone can own and close.

• Accuracy: does the data match reality? A phone number that dials nobody, a status marked 'active' for a churned account. Accuracy errors are the ones that produce confidently wrong AI output.
• Completeness: are the fields you need actually populated? Missing values force the model to guess, and it guesses plausibly, which is worse than guessing obviously.
• Consistency: does the same thing get represented the same way everywhere? 'NSW', 'N.S.W.', and 'New South Wales' are three values to a machine, and joins quietly fail across them.
• Timeliness: is the data current enough for the decision? A 6-month-old price feed is fine for a trend report and a disaster for a quoting bot.
• Accessibility: can the system actually reach the data when it needs it? Perfect data locked in a PDF or a permission silo might as well not exist for an automation.

How bad data actually shows up in AI systems

The abstract version of this argument is easy to nod at and ignore. The concrete version is what changes behavior. Here's how each failure mode lands in the three kinds of AI work we build most: retrieval (RAG), generation, and automation.

In a RAG system, your assistant's answers are only as good as the documents it retrieves. Feed it a knowledge base with three conflicting versions of the refund policy and outdated process docs, and it will cite the wrong one with total confidence. The model isn't hallucinating; it's faithfully summarizing bad source material. We've debugged 'hallucination' complaints that turned out to be the retrieval layer doing its job on a polluted corpus.

In generation and analysis, skewed or incomplete data becomes biased output. If your historical hiring data over-represents one profile, a model trained or prompted on it will quietly reproduce that pattern. And in automations, the failures are loud or silent depending on luck, a malformed date field that throws an error is the good case, because the silent case is a workflow that routes 200 invoices to the wrong approver and nobody notices for a month.

• RAG: stale or duplicate documents produce wrong-but-confident answers, and users lose trust in the whole assistant after one bad cite.
• Generation: gaps and skew in the source data show up as biased, unbalanced, or subtly off outputs that are hard to spot at a glance.
• Automations: inconsistent formats and missing fields break the happy path, or worse, pass validation and do the wrong thing quietly.
• Classification and routing: mislabeled training examples teach the system the wrong rules, and the error rate looks like a model problem when it's a label problem.

You don't need perfect data, you need good-enough data

Here's the stance that saves projects: stop trying to clean everything. Perfect data across an entire organization is a multi-year program that never finishes, and if you make AI wait for it, the AI never ships. That's how good initiatives die, scoped so wide they collapse under their own weight.

The reframe is per use case. You don't need a clean warehouse. You need the specific fields, on the specific records, that one decision or one assistant actually touches, to be good enough that the output is reliable. Everything else can stay messy for now. A quoting bot needs current prices and accurate product mappings; it does not care that your marketing tags are a disaster.

Good-enough has a definition: the data is accurate, complete, and current enough that the AI's output meets the bar a human would accept for that task. Sometimes that's 99% on a financial automation. Sometimes it's 85% on an internal research helper where a human reviews the output anyway. Set the bar to the stakes, not to a vanity ideal.

• Scope cleanup to the fields and records the use case reads, and ignore the rest until a later use case needs it.
• Define the quality bar by the cost of being wrong: high-stakes automations need near-perfect inputs, low-stakes assistants tolerate more noise.
• Ship the use case with a human in the loop while data improves, rather than blocking on a cleanup that has no end date.
• Treat 'good enough' as a number you agree on up front, not a feeling you argue about after launch.

A practical way to assess what you've got

Before you fix anything, you need an honest read on where you stand. Most teams skip this and start cleaning the loudest problem, which is often not the costliest one. The assessment doesn't need a platform; it needs an afternoon and a willingness to look.

We run a lightweight version we call the use-case data audit. Pick the one AI use case you're building. List the exact data fields it depends on. Then, for each field, sample real records and score the five dimensions. You're not boiling the ocean, you're profiling the slice of data that matters for this one thing, which is a job you can finish this week.

• Name the use case and write down every field the AI will read, including the joins and lookups behind the scenes.
• Pull a random sample of 100 to 500 real records, not the demo data, because demo data is always suspiciously clean.
• Score each field on accuracy, completeness, consistency, and timeliness, and note where it physically lives for the accessibility check.
• Rank the gaps by impact on the output, then fix the top two or three and re-sample before declaring victory.
• Write the bar down: 'this field must be 95% populated and current within 24 hours' gives you something to test against later.

Using AI to clean the data that feeds AI

There's a useful loop here: the same models you want to deploy are genuinely good at the unglamorous work of preparing their own inputs. Not as a magic button, but as a fast, tireless first pass that a human checks. We've cut deduplication and standardization work from weeks to days this way.

The trick is to use AI for the fuzzy, judgment-shaped tasks that rules-based scripts choke on, matching 'Acme Corp' to 'ACME Corporation Pty Ltd', inferring a missing country from a city and postcode, flagging records that look wrong even when no single rule is broken. Then you keep a human on the high-stakes merges and let deterministic code handle anything a regex can already do cheaply.

One caution we hold firm on: AI cleanup is a suggestion engine, not an autonomous authority over your source of truth. Have it propose changes, log every one, and require sign-off on the merges that can't be undone. The goal is to speed up judgment, not to replace it with a confident black box, because cleaning bad data with an unsupervised model is how you launder errors into something that looks official.

• Entity resolution: let a model cluster likely-duplicate records and propose merges for a human to approve in bulk.
• Standardization: use AI to normalize free-text fields (addresses, job titles, categories) into a consistent vocabulary.
• Gap-filling: infer missing values from context where it's safe, and flag rather than guess where it's not.
• Anomaly flagging: ask the model to surface records that look off, the outliers a fixed rule would never catch.
• Always log and review: keep an audit trail of every AI-suggested change and gate the irreversible ones behind human sign-off.

Where to start on Monday

You don't fix data quality with a strategy deck. You fix it by picking one use case and refusing to let it ship on data you haven't looked at. The teams that get AI working aren't the ones with the cleanest warehouses, they're the ones who scoped the data problem down to something finishable and then finished it.

If you take one thing from this: the question is never 'is our data good?' It's 'is our data good enough for this specific thing we're about to build?' That question has an answer you can get to this week, and it's the difference between an AI project that quietly underperforms and one people actually trust.

When we start an engagement, the data audit comes before the model conversation, every time. It's the least exciting slide in the deck and the one that decides whether the rest of the work pays off.

• Pick the single highest-value AI use case and freeze the scope to it.
• Run the use-case data audit on the exact fields it depends on before writing any prompts.
• Fix the top two or three gaps, set a measurable good-enough bar, and ship with a human in the loop.
• Stand up a cleanup loop using AI for the fuzzy work, with logging and sign-off, so quality improves while the system runs.