Your model is only as auditable as its lineage
The question "what data is this built on" gets asked at review time, and it cannot be answered retroactively. By then the honest answer is a shrug with a confidence interval.
There is a moment in every model risk review where somebody asks what the system was trained on, or retrieves from, or was last evaluated against. It is a reasonable question. It is also, in most firms, unanswerable — not because the answer is bad, but because nobody wrote it down while it was still cheap to write down.
Lineage is a build-time artefact
Data lineage cannot be reconstructed. You can approximate it, and teams do: someone spends three weeks tracing tables backwards through a warehouse, produces a diagram, and everyone agrees it is roughly right. Roughly right is worth very little to a reviewer whose job is to attest to it.
The alternative costs almost nothing if you do it at the time. Every dataset that enters a model carries its source, its extraction time, its permission basis and its version. Every run records the exact versions it consumed. None of that is difficult. It is simply invisible while things are going well, which is why it keeps getting deferred.
Nobody has ever been asked for lineage on a day when everything was fine.
The permission problem underneath it
The harder version of this question is not where the data came from but whether you were allowed to use it that way. A retrieval system that indexes an internal share drive inherits every access assumption that drive ever made, including the wrong ones. The model does not know that the folder was readable because of a 2019 migration error. It will cheerfully summarise it for whoever asks.
This is where lineage stops being a documentation exercise and becomes a control. If you can trace a retrieved passage back to a document and that document back to a permission set, you can enforce something. If you cannot, your access model is whatever the index happened to crawl.
What good looks like
The bar is lower than people assume. Three properties cover most of it:
- Every artefact — dataset, index, model version, eval run — has an identifier that appears in the logs.
- Every production inference can be traced to the artefact versions that produced it, without asking a person.
- Permission is evaluated at query time against the requesting user, not baked in when the index was built.
None of that requires a platform purchase. It requires deciding, before the first system ships, that the answer to “what is this built on” should be a query rather than an investigation.
The cost of deferring
The reason this is worth arguing about now rather than later is that lineage debt compounds in a way most technical debt does not. An undocumented dataset gets copied, joined, and derived from. Three quarters later the thing you cannot explain is not one table but a small ecosystem, and the reconstruction cost has gone from a week to a project nobody will fund.
The systems that clear review are rarely the ones with the best models. They are the ones that can answer questions about themselves.