The Question Your Master Data Hasn’t Been Asked Yet
And why the identity graph needs a streaming architecture too
For as long as I’ve worked on entity resolution and MDM, the question we’ve been answering has been the same: are these two records the same thing?
A batch job runs on a schedule. It compares records, scores them, merges what belongs together, and by the time anyone looks, there’s a clean, governed answer waiting. That answer does real work: finance closes the books against it, compliance points to it in an audit, analysts build a quarter’s worth of reporting on top of it. It’s not only humans reading that answer, either — a lot of agents run on it too. An agent working through a queue, planning a campaign, or scoring a portfolio against last night’s resolved customer base gets everything it needs from that cycle.
A second question, asked differently
Lately I’ve been anticipating a second question to show up soon next to the first one: not just “are these the same,” but “is this the same entity as of the events that have happened so far — including the one that just arrived.”
Most workflows — the ones working a queue, planning ahead, deciding against a resolved base — are still asking the first question, and batch answers it well. The second question comes from a specific need. As we get deeper into streaming architectures as well as agentic AI, a different pattern is bound to emerge: an agentic workflow that lives inside an event pipeline itself, reacting as things happen — an order placed, a record updated, a customer touched. That workflow isn’t asking the first question faster. It’s asking a question shaped differently, because it never stops to take a snapshot in the first place. It needs resolution structured around events as they occur.
I think this is the more interesting shift underneath everything else happening in AI right now. Every event that happens in a business — an order, a signup, a claim, a click — is a small act of identity being asserted, and right now, almost all of that gets set aside and reconciled later. What if it didn’t have to wait? What if identity resolution became something the business does continuously, as its own event stream, at the same pace the business itself moves? It’s identity treated as a live, ongoing process like the rest of the business — and I think it’s the direction this entire category is heading, whether or not it’s been named yet.
That gives us two architectures for the same underlying problem. The first one resolves identity as a periodic, governed batch process — which is where most reporting, marketing, analytics, governance, and a large share of agentic work already lives comfortably. The second one resolves it event by event, as each record arrives, for the future set of workflows and agents which will get built directly on a stream.
A workload shaped like a stream
An event-driven agent can’t cleanly consume the output of a batch job, for a simple structural reason: a batch job’s whole design is to collect events and process them later, in bulk, while an event-driven agent needs an answer the moment the event happens, not after it’s been folded into the next cycle. It’s less a speed gap than a shape mismatch — the same way a loading dock and a lobby entrance both belong to the same building, but neither one substitutes for the other.
In retail, most inventory and pricing agents work fine against a resolved batch base. An agent sitting inside an order-and-returns pipeline is a different case — it needs identity resolved inline, as each event fires, not pulled from yesterday’s merge. In financial services, plenty of risk-scoring and portfolio agents run well on batch, but an onboarding workflow built on a stream of KYC and core-banking events needs resolution that plugs directly into that stream. In healthcare, population-health and scheduling agents are largely batch-native, while a patient-event pipeline tracking admission, transfer, and discharge needs identity resolved as each event comes through.
Each of these is a small, specific version of a much bigger pattern: businesses are increasingly built on streams — of orders, of claims, of interactions — and identity, if it’s going to keep up, has to be built on that same stream rather than pulled off of it afterward.
Why real-time isn’t the fix
It’s tempting to think the answer to all this is simply “make it real-time.” In practice that instinct tends to go one of two ways, and I don’t think either one actually solves the problem.
The first version is a fast serving layer put in front of whatever’s already there. But that real-time, in the strict sense, is a claim about how fast you can retrieve an answer — it says nothing about whether that answer reflects the events that just happened. A millisecond lookup against a resolved record from last night’s batch run is still a millisecond lookup against last night’s answer; it’s just delivered faster. Speeding up access to a stale answer doesn’t make it current, it just makes the staleness harder to notice, because the response comes back so quickly that nothing about the interaction signals it might be out of date. The problem an event-driven agent actually has isn’t retrieval latency, it’s that the state it’s querying hasn’t caught up to reality yet — a freshness problem, not a speed problem, and a fast-serving layer bolted onto an otherwise unchanged batch process doesn’t touch it. It just gives you a quick answer to the wrong question.
The second version is a separate real-time resolution engine, living outside the data platform, that you feed a copy of your records so it can match them fast on its own. This solves the speed problem and reintroduces the exact one identity resolution was supposed to end. This architecture has three structural flaws.
Identity lives outside the governed platform by design. The core architecture is: pull data out via API/ingestion, build a persistent identity index in a separate system, then query that index instead of the warehouse. That means a second copy of your entity data exists somewhere your platform’s access controls, lineage, and governance don’t reach. You’ve traded a lag you could reason about for a disagreement you can’t. This is a hard thing to explain in an audit, and a harder thing to walk back once it’s become load-bearing infrastructure.
Two sources of truth, quietly. You’ve created a second place where “who this customer is” gets decided — with its own copy of the data, its own view of what’s been merged, and its own drift away from whatever the warehouse thinks is true. Once matching decisions are made in that external index rather than in the platform, there’s no guarantee the “resolved” answer there matches what the warehouse’s own tables say. Nothing forces the two to agree, and there’s no single audit trail spanning both. Downstream systems quickly go out of synch with the identities, and the identity layer by design becomes a sidecar, not a system of record. An agent querying that engine gets a fast answer, but nobody can tell you with confidence whether it’s the same answer the rest of the business is operating on.
Single-record query architecture struggles with bulk change. These systems are optimized for "resolve this one new record against the index" — not for cheaply reprocessing history when a matching rule changes, a new attribute is added, or an error needs correcting at scale. That's a real limitation when your matching logic evolves, which it always does.
Both “real time” versions solve for speed and leave the actual problem untouched. What an event-driven agent needs isn’t a faster answer to last night’s question, and it isn’t a second, disconnected place where identity gets decided — it’s a resolution process that treats each event as work to do the moment it arrives, running in the same governed place the rest of identity already lives. That’s a different problem than raw speed, and I think a more honest one to solve.
Where I think this is heading
The interesting design question isn’t “how do we make batch faster.” It’s what identity resolution looks like once it stops being a periodic pass over the past and becomes a live process running alongside the business itself — two architectures at once: the periodic, governed batch process that already serves reporting, analytics, marketing, compliance, governance, and most agentic work well, and an event-streaming architecture that resolves identity record by record, in step with the business as it happens, for the agents built to move at that pace.
I think that second architecture is going to matter more than most people currently expect, because agentic AI doesn’t just create a new consumer for identity — it creates a much larger number of moments where identity gets asked about or updated, continuously, all day, instead of in one batched pass. An organization that can answer “who is this, right now, as of what just happened” as naturally as it can answer “who was this, as of last night” has a genuinely different foundation to build agents on. That’s the infrastructure question I think the next few years of MDM are actually about.
There’s a second part to this that matters just as much as the architecture: where that event stream actually runs. For the event-streaming architecture to be trusted the same way the batch one is, it has to run natively inside the data platform itself — not adjacent to it, not upstream of it, not as a separate hop the data takes on its way in. Same governance boundary, same lineage, same access model, just consuming a stream of events instead of a periodic extract, and keeping pace with the business rather than catching up to it.
That’s the problem I’ve spent the last stretch of time building toward, and I’m not ready to talk about the specifics yet. But I wanted to put the shape of the question out there, because I think most teams are going to run into this before they have a name for it — an event pipeline that needs identity resolved inline, meeting a batch-shaped answer that wasn’t built to plug into it.
Have you run into that mismatch yet — a workload built on events that couldn’t cleanly consume a batch-shaped resolution process? I’d genuinely like to hear how it showed up for you.