In my last few posts, I wrote about the persistent ZINGG_ID and incremental flows. Quick recap for those who missed them. As data keeps changing in a growing business - new customers, updated addresses, records arriving from new channels - the identity graph has to keep up. The incremental flow in Zingg Enterprise handles exactly this. New and updated records get matched against existing clusters, clusters merge and split automatically, and the ZINGG_ID travels with the entity through all of it. The identity graph stays consistent even as the data underneath it keeps moving.

That was a big piece of work to build and ship. But as we rolled it out to more customers, we started hearing about a different scenario. One we had not fully planned for.

Some of our customers had been running Zingg for a while. Their identity graphs were in good shape. Incremental flows were humming. And then, their data got better.

Phone numbers that were unreliable at the start of the project - maybe coming from a poorly integrated source system, or just sparsely captured - were now being collected consistently. Email fields that were mostly empty were now populated. A new column from a recently onboarded data source was throwing up a great matching signal. The data engineering team had done some serious cleanup work on one of the source systems.

In other words, the matching could now be sharper than it was when the graph was first built.

And yet, teams were not acting on it. We started hearing the same thing again and again. Customers who loved their ZINGG_IDs - and were relying on them deeply - were finding that the very thing they valued was also holding them back. One customer had significant architecture changes and needed to run a full refresh, but was worried about losing their ZINGG_IDs. Others wanted to add a new matching signal but held off. Some wanted to move to a different data platform entirely, but the ZINGG_ID dependency made the migration feel too risky.

The Zingg identifier had become load-bearing infrastructure. Which we loved to see - it meant the graph was genuinely embedded in how the business operated. But it also meant that any change to the graph felt dangerous. Customers were not able to move from one model to another, capture more signals, or even change their data platform because of Zingg. That was not the position we wanted our customers to be in.

So, what do you do when you want a full refresh - run matching again on the full dataset, this time with the richer fields, the better signal, the cleaner data? But your downstream systems are wired to the existing ZINGG_IDs.

Reporting tables, Customer 360 dashboards, ML feature stores, operational workflows at the call centre - all of them reference the ZINGG_IDs from the previous run. If you blow away the graph and rebuild it, you get better clusters but you lose the identifier continuity. Every downstream system breaks or needs a migration. That is a huge cost, and it tends to make teams postpone the refresh indefinitely. The identity graph gets frozen at the quality level it was when it was first built, because improving it feels too disruptive.

This is the gap that Reassign closes.

After you run a full refresh with your updated fields and retrained model, reassign maps the new cluster assignments back to the existing ZINGG_IDs. It carries the established identifiers forward wherever the entity continuity holds. Downstream systems keep their reference point. The identity graph gets the benefit of the improved matching signal. The refresh is no longer a disruptive operation.

Here is the even more interesting bit - what taking the persistent ZINGG_ID one step further really means. Persistence, as we built it earlier, means the identifier travels with the entity through routine changes - new records, updated records, incremental loads. Reassign extends that guarantee to cover a deliberate, intentional rebuild of the graph itself. The identifier now survives not just the churn of normal operations, but also the data team’s decision to go back and do it better.

We think of this as data maturity support built into the identity graph. Because in practice, the signals get better over time. Source systems get cleaned up. New integrations bring in new attributes. The identity graph should be able to evolve with that maturity - not fight against it.

If you have a full refresh sitting in your backlog because the downstream impact felt too risky, we would love to hear from you. Hit reply and tell us what triggered it - what improved signal are you sitting on that you have not yet been able to incorporate?

A16z just published a piece that every data practitioner should read. Your Data Agents Need Context makes the case that AI agents have been failing not because the models are bad, but because they’ve been operating without the business context they need to reason well — no understanding of how revenue is defined, which tables are the source of truth, what a fiscal quarter actually means for this organization. The fix, they argue, is a modern context layer: richer than a semantic layer, encompassing canonical entities, tribal knowledge, governance logic, and identity resolution.

They’re right. And I want to extend that argument into territory I think deserves more attention — because the implications of taking it seriously go well beyond the examples that are easiest to reach for.

This Is Data Teams’ Moment — The Question Is How Big We Think

I wrote recently about how agentic AI is finally giving data teams their long-overdue moment. For years, data teams spent budget cycles defending their existence. That is changing fast. AI agents are only as good as the data they run on, and the foundational work data teams have been quietly doing for years — quality, governance, lineage, entity resolution — is now the prerequisite for every enterprise AI initiative that matters.

The budget conversation is flipping — from “justify why we need you” to “how fast can you get us AI-ready?” That’s a genuine shift. But it comes with a risk: if the measure of success for data agents settles at “answering questions BI could already answer,” we’ll have spent enormous energy rebuilding, more expensively and with more failure modes, something that already worked.

Revenue last quarter? Your BI team has a Looker dashboard for that. It refreshes nightly, the fiscal calendar is baked in, and the CFO trusts it. The a16z piece walks through exactly why an agent struggles to replicate even this — wrong revenue definition, stale YAML, ambiguous tables. That’s a real problem worth solving. But it’s also worth asking: once we’ve solved it, what have we actually unlocked?

The answer, I’d argue, is: the foundation for something far more valuable.

The Questions That Context-Plus-Identity Actually Makes Possible

The a16z piece mentions that a modern context layer should go beyond metric definitions to include canonical entities and identity resolution. I want to take that seriously and spell out what it actually unlocks. Walk into any commercial or strategy meeting and listen for the questions that make the room go quiet — not because the answer is sensitive, but because nobody actually knows: How many real customers do we have — not accounts or rows in a CRM, but distinct humans or organisations with an active relationship? Who are our most valuable customers, across product lines, referrals, and renewal behaviour? What is our customers’ lifetime value when “Acme Corp” in the CRM and “ACME Inc.” in billing are finally recognised as the same entity? Who can we upsell, and when, based on what similar cohorts did next? Which customers are three months from churning, based on signals scattered across product, support, and finance systems? These questions couldn’t be asked before not because agents were too dumb, but because the identity layer that makes them answerable was never in place.

What Agents Actually Do with a Resolved Identity Graph

This is where the discussion stops being theoretical. Once entity resolution is in place — once the agent knows who it’s reasoning about across every system — a new class of autonomous workflows becomes possible. Here are four that illustrate the shift.

The Churn Prevention Agent. The agent monitors product usage daily — login frequency, feature adoption, support ticket volume — and when a customer’s pattern matches historical churn signatures, it autonomously cross-references their contract renewal date, checks their NPS history across systems, and drafts a personalised outreach for the customer success manager with a suggested intervention. The entire workflow collapses if the agent is seeing three different records for the same customer across the product database, CRM, and support tool. Entity resolution is not a prerequisite step — it is the agent’s eyes.

The Upsell Timing Agent. A customer’s usage of Product A crosses a threshold. The agent compares their trajectory against cohorts of similar customers who went on to adopt Product B within 90 days, finds the pattern match, creates a qualified opportunity in the CRM, and alerts the account manager — without anyone pulling a report. The cohort comparison is only valid if the purchase history and usage data it’s drawing on are resolved to the same underlying customer. A fragmented identity graph means the agent is comparing apples to a mix of apples and ghosts.

The Cross-Sell Across Household or Org Agent. In B2C — a bank, an insurer, a retailer — one household member has a mortgage, another has a current account, a third just opened a savings product. An agent that understands household relationships, not just individual records, can surface the cross-sell opportunity and route it to the right team before a competitor does. In B2B, the same logic applies at the organisational level: one division is already a customer, another is in a competitor’s trial. The agent identifies the relationship and acts. Both scenarios are structurally impossible without entity resolution at the household or org level — and both are routine once it’s in place.

The Supplier Risk Agent. The agent continuously watches news feeds, financial filings, and logistics data for signals about key suppliers — credit downgrades, port disruptions, leadership changes. When a risk threshold is crossed, it maps which SKUs are affected, what inventory buffer exists, and which alternative suppliers are pre-qualified, then surfaces a recommended action to the procurement team. The catch: this requires knowing that “Acme Logistics Ltd” in the ERP, “ACME Logistics” in the contracts system, and “Acme Log.” in the invoicing platform are the same supplier. Without that, the agent is monitoring fragments, not entities — and the risk signal it’s supposed to catch slips through the gaps.

What “Canonical Entities” Actually Requires

It’s worth dwelling on the practical aspects of having canonical entities and identity resolution as components of a modern context layer — because this is where most implementations will either succeed or quietly fail.

I’ve written before about why building a unified customer identity is genuinely hard, and why so many teams feel like imposters for still wrestling with it. The industry narrative treats unified identity as a prerequisite — something that should already be done before you get to the interesting work. In reality, the vast majority of organisations are still fighting this battle. They’re just not talking about it publicly because it feels like admitting failure.

Entity resolution — sometimes called record linkage, identity resolution, or master data management — is the discipline of determining that two records refer to the same real-world person, company, or object, even when names, formats, and identifiers don’t match. It requires probabilistic matching across messy strings, address normalisation, fuzzy deduplication, graph-based clustering, and ongoing maintenance as records evolve. It’s one of the oldest unsolved problems in enterprise data, not because people haven’t tried, but because the messiness is structural. Organisations grow through acquisition. Customers interact across channels. Nobody standardised data entry in 1997.

For the context layer to include this, entity resolution:

• Cannot be hard-coded into YAML or a semantic layer — it requires ML-powered probabilistic matching trained on actual data

• Cannot be a one-time batch job — it needs to run continuously as new records arrive and existing ones change

• Cannot live outside the warehouse — moving billions of records to a separate resolution system reintroduces the very data silos and lack of context that AI agents are struggling with

• Cannot be fully automated — it needs human-in-the-loop workflows to handle edge cases and feed corrections back into the model

This is not background infrastructure that someone configures once and forgets. It’s an ongoing data capability — closer in character to a production ML system than to a semantic layer definition. Teams that treat it as the former will find their context layers actually working. Teams that treat it as the latter will find their agents confidently wrong.

What This Looks Like When Done Right

Two examples illustrate what the context layer vision looks like when the entity resolution component is actually implemented well.

Fortnum & Mason, the 300-year-old luxury British retailer, had customer data fragmented across restaurant bookings, email signups, online transactions, and in-store purchases. They tried a third-party resolution service first — it created non-persistent identifiers, offered limited visibility, and raised privacy concerns about sending their entire customer dataset externally. By implementing Zingg natively in their Databricks environment, they built persistent, unified customer identifiers across all touchpoints, with full control over the matching process and privacy-compliant resolution that kept sensitive data in their own infrastructure. For the first time in their history, they could understand how customers were shopping across every channel and devise clienteling and personalisation around that. That’s not a BI question answered better. That’s a question that was previously unanswerable and an action that could not be done earlier.

Orthodox Union, operating over 40 websites and 5 mobile applications across disparate CRMs, used Zingg on Snowflake to power their golden records — resolving not just individual identities but household relationships across their entire digital estate. Their agents now reason about constituents as whole people with family connections, not as disconnected records across systems.

These outcomes are exactly what the context layer is promising. They require taking the entity resolution component as seriously as the metric definition component — which means treating it as an engineering discipline, not a checkbox.

The Architecture That Makes This Real

The a16z piece lays out a thoughtful five-step architecture: access the right data, build context automatically, refine with human input, connect to agents, keep it self-updating. That framework is right. The entity resolution layer sits at the foundation of step two — and its quality determines the ceiling for everything above it.

Concretely: before an agent reasons about revenue, lifetime value, churn risk, or upsell opportunity, it needs a stable, persistent entity graph that tells it who “this customer” actually is across every system. That graph needs to be built where the data lives, continuously maintained and grounded in human judgement for edge cases :

Only on top of this identity foundation does the rest of the context layer — the metric definitions, the table routing, the tribal knowledge — become genuinely trustworthy. Knowing what “revenue” means is important. Knowing that the revenue from “Acme Corp” and “ACME Inc.” should be attributed to the same customer is what makes the answer actually right.

The Real Payoff

As I argued in The Data Team’s Moment, AI doesn’t paper over data problems — it runs them at scale, in production, with consequences. A customer success agent operating on a fragmented identity layer doesn’t just give one bad recommendation; it systematically misjudges an entire customer segment, at machine speed, before anyone notices. The autonomous nature of agents is what makes the identity foundation so critical — errors compound rather than get caught.

But the inverse is equally true, and more exciting: an agent operating on a trustworthy identity graph can answer questions that were structurally impossible before. Not because the model got smarter, but because the data it’s reasoning on finally reflects reality.

AI agents that can tell you which accounts are actually the same company, which of those companies are in an expansion window, which customers share a behavioural signature with cohorts that churned six months ago — that’s the payoff the context layer is reaching for. The revenue-growth question is the proof of concept. The customer intelligence actions are the business transformation.

The context layer is necessary. Entity resolution is what makes it sufficient. And the questions worth asking, and the agents worth running — about customers, not metrics — are what make the whole thing worth building.

Building on the a16z post “Your Data Agents Need Context” by Jason Cui and Jennifer Li. Further reading: The Data Team’s Moment and The Identity Crisis. If you’re building the entity resolution layer, Zingg is open source and fully documented.

There’s a meeting that happens in every organization, usually sometime in Q3 or Q4. A CFO pulls up the budget spreadsheet, squints at the line items, and asks the question that makes every data leader’s stomach drop: “What exactly are we getting out of the data team?”

It’s an uncomfortable question — and for a long time, it didn’t have a clean answer.

Data teams built pipelines nobody asked about, maintained dashboards that were exported to Excel before anyone actually read them, and spent political capital defending headcount to executives who viewed them as a sophisticated cost center. As Joe Reis, author of Fundamentals of Data Engineering, put it plainly: “Data teams will continue struggling to justify their value as long as data in Enterpriseland is a back-office function.”

That was the reality for most of the 2010s and into the early 2020s. Data teams worked in the shadows — vital infrastructure, but rarely celebrated. They were the plumbers of the modern enterprise: essential when things worked, blamed when things didn’t, and invisible the rest of the time.

The era of agentic AI is changing all of that — right now, in real time.

The Old Indignity: Fighting for a Seat at the Table

The data ROI conversation has long been a recurring humiliation dressed up as a business exercise.

Unlike sales, whose numbers are on the board every Monday morning, or marketing, which can point to impressions and conversions, data teams have operated in the murky middle. Their impact is indirect — they help other teams make decisions. As data writer Anna Geller captured it well: “Most data teams work as a support function... their involvement in value creation is indirect. You can’t directly quantify the impact of a new table, dashboard, or pipeline.”

This indirectness has been weaponized against them at budget time. Data leaders find themselves in a paradoxical position: the team responsible for quantification can’t quantify itself. Some resort to building internal P&L reports to justify their own existence. Others try charging other departments “internal fees” for data services — a workaround that says more about the dysfunction of how data is valued than it does about the team’s actual contribution.

Meanwhile, data stacks keep growing enormous. As Monte Carlo’s blog noted, “data teams made history as one of the first departments to spin up 8-figure technology stacks with little to no questions asked” — a fact that makes them conspicuous targets whenever CFOs start scrutinizing every line item with new intensity.

The modern data stack is a marvel of engineering. But without a clear, legible link to revenue, it remains a liability in any budget conversation. At least, it has been — until now.

The Shift Underway: AI Agents Are Looking for a Nervous System

Here’s the thing about agentic AI: it is only as good as the data it runs on.

This seems obvious, but its implications are still rippling through the enterprise in real time. When a company deploys an AI agent to autonomously manage customer service tickets, optimize supply chains, approve procurement requests, or run financial reconciliations, every decision that agent makes is being built on a data foundation. If that foundation is shaky — inconsistent definitions, poor lineage, unresolved duplication — the agent doesn’t just underperform. It fails loudly, in production, with consequences.

Robin Sutara, Field CDO at Databricks, is making this organizational implication explicit in Databricks’ 2025 strategic priorities report: “A successful AI strategy starts with a solid infrastructure. Addressing fundamental components like data unification and governance through one underlying system lets organizations focus their attention on getting use cases into the real-world, where they can actually drive value for the business.”

The work that data teams have been doing for years — data quality, entity resolution, governance, lineage, pipeline reliability — is no longer background noise. It is becoming the prerequisite for the most important initiative in every company.

The budget conversation is flipping. It’s no longer “justify why we need you.” It’s “how fast can you get us AI-ready?”

The Numbers Bearing This Out

The scale of the agentic AI wave is making clear why data teams are starting to matter in a way they simply haven’t before.

Gartner predicts that 40% of enterprise applications will be integrated with task-specific AI agents by the end of 2026 — up from less than 5% today. By 2028, at least 15% of work decisions across organizations will be made autonomously by AI agents, up from virtually zero in 2024. And Gartner’s best-case projection suggests agentic AI could drive approximately 30% of enterprise application software revenue by 2035, surpassing $450 billion.

The old framing — tools automate tasks, people make decisions — is no longer holding. And the people who build, govern, and maintain the data powering this new paradigm are moving from the back office to the top of the boardroom agenda.

The Verizon Test Case

Few leaders are articulating this shift as clearly as Kalyani Sekar, Verizon’s Chief Data Officer. In a 2025 interview with CDO Magazine, she describes the arc of her team’s work — and its growing centrality — with unusual candor:

“Data quality is foundational to Verizon’s long-term goals. As we continue to roll out more analytical use cases, from predictive and prescriptive to generative and agentic, it’s critical that these AI and analytical capabilities, including the reports built on them, are grounded in data that is trustworthy and of the highest quality.”

Verizon handles petabytes of data daily — from network devices, customer touchpoints, service channels. The data team’s job was always to make sense of it and surface insights. Today, that same infrastructure is becoming the operating layer for autonomous AI systems making real-time decisions affecting 140 million wireless retail connections.

The data team isn’t changing. The stakes around its work are.

From Cost Center to Control PlaneFrom Cost Center to Control Plane

McKinsey’s 2025 research on the “agentic organization” is describing what’s unfolding as the largest organizational paradigm shift since the industrial revolution. The key insight for data leaders: in this new world, data quality cannot remain a background maintenance task. It needs to be the foundation every AI agent is built on — before a single workflow goes live.

Nowhere is this more visceral than in entity resolution — the problem of making sure your AI actually knows who it’s dealing with.

I have been watching this play out in real time across industries: “Agentic AI is the race car. Entity resolution is the track. You wouldn’t run a Formula 1 machine on gravel and expect to win.”

The examples are stark. In retail, duplicate customer records lead autonomous agents to dispatch multiple shipments, issue multiple offers, and trigger multiple refunds — quietly eroding margins at scale. In finance, fraudsters opening several accounts with small name variations go undetected because the fraud-detection AI treats each record as a separate person. In healthcare, a patient known as “Mary Chen” at one hospital and “M. Y. Chen” at a clinic becomes two different people in the AI’s world — and an autonomous care coordinator misses her allergy record before booking a procedure.

In each case, the AI wasn’t making bad decisions. It was making confident decisions on bad data. And because agentic systems are autonomous, they don’t make that mistake once — they repeat it hundreds or thousands of times before anyone notices.

Tools like Zingg — which resolve entities at scale across CRMs, EHRs, billing systems, and data lakes, running natively on Snowflake and Databricks — are becoming the kind of foundational data infrastructure that determines whether an AI deployment succeeds or silently fails. One bank that placed Zingg’s entity resolution layer beneath its fraud-detection AI saw fraud detection improve 4x — without changing the AI model at all. The model didn’t get smarter. The data it was reasoning on did.

That’s the new pitch data leaders are making — and it’s a powerful one. AI doesn’t paper over your data problems. It runs them at scale, in production, with consequences. Every dollar not being invested in data quality is a growing tax on every AI initiative downstream.

What This Means for Data Leaders Right Now

The opportunity is real and it is opening — but it requires a change in posture. Data teams have spent years learning to survive by making themselves useful to whoever controls the budget. The new play is positioning as the precondition for everything the business cares about.

Some practical realities for data leaders navigating this moment:

Your backlog is becoming a risk register. Every unresolved data quality issue, every undocumented pipeline, every inconsistent metric definition is a live failure point for an AI agent that will execute on bad information at machine speed. Frame your roadmap accordingly.

Entity resolution is becoming a first-class AI problem. Before an AI agent can act intelligently, it needs to know who it is acting on — and right now, most enterprise data lakes are full of the same customer, supplier, or patient under a dozen slightly different names. Deduplicating and resolving those identities isn’t a one-time cleanup job anymore; it’s an ongoing data foundation that every agentic workflow sits on top of. Teams that are investing in entity resolution infrastructure — using tools like Zingg to continuously resolve identities across CRMs, EHRs, and data platforms — are finding that their AI deployments perform dramatically better, without touching the models at all. The intelligence was always there. The data just wasn’t ready for it.

Data governance is becoming AI governance. The governance frameworks data teams have been building for years — classification rules, data retention policies, lineage tracking — are exactly what enterprises now need to keep AI agents grounded in reality.

The CDO is becoming a critical executive. The function that once reported to the CIO is increasingly driving the AI strategy conversation alongside the CEO and CFO.

The dbt Labs 2025 State of Analytics Engineering survey is already showing the shift: 40% of data teams added headcount in the past year — compared to just 14% the year before. Data budgets are growing too, with 30% of respondents reporting increases versus just 9% in the prior year. The most common complaint is no longer budget cuts — it’s that the team can’t keep pace with the business’s appetite.

As dbt Labs CTO Mark Porter put it directly: “As companies increase AI investments, leaders are prioritizing the teams responsible for data quality and governance — the essential foundation for AI effectiveness.”

The Table Is Turning

There’s something quietly satisfying about watching an industry vindicate itself not through persuasion, but through circumstance.

Data teams have spent years arguing that their work mattered. They built ROI frameworks and internal P&Ls and political coalitions. They embedded with business units and learned to speak in revenue terms. Some of it worked. Most of it was an uphill grind.

Now, agentic AI is making the argument for them — not through rhetoric, but through dependency. The most transformative technology initiative of the decade cannot function without clean data, governed pipelines, and trustworthy infrastructure. And the people building and maintaining those things are, right now, becoming central to the enterprise.

The data team is no longer supporting the business.

The business is starting to run on the data team.

If you found this useful, share it with a data leader who’s still spending Q4 justifying their existence to a skeptical CFO. Their moment is here — and it’s only getting bigger.

Matthew Niederberger, Founder of Martech Therapy recently wrote something that stopped me in my tracks:

“If you’re a practitioner still trying to get a unified customer ID working across channels, you feel left behind technologically and question your competence.”

This single sentence captures a crisis that’s quietly plaguing data engineering teams across the industry. While vendor booths showcase AI-driven personalization and predictive customer journeys, talented practitioners are privately wrestling with something supposedly “simpler”: building a unified customer ID that actually works.

The cruel irony? They’re doing the hardest, most important work. But in a market obsessed with bleeding-edge features, nobody’s celebrating it.

The Vendor Narrative Gap

So why do talented teams feel like they’re failing when they’re still working on identity?

Matthew Niederberger nails the dynamic: vendors showcase what’s possible, not what’s required to get there.

Vendor conference presentations follow a formula:

1. Demonstrate the amazing outcomes (real-time personalization! predictive journeys!)

2. Show the sleek UI that makes it look effortless

3. Mention “unified customer data” as an assumed prerequisite

4. Move quickly past the foundation to focus on the flashy features

The implicit message: if you’re still working on unified identity, you’re behind. Everyone else has already solved this.

But that’s not remotely true. The vast majority of organizations are still struggling with identity resolution. They’re just not talking about it publicly because it feels like admitting failure.

The result is a massive selection bias. We hear about the 5% who’ve moved on to advanced use cases. We don’t hear about the 95% still building foundations—not because they’re incompetent, but because the foundations are genuinely difficult to build well.

The Imposter Syndrome Epidemic

I’ve had countless conversations with data teams at conferences, in Slack channels, and on video calls. A pattern emerges again and again:

“We’re still working on identity resolution.”
“We haven’t gotten to the AI stuff yet.”
“We’re just trying to get our customer IDs right first.”

The word “just” does a lot of heavy lifting in those sentences. It diminishes foundational engineering work into a checkbox, a prerequisite you should have already mastered before moving on to the “real” work.

But here’s what nobody is saying loudly enough: unified identity isn’t a prerequisite. It’s the final boss of data engineering.

Why Unified Identity Is Actually Hard

Let’s be brutally honest about what “getting a unified customer ID working across channels” actually requires:

The Data Quality Reality

Real-world customer data is messy in ways that mock deterministic rules:

• Sarah Johnson gets married and becomes Sarah Martinez

• john.smith@gmail.com and jsmith@gmail.com might be the same person (or different people with similar names)

• Phone numbers get recycled—last year’s customer is this year’s prospect

• Addresses have hundreds of valid formatting variations

• People move, change jobs, update preferences

• Data entry errors create systematic inconsistencies

A unified customer ID system has to handle all of this, at scale, while maintaining high accuracy. Miss too many matches and your “360-degree customer view” is fragmented. Create false positives and you’re merging different people, violating privacy and destroying trust.

The Technical Challenge

You’re not matching records in a single clean database. You’re resolving identities across:

• Web analytics (cookie-based, increasingly unreliable)

• Mobile apps (device IDs, app-specific identifiers)

• CRM systems (email, phone, account IDs)

• Point-of-sale systems (loyalty cards, payment methods)

• Call center interactions (phone numbers, case IDs)

• Marketing platforms (each with their own identifier schemes)

• Third-party data sources (with their own quality issues)

Each source has different identifier types, different data quality standards, and different update frequencies. You’re not just joining tables—you’re probabilistically determining which records across fundamentally different systems represent the same human being.

The Scale Problem

Now multiply this across millions or billions of records. The naive approach—comparing every record to every other record—is computationally impossible. You need sophisticated blocking strategies, efficient algorithms, and infrastructure that can process massive datasets without collapsing under its own weight.

And you need to do this continuously, not as a one-time batch job. New data arrives constantly. Identities evolve. Your resolution system has to keep pace.

The Governance Complexity

Layer on consent management, privacy regulations, data retention policies, and the right-to-be-forgotten requirements. Your unified ID system isn’t just a technical challenge—it’s a governance framework that has to work across jurisdictions, respect customer preferences, and provide audit trails.

This isn’t “basic.” This is some of the most complex data engineering work you can do.

The Three-Layer Approach That Actually Works

So how do you actually build identity resolution that works at warehouse scale? Let me break down the architecture that successful implementations share:

Layer 1: Blocking

You can’t compare every record to every other record—that’s O(n²) complexity and computationally impossible at billion-record scale. Smart blocking reduces potential comparisons by 99%+ while maintaining high recall.

The key is choosing blocking keys that cast a wide enough net to catch true matches while being selective enough to make comparison tractable. This might mean blocking on:

• First three characters of last name + ZIP code

• Phone area code + first name

• Email domain + approximate birth date

• Phonetic encoding of names

Good blocking strategies use multiple passes with different keys, ensuring that records with typos or variations still get compared. At Zingg, we learn blocking directly on all the fields present in the data from the user data and labels to block different datasets efficiently.

Layer 2: Matching

This is where ML earns its keep. For each pair of records that passed blocking, you need to score the likelihood they represent the same entity.

Modern matching models use features like:

• String similarity (edit distance, Jaro-Winkler, token-based matching)

• Phonetic similarity (Soundex, Metaphone, NYSIIS)

• Behavioral patterns (purchase timing, channel preferences, engagement signals)

• Temporal proximity (how close in time were interactions?)

• Geographic consistency (do locations make sense together?)

The output isn’t binary—it’s probabilistic scoring. This pair has a 0.95 probability of being a match. That pair scores 0.45. This gives you flexibility in tuning precision vs. recall based on your use case.

Layer 3: Clustering

The final challenge: transform pairwise match scores into entity groups. This is where you resolve transitive relationships.

If Record A matches Record B with 0.92 confidence, and Record B matches Record C with 0.88 confidence, are A, B, and C all the same entity? What if A and C compared directly would only score 0.65?

Clustering algorithms handle these transitivity questions, creating stable entity groups even as new records arrive and scores shift over time. This layer also handles entity splits (when you discover you incorrectly merged two people) and entity merges (when new evidence shows records you thought were separate are actually one entity).

Why This Must Happen In the Warehouse And Datalake

Here’s a critical architectural decision: entity resolution must run natively in your data warehouse or data lake. Moving billions of records out for processing defeats the entire purpose of warehouse-native architecture.

Data Gravity

Your customer data is already in Snowflake, Databricks, BigQuery, or Redshift. That’s where your transformation pipelines run, where your analytics queries execute, where your data science models train. Extracting data to a separate entity resolution system creates:

• Data movement costs (bandwidth, storage, processing)

• Synchronization complexity (keeping multiple copies current)

• Latency (time to move data before resolution can happen)

• New data silos (exactly what warehouse consolidation was supposed to eliminate)

Incremental Processing

Entity resolution isn’t a one-time batch job. New customer records arrive constantly—from web signups, mobile registrations, purchase transactions, support interactions. You need incremental processing that resolves new records against existing entities without full dataset re-processing.

This only works efficiently when resolution runs where the data lives, using the warehouse’s native capabilities for incremental updates and change data capture.

Integration with Data Pipelines

Your entity resolution system needs to integrate seamlessly with:

• dbt transformations and data modeling

• Reverse ETL for audience activation

• BI tools for reporting and analytics

• ML platforms for model training

• Data quality frameworks for monitoring

• AI agents for context

This integration is natural when everything operates in the same environment. It becomes painful when entity resolution is a separate system with its own APIs, data formats, and operational requirements.

Governance and Security

Your data warehouse already has:

• Access controls and role-based permissions

• Audit logging for compliance

• Encryption at rest and in transit

• Data masking and tokenization capabilities

• Retention policies and deletion workflows

Replicating customer data to an external entity resolution system means duplicating all of this governance infrastructure. It becomes a privacy and compliance nightmare with GDPR, CCPA and other applicable laws. Run resolution in the warehouse, and you inherit these controls automatically.

The Human-in-the-Loop Problem

Here’s what the automation-obsessed narrative misses: pure ML automation for entity resolution rarely works in practice.

Why Full Automation Fails

Even the best ML models make mistakes on edge cases:

• Common names with similar addresses (are these two John Smiths in the same apartment building one person or two?)

• Shared email addresses (family members, assistants, corporate accounts)

• Identity changes (marriages, name corrections, gender transitions)

• Data entry errors that create systematic patterns the model learns incorrectly

• Inadequate data capture

You need human judgment on ambiguous cases, and you need to feed that judgment back into the model to improve over time.

Workflows That Actually Work

Successful implementations build workflows for:

Training data labeling: Which record pairs are matches, which are non-matches? Your initial model needs labeled examples, and getting these labels right determines everything downstream.

Threshold tuning: ML models output probability scores, but you choose the threshold. Score above 0.90 is an automatic match, below 0.40 is automatic non-match, but what about the 0.65 that could go either way? Human review helps calibrate these thresholds based on business impact.

Edge case review: The model flags cases it’s least confident about. A data steward reviews them, makes decisions, and those decisions become training examples for model tuning.

Continuous model refinement: Data patterns evolve. New data sources get added. Data quality issues emerge and get fixed. The model needs periodic retraining, and human feedback on recent predictions drives that retraining.

Dispute resolution: When downstream teams question why records were merged or split, you need workflows to investigate, explain the decision, and potentially override the model when it’s wrong.

The best systems don’t eliminate human review—they make it efficient. Instead of manually reviewing millions of records, you review the hundreds that matter most.

Common Implementation Mistakes

I’ve seen these patterns repeatedly as organizations tackle identity resolution:

Mistake 1: Starting Too Big

Teams try to resolve all entity types simultaneously—customers, accounts, products, locations, employees. This creates enormous scope, makes success criteria fuzzy, and delays time to value.

Better approach: Start with one high-value entity type (usually customers). Prove the value, build organizational capability, then expand to other entity types. Each new entity type gets easier because you’ve learned the patterns.

Mistake 2: Ignoring Data Quality

Entity resolution surfaces every data quality issue you’ve been ignoring. If 40% of your email addresses are null or invalid, you can’t rely on email matching. If name fields contain inconsistent formatting, parsing errors, or non-name data, string matching becomes unreliable.

Better approach: Treat entity resolution and data quality as inseparable workstreams. Profile your data first. Fix systematic quality issues before expecting resolution to work well. Use entity resolution results to identify and prioritize remaining quality problems.

Mistake 3: Treating It as a Project

Organizations approach entity resolution as a one-time project: scope it, build it, deploy it, move on. Then they’re shocked when accuracy degrades over time as data patterns shift.

Better approach: Treat entity resolution as infrastructure that requires ongoing ownership, monitoring, and refinement—just like your data pipelines or ML models. Assign clear ownership, establish SLAs, build monitoring dashboards, schedule regular review.

Mistake 4: Optimizing for Perfection

Teams spend months trying to achieve 99.9% accuracy before releasing anything. Meanwhile, downstream teams make decisions based on no identity resolution at all (effectively 0% accuracy).

Better approach: Ship at 95% accuracy, deliver value, iterate to higher accuracy based on real-world feedback. You learn more from production usage in two weeks than from six months of pre-release testing. Start good, iterate to great.

Mistake 5: Building From Scratch

Organizations underestimate the complexity and decide to build custom entity resolution systems. Eighteen months later, they have something that works on small datasets but can’t scale, has accuracy issues they don’t understand, and requires specialized knowledge that only one person has.

Better approach: Use proven solutions—whether open source frameworks or commercial platforms—that solve the hard algorithmic and scaling problems. Invest your engineering effort in customization, integration, and business-specific logic, not reinventing core entity resolution algorithms.

The Real Question

At this point in conversations, someone usually asks: “Should we do ML-based entity resolution?”

That’s the wrong question.

The real question is: “Can our organization afford to build composable architectures on a foundation of deterministic rules that we know will fail at scale?”

Every organization moving to warehouse-native architectures will eventually face the identity resolution challenge. The only question is whether you address it proactively—as part of your architectural planning—or reactively, after your first major data quality incident.

The reactive path looks like this:

• Build your composable architecture on simple deduplication rules

• Activate audiences and launch campaigns

• Discover that 15% of your “unified” customers are actually duplicates

• Or worse, discover you’ve merged different people and violated privacy

• Scramble to retrofit proper identity resolution

• Rebuild audiences and downstream dependencies

• Deal with the business impact and loss of trust

The proactive path looks like this:

• Recognize identity resolution as foundational, not an afterthought

• Build or adopt ML-based entity resolution that runs in your warehouse

• Start with one entity type, prove value, iterate

• Build downstream capabilities on accurate identity from day one

• Scale confidently knowing your foundation is solid

The reactive path is more common (because people underestimate the challenge). The proactive path is more successful (because foundations matter).

What Actually Endures

Here’s the uncomfortable truth about chasing bleeding-edge features without solid foundations:

A CDP that can’t accurately unify customer IDs is just an expensive data warehouse with a marketing UI.

You can bolt on AI-driven orchestration, predictive segmentation, and real-time decisioning. But if your underlying identity graph is wrong—if you think one customer is three people, or three people are one customer—every downstream capability inherits those errors.

Your AI doesn’t fix bad identity data. It amplifies it. Your personalization engine delivers fragmented experiences. Your predictive models train on fictional customer journeys. Your real-time decisioning makes decisions based on incomplete context.

The organizations actually positioned to leverage advanced CDP capabilities aren’t the ones with the longest feature lists. They’re the ones who took the time to build reliable identity resolution first.

The Real Achievement

Let me share two examples of organizations that built this foundation right.

Fortnum & Mason, the 300-year-old luxury British retailer, faced a classic challenge: customer data fragmented across restaurant bookings, email signups, online transactions, and in-store purchases. They initially tried a third-party service for identity resolution, but it created non-persistent identifiers, offered limited visibility into the matching process, and raised privacy concerns about sharing their entire dataset externally.

Jon Moss, Fortnum’s Customer Engagement Director, describes the transformation:

“For the first time, we’re able to understand how customers are shopping with us—online, in-store, over the phone, or in restaurants. Zingg has helped us unify this data and gain insights we never had before.”

By implementing ML-based entity resolution running natively in their data warehouse, Fortnum & Mason built:

• Persistent, unified customer identifiers across all touchpoints

• Complete visibility and control over the matching process

• Privacy-compliant resolution that keeps sensitive data in their own infrastructure

• A foundation for their composable CDP architecture

Orthodox Union, one of the largest Orthodox Jewish organizations in the US, operates over 40 websites, 5 mobile applications, and custom CRMs for different departments. Shelomo Dobkin, their Director of Product Development, explains:

“We moved from using Zingg Open Source to the Enterprise version on Snowflake, making it the engine powering our golden records. Zingg’s powerful features and clean results have really set us apart.”

Their implementation required understanding not just individual identities but household relationships—a critical requirement for their community-focused mission. They built entity resolution that could:

• Match people across multiple websites and CRM systems

• Understand household connections and family relationships

• Create golden records that serve as the foundation for all downstream systems

• Process data incrementally as new interactions occur across their digital properties

What makes these achievements remarkable isn’t just the technical implementation. It’s what they enable. This is world-class data engineering that delivers measurable business value:

• Marketing teams can trust their audiences

• Analytics teams can build on reliable customer journeys

• Data science teams can train models on accurate historical data

• Operations teams can deliver consistent experiences across channels

• Legal and compliance teams can demonstrate proper governance

Everything else—the AI features, the real-time orchestration, the predictive models—becomes possible because the foundation is solid.

Celebrating Foundational Work

The industry needs to reframe how we talk about identity resolution and other foundational data work.

These aren’t prerequisites that should be quick and easy. They’re achievements that deserve recognition.

When a team successfully builds unified customer IDs across disparate systems, that’s a case study worth sharing. When an organization finally has audiences their teams trust, that’s an accomplishment worth celebrating. When identity resolution maintains accuracy as data scales from millions to billions of records, that’s engineering excellence.

The castle that survives wave after wave isn’t the tallest or flashiest. It’s the one with foundations deep enough to withstand the tide.

Moving Forward

If you’re a data practitioner still working on unified customer identity, you’re not behind. You’re doing the work that matters most.

Don’t let vendor demos and curated LinkedIn posts make you feel like an imposter. The bleeding-edge use cases get all the attention precisely because they’re rare. The foundational work gets less visibility precisely because it’s what everyone is actually struggling with.

Focus on making your identity infrastructure trustworthy and durable:

• Invest in ML-based entity resolution that can handle real-world data messiness

• Build in your data warehouse and datalake so you’re not creating new data silos

• Design for continuous operation, not one-time batch processing

• Create human-in-the-loop workflows for edge cases and model refinement

• Treat it as infrastructure that requires ongoing ownership and monitoring

The teams that master this foundation aren’t behind. They’re building something that will outlast the next wave of features, the next vendor pivot, the next technological hype cycle.

That’s not just the basics. That’s the real work. And it’s work worth doing right.

What’s your experience building unified customer identity? Where did you hit the biggest obstacles? I’d love to hear your stories—the successes and the struggles.

Last week, I was finetuning the Zingg website, when something caught my eye. We have a quote on Fortnum & Mason’s journey from fragmented customer records to a unified view with Zingg:

That sentence got me thinking: in an era where we can track a customer's digital breadcrumbs across every touchpoint, why do so many brands still struggle with the simple question: "Do you really know your customer?"

The Identity Resolution Promise That Keeps Getting Broken

Most of us have watched the same promises get made over and over again. First, it was Master Data Management (MDM) systems that would create "one golden record." Then identity services like LiveRamp promised to connect everyone's messy data through their identity graph. Then for the last few years, Customer Data Platforms (CDPs) became our solution to the elusive single customer view.

But here's what I keep seeing: enterprises continue to struggle with messy and fragmented data. Multi-year MDM implementations that are still not very useful. The MDM system could tell you the exact lineage of how John Smith's email address made it through seven data transformation steps, but it couldn't figure out that J. Smith from the mobile app was the same person. The system was simply not ready for the messy reality of how customer data actually behaves.

The same pattern plays out with identity services. A different promise: "Don't worry about cleaning your own house — we'll connect everyone's messy data." Unfortunately, identity services operate as closed systems where you can't see or validate their matching logic. When they say Customer A matches Customer B, you're essentially trusting a proprietary algorithm without transparency. This becomes problematic when you need to adjust matching to your specific business context, explain results to stakeholders and debug why certain matches seem incorrect. Add to that the privacy risk of sending trusted data to a third party. Designed around fixed data models (email, phone, cookie IDs, etc.), it is tough to resolve on more complex or domain-specific attributes (like healthcare patient IDs, loyalty card numbers, or industry-specific identifiers).

CDPs were supposed to fix everything by democratizing customer data for marketers. The reality? Most became expensive data silos with pretty dashboards, lots of data movement yet the hard work of actually resolving who is who got pushed to the background.

What Actually Works

This is why I'm excited about what the team at Fortnum & Mason have built. Starting with data scattered across restaurant bookings, email sign-ups, online orders, and in-store transactions, they built a composable CDP using Zingg for entity resolution on Databricks.

The transformation wasn't about technology for technology's sake.

What made their approach different? They treated entity resolution as a first-class capability, not an afterthought. They started with Zingg's open source version to build confidence, understood how the matching worked, then moved to the enterprise version. The entire journey from proof of concept to production took just 2-3 months. For the first time, they can understand how customers shop across all channels. Suddenly, “one customer” meant the same thing everywhere — and the results have been transformational, not just for analytics, but for personalization and customer experience.

Or look at the Orthodox Union, a non-profit serving millions globally. They struggled with donor and member records scattered across legacy systems, event platforms, and CRM databases. Without a reliable way to unify identities, their engagement strategies risked missing context and duplicating outreach. By resolving identities natively in their Snowflake data warehouse, they were able to create a consistent single view of donors and members, while respecting privacy and compliance requirements central to their mission. For them, identity resolution wasn’t just a data challenge — it was a way to deepen community trust.

These are two very different organizations, but the pattern is the same: composability delivers agility, while identity delivers coherence. Without both, the vision of modern customer experience falls short.

The Composable CDP Revolution

Fortnum & Mason's and Orthodox Union’s approach reflects a broader trend that's reshaping the market. Composable CDPs sit on top of existing data clouds to preserve the customer 360 and activate real-time data across channel tools. Instead of moving data around, they work where the data lives. They are private by design. They leverage existing investments in governance and observability. They scale as you grow.

But here’s the hard truth: Composable architectures give you flexibility — but only if the components are stitched together with a stable, reliable, and explainable identity layer. You can have the most elegant semantic layer, real-time pipelines, and activation tools, but if “Acme Corp” and “ACME Inc.” are treated as two different customers, your insights fracture. If one table says “Tim Chen” and another says “T. Chen,” your campaigns lose relevance. If customer data is duplicated at ingestion, those cracks echo through every model, dashboard, and recommendation.

And here’s the other piece we don’t talk about enough: owning your identity layer means owning your customer understanding. Instead of shipping your most sensitive data out to a black-box vendor, you resolve identities where your data already lives — in your own warehouse or lake. That means:

• Data ownership: You stay in control of your customer data, critical in a world of tightening privacy regulations.

• Schema flexibility: As your business evolves — new channels, attributes, and data types — your identity system evolves with it, instead of breaking because it was designed for someone else’s schema.

• Privacy and compliance: By resolving identities within your existing data governance framework, you respect customer consent and meet regulatory requirements without creating extra data copies.

• Stack leverage: You’re not rebuilding infrastructure. You’re making your existing investments in warehouses, lakes, and analytics more powerful by adding the missing link: identity.

Why I'm Hosting This Roundtable

This brings me to London. On 9th September, I'll be at the Martech World Forum hosting a roundtable titled "Do You Really Know Your Customer?" — and it will be lovely to meet you if you are there!

Here's what we'll cover:

The Identity Crisis in Marketing Why traditional approaches to customer identity keep failing, and how agentic AI makes the problem more urgent than ever.

Building Composable CDPs That Actually Work Moving beyond tool marketing to understand what composable really means, when it makes sense, and how to implement it successfully.

The Zingg Approach to Entity Resolution How modern ML-based entity resolution differs from traditional MDM, why transparency matters, and how to get started with open source tools.

Real Customer Stories on fragmented data to unified customer understanding, including the practical challenges and business outcomes. The folks from Fortnum & Mason will join us to share their real-world experience, the challenges they overcame, and the business impact they've achieved.

The roundtable format means we can go deep, ask tough questions, and learn from each other's experiences. Whether you're just starting your customer identity journey or you've been through multiple implementations, there is value in understanding what's working now and what's coming next.

Join Us in London

I have a limited number of VIP passes available for the Martech World Forum. If you're a data or marketing leader in London, I'd love to share the passes and have you join the conversation in person. Because at the end of the day, the real question isn’t just “do you know your customer?” — it’s whether your systems know them too.

Not just their demographic data or their transaction history, but their actual journey across all your touchpoints? Can you recognize them when they call customer service after browsing your website? Do you know that the person who signed up for your newsletter is the same one who made a purchase in your store?

If you can't answer these questions confidently, you're not alone. But you also can't afford to stay in that position much longer.

The companies that figure this out — the ones that truly know their customers — are going to have an unfair advantage in the age of AI-driven experiences. They'll deliver more relevant products, more timely services, and more personal interactions. They'll waste less money on duplicate processes and misguided campaigns.

Most importantly, they'll build the kind of customer relationships that survive economic downturns, competitive pressures, and technology transitions.

And if you can't make it to London but have war stories about identity resolution initiatives (successful or otherwise), I'd love to hear them. This isn't really about technology — it's about understanding the humans behind the data. And that, I've learned over the years, is both the hardest and most rewarding part of what we do.

Want to learn more about entity resolution and composable CDPs? Check out Zingg's open source project or read about how we're helping companies like Fortnum & Mason build better customer understanding.

Two weeks in, the VP of Marketing noticed something strange. One of their top VIP customers had received three different “Welcome to our brand!” offers — each with a different discount.

Turns out, their CRM had John A. Smith, J. Smith, and Jonathan Smith as three separate customers.

The agent wasn’t broken. It was doing exactly what it was told — treat each record as a unique person and delight them with a welcome offer. The problem? The records were wrong.

And this, I’ve realized over and over again, is where agentic AI stumbles: when the entities are messy, the automation multiplies the mess.

Over the past few years, I’ve seen this pattern play out in every industry:

Retail — Duplicate customer records mean multiple shipments, multiple offers, multiple refunds. Margins erode quietly.

Finance — Fraudsters open several accounts with tiny name variations. Without linking those identities, your fraud-detection AI treats them as separate people — and misses the pattern.

Healthcare — Patient “Mary Chen” at Hospital A is “M. Y. Chen” at Clinic B. An AI care coordinator misses her allergy record and books a risky procedure.

In each case, the AI wasn’t “dumb.” It was confidently acting on bad data. And because agentic AI is autonomous, it didn’t just make one mistake — it repeated the mistake hundreds or thousands of times before anyone noticed. Smart agents get the “who” wrong and this really hurts the business.

If you capture the hype, everyone talks about the cost of building and running agentic AI. Yet, fewer people talk about the cost of running it on messy data. Agents run on LLM calls, and every action burns tokens. When your entities aren’t resolved, the agent:

• Processes duplicates

• Repeats actions unnecessarily

• Sends extra API calls trying to reconcile

Here’s the math from just one example scenario:

*Based on $15 per 1M tokens (GPT-4.1 tier).

Even with cheaper models, the % waste stays the same. That’s just compute waste. It doesn’t even include the cost of fixing the damage — reissuing refunds, restoring trust, or dealing with regulatory fallouts.

Why I care about this (and what we built at Zingg)

Before I started Zingg, I kept running into the same frustration:
We had amazing ML models, sophisticated rules engines, smart agents… but they were all reasoning on fragmented, messy entities.

At Zingg, we decided to treat entity resolution as a first-class product, not a one-off data cleanup exercise. That means:

• Matching at scale across CRMs, EHRs, billing systems, and data lakes

• Adaptive learning so the matching improves with every variation we see

• Transparent decisions so you can explain why two records were linked

When customers put this layer under their agentic AI, they stopped paying for their agents to repeatedly do the wrong thing. In one case, a bank’s fraud detection improved 4x — without touching the AI model.

My advice to data leaders

If you’re about to launch an agentic AI project, take a breath.
Before you buy another GPU hour or fine-tune another model, ask:

“Do my agents actually know who they’re dealing with?”

Because in my experience, agentic AI is the race car. Entity resolution is the track.
You wouldn’t run a Formula 1 machine on gravel and expect to win.

P.S. If you’re curious about what we are building at Zingg, or have war stories about entity chaos in AI systems, I’d love to hear from you.

Earlier last month, we started working on a feature we are calling Match Statistics. The idea about statistics came from the incredible folks at Fortnum And Mason, who are using Zingg as the backbone for Single Customer View. While running Zingg incrementally, the Match Statistics would expose how clusters numbers change as records get inserted and updated into the identity graph. This information about changing clusters would be a good first step to observe the entity resolution pipeline. If the number of clusters change disproportionately to the number of records updated and added, an alert could be triggered.

As we started thinking about this feature, we realized that the Match Statistics feature could be made even more powerful. Currently the Zingg output consists of the original data along with the Zingg ID and the match probabilities. If we could surface information about the linkages we found in the records within the cluster, we could help users with matching internals and anomalies. When our users are dealing with millions of records, finding the needle in the haystack is critical. Hence, the specification for the feature expanded to additional statistics.

While building new features, or augmenting existing flows within Zingg, we have to design both the Community and Enterprise products, since they share the same code base. The design of any feature on either product has to take into consideration the impact on both products and ensure that they continue to work individually as well as in tandem. And then comes the trickier part. The design has to work well for both Snowflake and Spark deployments. Not just functional. Performance wise as well. The Zingg Enterprise code base is mostly common across Spark and Snowflake. However, since both platforms have very different APIs, query planning, optimisation and execution, we have a lot of ground to cover while building or modifying anything.

Thats what we thrive at :-)

Eventually we designed a stats package which would get the record level and cluster level matches from the match process. This package would be responsible for computing the metrics we cared about, and writing them to appropriate locations, or Pipes in Zingg lingo. The main flow would mostly be unaltered, except for invoking the stats package and passing the dataframes around.

As we started building the core functionality, we started seeing a very strange issue. Within our flow, we do probabilistic matching only on records which do not match deterministically. If two records match deterministically, we do not try to figure out if they match probabilistically as well since that is a waste of computation time.

Now, something strange started happening. Within the stats package, which was consuming the probabilistic and deterministic structures, we found some records which matched probabilistically as well as deterministically. This threw an error in the statistics computations, as they banked on the record pair to match one way or the other or not match at all. Our deeply crafted logic was breaking.

However, this part of the flow never changed. All we were doing was computing additional metrics in this package. And the main flow was unaltered. So how could stuff which was working earlier AND which was untouched stop working? What followed was a crazy cycle of runs, tests and validations.

Our initial thought was that we had made an error in collecting the data. Why not run it in another sandbox? Umm, the same result. Maybe we had run the flow twice appending to original intermittent data leading to the wrong state? But if that was the case, every record pair would appear twice. In our case, only some of the record pairs were being matched probabilistically as well as deterministically. We then sought to understand if there was anything special about those records. We looked at column values, deterministic rules, and the probabilistic match criteria. To our dismay, every run threw up a different set of extra records. But clearly, not data or code issue then.

Things were surely getting curiouser and curiouser. We were completely off track from the original feature we set out to build :-(

Next, we turned our focus to the execution environment. When we ran the build on Snowflake, there were no extra records. This was a relief, since it meant it was the Spark side of things that we needed to focus on. Now we ran the workload on Databricks. No issue at all! This was a relief too, since it meant earlier released production code deployed at customer locations was working fine and there was no snake in the grass.

After two weeks of racking our brains, we were finally making progress :-) Things started moving pretty fast post that. We searched for known issues and discovered a issue on Apache Spark. When we changed the Apache Spark version, the issue disappeared. To be absolutely sure, we tested different datasets, different volumes, and all looked good.

Debugging complex systems is always challenging, but some techniques always help.

1. Time travel to the last known good state helps understand which change or set of changes could be the cause.

2. Surgical elimination is super critical in such scenarios. Being able to quickly run different non-overlapping tests can lead to faster results.

3. Weird behavior - missing records, extra counts etc which alter at different runs are more likely to be caused by the environment than the actual code.

4. Such non deterministic errors usually stress teams out, since things feel very out of control. For engineers like us who are used to predictable outcomes, such issues can be very frustrating. Having a cool head goes a long way.

After this wild chase, we are now back to building the Match Statistics feature. We hope to ship it soon!

Just one last thing that I forgot to share. Many wise developers have said this before me, but I must say it again! Plan at least twice the time you anticipate it will take :-)

If you have a similar story to share, hit reply!

We built data extraction, transformation, and loading pipelines along with the reporting using Sqoop, Pig, Hive, and Cascading. But, as we got along the project, we faced a critical issue. Even when we got all the data in one place, it was extremely challenging to identify the real world entity represented by the varying records from different systems. Since the records did not have any common identifiers and they also had all sorts of variations, unifying the data became the biggest issue for us on that project.

Once I became aware of the problem, I started seeing it in other projects. Even when I was doing different work, the problem kind of never left me. How do we say that the records are a match? How do we scale this out? How do we run it for any entity which can appear for an enterprise? For the programmer in me, it became a thrill ride of problem solving with lots of first principle thinking. The more I worked on it, the more challenges I became aware of. Eventually, I got completely immersed and stopped taking other consulting work to focus on building a generic solution.

When I showed an early version to my friend and mentor Joydeep Sen Sarma, creator of Apache Hive, he encouraged me to open source it. As an active consumer of amazing open source technologies, it was a great way to contribute back. So I latched on to the idea. Here is what I wrote when I published Zingg:

I also feel that open source Zingg is more powerful than closed source, because then it is up to the community to put it to use in more ways than I have seen and could ever anticipate. It is also a promise to work closely with different people and collaborate, something I am very sincerely hoping will happen!

I was convinced there were more people like me, but it was not clear if they would find me. Fast forward to today - the Zingg open source community is 700+ members strong. Turns out what felt like an esoteric problem to me is a strongly felt pain for enterprises. Starting with a single person, Zingg now has a founding team that cares deeply about entity resolution. Zingg resolves billions of records every month and our users and customers come from diverse industries like healthcare, insurance, government sector, non profits as well as startups. This love and support is driving us to innovate more, and we are building amazing stuff to run production loads with a persistent ZINGG_ID along with incremental flows. We continue to work extensively on accuracy and performance. Zingg also supports diverse deployments on Microsoft Fabric, Snowflake, AWS Glue, Databricks, and others.

All this would not have been possible without our community members, well wishers and supporters. Thank you all, you have made this journey well worth it

!

“The Data Weaver’s Tale”
In fields of data, vast and wide,
Where records dance and entities hide,
A weaver works with threads so fine,
To link and match and intertwine.

Deduplication clears the way,
As fuzzy matching comes to play.
Record linkage, strong and true,
Brings scattered pieces into view.

Entity linking, precise and keen,
Connects the dots once unforeseen.
The knowledge graph, a tapestry,
Reveals hidden identity.

With AI’s wisdom as our guide,
We stitch and resolve, side by side.
In this grand quest for truth and light,
Entity resolution shines so bright.

When I open sourced Zingg three years ago, Entity Resolution felt like an esoteric problem. While I had personally experienced the issue of unharmonized records, it was hard to say if I was an outlier or if this was a much more common problem. Sure, master data management systems have existed since the beginning of the data industry. Or even before it. Customer data platforms have also had their share of the spotlight. Yet, for all of the modern data stack maps out there, identity and entity resolution have been largely omitted.

For someone who had burnt their life’s savings in working on Zingg, I was treading unknown territory. The encouragement I had received from people who understood the problem truly meant a lot to me. Yet, the way forward was unclear. Starting and building a company is terribly hard. It was tempting to think of a cushioned job which would save me from a lot of hassles, guarantee a handsome package and possibly some other interesting problems.

However, I could not bring myself to do anything else - entity resolution felt like a problem I could not forget. If I could help a few practitioners like myself who had struggled with entity resolution, I would be satisfied. If I think back, it is likely that there was a builder bias in my thinking. Working on Zingg is fun! The challenges we see are not common place. They force you to think, they push you to the drawing board and scratch your head again and again in the quest to find a solution. It is wonderful to apply tools and techniques learnt over multiple years, yet see oneself falling short and researching and learning more and more. Working on Zingg feels like an endless maze of hard computational puzzles thrown our way. As a professional, one could not ask for more!

Also, somewhere deep down, I was not too worried about the selling bit. I was convinced that it was worth a try. One big reason was that the existing tooling in entity resolution is outdated. MDM and CDP tools aim to do too much, and the core problem that they try to solve - entity resolution, is something Zingg specializes in. Zingg’s identity resolution capabilities make it a drop in replacement for an MDM system and a key component in composable CDP stacks. Zingg can work with large datasets and resolve different kinds of entities with ease. We are privacy first, with no data ever leaving customer premises. Our warehouse/lakehouse native approach enables an elegant data architecture, letting people leverage their existing investment into data tooling and running Zingg AI as part of their data pipeline. Without any specialized AI skills, one can straight off use Zingg results in marketing attribution, risk or personalisation models. Or pipe Zingg resolved entities to the RAG of an LLM model or to a knowledge graph.

I have surely made my share of mistakes in life, but turns out I was right on betting wholeheartedly on Zingg. Adoption by open source users has been motivating. Due to on premise and multicloud environments along with the proliferation of SAAS tools, there is a growing need for trusted and enriched entity data. The race to deploy large language models, enterprise knowledge graphs and RAG based systems is pushing the demand for entity resolution even more. An enterprise data leader recently told me that Customer 360 is among the top 3 problems their team worries about. We are repeatedly hearing single customer view, SCV, Customer 360, C360 and variants in sales conversations. A very healthy and growing revenue from Zingg Enterprise is giving us the lever to go deeper into the problem, innovate even further and build simple yet delightful experiences for end users.

We are onto something special, and it is an experience worth having!

This would not have been possible without some really solid support and I am extremely grateful for that.

A big thank you to each of our users for trying Zingg, mentioning us to friends and coworkers and for joining our Slack. We hope that Zingg will continue to help you in your data journey. Special thanks to our investors and paying customers of Zingg Enterprise. We could not have reached here without you. Thank you for your early confidence and bet on us. We have built a lot of valuable features like ZINGG_ID, incremental flows, and determinstic matching based on customer asks and our understanding of customer needs. The product is so much richer due to your feedback and feature asks.

A lot of ground has been covered in the last 3 years, and a lot more remains. In the coming months, we plan to grow the team to build more cool stuff we have been aching to, and continue to dabble in interesting problems at the intersection of technology and company building. Wish us luck!

Thank you for reading, and if this story resonates with you, do say hello!

From the point of view of data teams in a fast growing business, what really does growth amount to? How does it manifest in the day to day data collection and processing?

If we dissect the key activities happening across businesses during the growth phase, there are a few commonalities that stand out. New customers get acquired at a fast pace. Existing customers buy more products and services. They also buy from additional channels, like a website, if they earlier only bought from the brick and mortar store. The business keeps in touch with existing customers and makes sure it has the ways and means to contact them with offers and product updates. Customers inform the business when they move or change jobs. Strategic mergers and acquisitons happen. Additional procurement of products and services happens to fuel the growth and new suppliers get onboarded. New product lines are introduced and additional customers onboarded.

Thus, data about core business entities - customers, suppliers, products and parts starts changing rapidly. New entity records get added, existing ones get updated. Some of the records are deleted.

A small detour before we go any further. In my last post, I discussed about the cross reference and persistent ZINGG_ID. Let us recap a bit. When we build our customer journeys, segment users and attribute acquisition to campaigns and channels, we need a consistent view of the customer data. We want to associate an identifier that inherently represents the entity. And we want to use this identifier in all our reporting and analytics, as well as operational systems. If a customer gets in touch with us at the call centre, we want to use the ZINGG_ID to get the context of their purchases and their relationship with us.

Coming back on the current topic. There is immense business value in having a persistent and continuously updated identity graph for each of the downstream usecases like marketing attribution, personalisation, life time value, fraud and risk.

However, as we saw earlier, in a growing business, records are no longer static. They are getting added and deleted and updated. So, how does all this change affect identity resolution? New records would start matching with existing ones. Updated records may start matching with other records they did not match with earlier. They may stop matching with records they matched up earlier. Hence clusters of resolved identities are evolving with the data. They are merging or splitting and new ones are created all the time.

Let us dig deeper how that looks by going over some examples.

In the following dataset, the records with ID rec-1020-org-incr and rec-1023-org-incr have been updated during business operations and continue to exhibit strong similarity with their original matches. Two new records with ID rec-10000-org-new got added, matched with no other existing record but with each other.

With me so far? Now, here comes an even more interesting bit! In the following dataset, rec-1022-org and rec-1022-dup-4 had matched originally.

In an update at a later date, rec-1022-dup-4 got altered and these records now stop matching with each other. This usually happens in the case of wrong data capture in an original record.

Thus, when a record is updated yet it is close enough to the original cluster values, we ought to keep the cluster relationship. In the case of adding a new record that does not match any existing ones, a new cluster should get created. When a new record shows up that matches an existing cluster, the new record has to get attached to the cluster. When a new record that matches two clusters is passed, the clusters need to get merged. If we make edits to a record so it no longer matches with its existing cluster and then reverted those edits, the models should correctly track the updated and reversed entry, with the record remaining in the correct cluster throughout the incremental runs. There are myriad possible scenarios of clusters splitting and merging with each other. Matching new and updated against existing records gets extremely complex before we realise.

Technically, we could run a full match with Zingg Community version on the updated dataset and get the new clusters.

But.

If we run the entire matching process again, how do we preserve the persistent ZINGG_ID for the records? How do we tie back the newly generated ZINGG_IDs on the updated set to the original ZINGG_IDs? Especially when a new cluster will now have multiple older ZINGG_IDs, or when a new cluster may have some old different ZINGG_IDs and some fresh ones. How do we choose the surviving ZINGG_ID? On top of that, running full matching loads on growing datasets is costly in terms of time taken and compute.

Here is how we handle this in Zingg Enterprise.

Clusters automatically merge, unmerge and new clusters with ZINGG_IDs are created based on new and updated values while only matching the updated records.

Which means that

The identity graph stays updated even when the data changes.

When I had open sourced Zingg, I felt that it was my life’s best work and wondered if I would ever work on anything more challenging. While building the incremental update, I realized that this functionality is way way tougher to build than everything else I had built so far. Magnitudes tougher than the active learning aspects of Zingg, or the blocking model. There are tons of algorithmic and performance aspects to consider. Even chalking out the test strategy and test data proved to be quite a challenge.

After intensive testing on internally generated datasets, we released the runIncremental phase to our early customers. We knew our test data could not mimic every possible scenario, and though we were confident of the algorithms, we suspected edge cases which we had not envisaged. We requested our customers to test the incremental flows on their data and to report any inconsistencies in matching or cluster assignment. We were thrilled to get an extremely positive response!

Here is an excerpt from one of their test reports:

Each test scenario was carefully constructed to challenge the model and verify the expected behavior. The results were consistently positive, demonstrating the model's precision and adaptability in every test case. Its ability to accurately handle minor data modifications and correctly process entirely new, atypical entries was particularly noteworthy. Equally impressive was its capability to manage entries that aligned with several clusters without compromising accuracy and its efficiency in reverting changes to their original form.

The success of these tests is an essential step on the way to releasing the incremental Zingg models in production.

We are super proud to have come so far, and happy to roll it to all our customers. If the idea of an identity graph on your own data within your own data lake and warehouse appeals to you, why not hit reply and let us grab a coffee? :-)

Imagine we have multiple records of a customer spread across ticketing, billing, offline stores, and web properties. This is a usual scenario we witness everyday in our data land. Different sources of data come with multiple versions of truth. When they land centrally in a warehouse or datalake, its hard to figure things out. Revenue attribution and lifetime value can not be calculated accurately. Nor can we estimate churn or calculate confidently how many new customers were added last quarter. Increasing revenue through personalized offers and exploring opportunities to cross sell and upsell will not be possible either. Let alone anti money laundering or risk. Visualize a multimillion-dollar investment in a GenAI customer service agent. How far can it go using this fragmented data?

To establish the right data foundation, analyze and report on business critical metrics, and productionize AI and ML technology, we will need to unify such records and build a trusted single source of truth for the customer.

Enter entity resolution. Variations in attributes can all be resolved, and clusters of matching records can be created. The Zingg Community Version offers robust clustering and fuzzy matching, which bring these records together with a unique Z_CLUSTER field. Matching records share the same Z_CLUSTER. Voila, there is a way to say who is who!

What happens when we want to track the customer over time? Or build a holistic view? Establish their journey and reference them or search for them in different enterprise systems. When there is no identifier to refer to the customer, we need a way to associate them with an identifier. Kind of like an SSN or a passport number. Or a primary key in the database.

The Z_CLUSTER is extremely powerful and useful. But it is not persistent. There is no guarantee that it will not change in the next Zingg run. If we stream our resolved records into our source systems and want to use Z_CLUSTER as the primary key, it would break.

As we saw different Zingg deployments, this was an ask that we heard again and again. A globally unique persistent identifier that acts like a reference key to the entity and all its records. The same key even when the record changes. The unambigious way to refer to an entity across multiple enterprise data systems, operational processes, and workflows. Even as new records come in and earlier ones get updated.

This is exactly the ZINGG_ID in the Zingg Enterprise version. Resolved, unified records across the entire enterprise which can be referenced through a persistent key. A help desk agent could use it to look up the entire customer interaction journey. A GPT based chatbot operating on patient data would work seamlessly with all the information at its disposal.

It's been fun solving this complex piece of the puzzle, and it's extremely satisfying to witness the value it generates for Zingg Enterprise customers.

Sounds interesting, or something you care about? Do get in touch!

In the entity resolution space, we refer to exact matching on attributes as deterministic matching. If the application systems are consistently collecting and persisting trusted and known identifiers like Social Security Number, deterministic matching can help to resolve such records and help us identify the true individual.

In most real world systems, data in different systems in an organisation has different attributes. Let us imagine a retail store with both online and offline presence. A web support system would capture the email correctly while also logging name, a call centre would mostly have a correct phone number but other details may not be as reliable. The offline store would have some or all of the details, as accurately as the human behind the computer tasked with entering them can type it in. Thus some of the individual records have identifiers, like phone and email in the above case, which may match exactly, while many of the records have names and addresses with all kinds of variations, typos, abbreviations etc. Assessing the extent of such variations for an entity and reconciling the fuzzy and inexact matches is known as Probabilistic Matching.

At a simplistic level, one can think of deterministic matching as a database join on trusted id columns like passport number or email. Probabilistic Matching involves statistical and ML based techniques to determine if a given record pair is indeed the same real world entity. If we search these terms, there are multiple articles around Deterministic Vs Probabilistic Matching, pitting one against the other. Since the approaches to solve deterministic matching are very different from the ones for probabilistic matching, these techniques are often viewed as contrary.

However, we have a different view.

As we dived deeper and looked at multiple datasets, we saw that in most cases, data contained multiple signals, some with trusted identifiers and most times without. Even when there are identifiers, they tend to be more than one. For example one system could have driver’s license id mandatory and passport number optional. While another system could have the mandatory rule reversed. A third system may have both of these ids as optional. If we match deterministically on passport and license in this case, we would only resolve some entities, not all. Worse, a person who has provided license id in the third system would only match with the first, and the record in the second system would go unmatched.

This has serious ramifications. In an ecommerce store, the inability to reconcile effectively would jeopardize the million dollar investment in recommender systems that consume the customer preferences. In an anti money laundering system, the ability to catch fraud is directly proportional to the quality of the matching. An insurer can not understand risk at a household or person level if records remain unresolved.

Thus, we need to solve for both determinstic and probabilistic matching at the same time. An either/or approach with respect to deterministic and probabilistic matching would lead to identifying records with identifiers as a different entity than those without. Why not use every single data point of the record? There is tremendous value to be unlocked by looking at the problem holistically. If a deterministic match can be found, use that. If not, use the other attributes. Essentially, weave them all together holistically to ensure that the probabilistic and deterministic matching records are resolved to a single entity.

The above is a simplistic take and there is a lot of software engineering involved under the hood to build this out. Effort nothwithstanding, we are convinced that it is the right approach. In fact we have gone all out while building this feature, with support for multiple combinations of attributes to make sure no record which has a match remains unmatched. Determistic and probabilistic matching are better together, and early Zingg Enterprise customers see an immediate jump in match accuracy.

Is this something you care about too? Hit a reply and lets spark a conversation!

First thing first.

We now have paying customers for our enterprise products! There is nothing more exhilarating than seeing our life’s work come alive. And to be paid so that we can continue to build and move in the direction we aspire to. In the startup world, we hold a lot of discussions around product market fit, product founder fit, building things people want, user research, and other associated activities. Product-led startups like ours get the first set of customers through the urgent need of a visionary early adopter. The visionary early adopter is the person in a team who has a pressing need along with the foresight to solve the problem and the confidence to trust a new team and an innovative product. And who can influence and move budgets to solve that need. I am very glad to have this opportunity to work closely with this first set of customers who are helping us shape the product. Not just the features, but also the pricing.

Thank you for supporting us so that we can help you!

We are building three flavors of Zingg Enterprise. Zingg Snowflake, Zingg Databricks and Zingg Spark. All of them with the same roots in Zingg open source. Each carries forward Zingg OSS and gives the required ammunition to build and use the persistent and evolving enterprise identity graph. Zingg Enterprise creates global unique identifiers for entities, explains model behaviour and maintains and enhances the identity graph through resolution of new and updated records against previously matched records. Of course there is the improved accuracy and scalability :-)

On top of this, we have added platform-specific features. The Snowflake version runs inside Snowflake without any other infrastructure or software dependence. The Databricks flavour adds Unity Catalog support and a lot of other Databricks goodies. The Spark one has the enterprise features and runs on all other Spark environments like EMR.

On the open source side, we just hit 1000 downloads per month on PyPi! The rate of downloads has doubled over the past few months, and we now have a Slack group touching 450 people, from varied organizations like LinkedIn, Maersk, Newfront, Samba TV, Canadian Football League, Heb and John Deere to name a few. We have federal agencies like Pandemic Response Centre, and established startups like Provenance.org, Redica Systems, Ninjacart and Cherre. We have a good number of nonprofits and data consultancies as well as friends from Databricks, Snowflake and Exasol. The strong code contributions from these partner companies have helped us to support their users more deeply.

We are clearly onto something :-)

I am truly amazed by the depth of use cases we are seeing across big and small organizations in varying domains. Zingg is resolving customer identity for marketing and personalization use cases in retail. It is resolving patient identity for comprehensive health, and healthcare provider identity to understand drug recommendation patterns. And the identity of an insurance policyholder, for compliance and risk, and also from the point of view of a reinsurer. Identity resolution is much broader than we thought initially. It is the identity of members of religious establishments, of donors or recipients. It is the identity of the fans of a football club to send out the right information at the right time for a game of their liking. B2B accounts, which buy from multiple subsidiaries in different geographies, need identity resolution as well. So do products on an e-commerce website for competitive analysis on other sites.

Identity is anything and everything a business cares about.

We are happy that we built Zingg so that users can work on their schema and train on their data without specialized ML skills. All in a privacy centric way. This has allowed us to see Zingg in action in the different use cases we are discovering every passing day.

To each of you who has used Zingg, mentioned us to friends and coworkers or checked us out - thank you for your time and we hope we have removed some data troubles from your life.

I also wanted to thank our investors and advisors, who backed Zingg when no one had heard of us, and who have been generous with their time and support.

These two years have been a tremendous learning experience, and our plans are only getting bigger. There is a lot of AI yet to be applied to enterprise data, and we will love to create more value for our users and customers. For that, we need to continue to work closely with them, ship more and more often, keep the momentum going and work twice as hard.

We are excited for the path ahead. Wish us luck!

To begin with, we had a 500k test record set running on the x-small size Snowflake warehouse in roughly 12 hours. Though the code was not tuned for Snowflake performance, the job ran successfully to completion without hiccups and we used this as our baseline. Entity Resolution is a tough problem, and harder to do at scale. We want to get maximum mileage out of Snowflake and ensure Zingg ran efficiently for our customers. Our philosophy is that the customer should not have to pay the computing bill for any inefficient code. Hence we always aim to extract maximum performance without throwing more hardware at the problem. Unless we absolutely have to. We also had a target dataset of roughly 2.5 million records, 5 times the baseline. This was a real customer dataset, with non-uniform distribution across fields and with triple the columns of our test set. When we do entity resolution, a 5-fold increase in the input with the same number of columns and match types leads to a 25-fold increase in the comparison complexity. As we have robust index learning based on training data in Zingg along with tons of performance optimizations, we expected the matching job to be somewhere close to 18 hours, which we were planning to tune. However, when we ran this dataset on the x-small Snowflake warehouse, we started seeing errors on session timeout intermittently after roughly 8 hours.

com.snowflake.snowpark.SnowparkClientException: Error Code: 0408, Error message: Your Snowpark session has expired. You must recreate your session.
Authentication token has expired. The user must authenticate again.

Running the Snowpark client as a background process or with nohup helped, and we were able to see the job continue to run beyond 24 hours. Surely this performance was not something we wanted to ship. So we killed the run after identifying the first set of bottlenecks. Our approach was to zero in on the hotspots and solve them surgically in the order of highest impact first. The highest impact was mostly about the functionality taking maximum time, but in some cases, we also fixed stuff that was earlier in the flow but slow enough to not let us go past a particular stage in a reasonable time. We used the query profiler by Snowflake extensively for understanding and improving the flow, along with some traces and timing logs in our own code base. The query profiler gives a lot of valuable statistics about the query plan, function timing, and which part of the code triggered the query. It also provides partition-level and row-level information, so it was our go-to source for hotspot identification as well as figuring out if our changes worked as expected. Here is a snapshot of how it looks.

At a high level, here is how the Zingg flow is structured:

Input Data → Index → Join On Index to get Approximate Similar Pairs → Pairwise Similarity Features → Predict Matches → Graph Processing

We learn an index on the input data to create approximately similar pairs. We then compute similarity features on these pairs and predict if the pair is a match or not. This output is clustered using graph algorithms to convert matched pairs to clusters.

Our logs indicated that the biggest bottleneck was the Predict Match flow. The. predict flow leveraged a Snowpark user-defined table function to return two columns. The first was if a given pair was a match or not. The second was the score indicating the similarity of records in the pair with each other. The higher the score, the better the match. Since we returned more than one value from this function, we built it as a UDTF instead of a UDF. However, this UDTF was consuming the bulk of our processing time. Given that we were not really creating multiple rows per input row but returning multiple values per row, all the lateral joins we saw on the query plan did not make sense. It made a worthwhile experiment to convert the UDTF to a user defined function, concatenating the prediction and the score in one value and extracting them out through another UDF. This worked like a charm and immediately cut down the processing on our baseline dataset from 12 hours to 5 and a half hours. More than a 2x improvement! This was a massive gain, but we clearly had more work to do.

We then set out to optimize the queries generated by Snowflake through the Snowpark code we had written. The bottlenecks here were mostly while indexing and generating the near-similar pairs. We saw that if we did computations on a Snowpark Dataframe to build the index and then self-joined the Dataframe on the index, the index computations were repeated. This was counterintuitive to us, as we were using the same instance of the Dataframe. We tried to clone the Dataframe as advised but that did not work. Finally, when we cached the Dataframe, the query plan was altered and our costly computations happened only once. The lesson here was that if the computation is costly and the Dataframe has to be used in a self-join, it is wise to cache the intermediate results. However, this has to be used with caution, as there is an overhead in temporary table creation and disk spillage.

Another optimization we did was on the range scan. We join the records on the computed index to build the pairs. To avoid pairs like Record A, Record B repeat as Record B, Record A, we filter out the pairs on ids. This is critical as the subsequent similarity feature processing is expensive. However, we saw that the join became a cartesian join with filtering pushed out to much later in the flow. This led to slow performance along with a suboptimal flow, making similarity feature computations that we did not need. We cached the Dataframe arising out of the join and changed the id1 > id2 range query to id1-id2 > 0, which pushed the filter at the right junction for our flow as well as changed the earlier cartesian join to a normal join.

Together, the above changes shaved off roughly 20% of runtime on our baseline dataset, and we were now down to 4 hours and 30 minutes of the entity resolution workflow on 500,000 records.

With these major fixes in, we profiled the run again, closely looking through the hotspots, and to our utter dismay, found that the predict function was still the most time-consuming part of the entity resolution workflow. This is a Python UDF that uses machine learning for the prediction. A pre-built model learnt from the training data is passed as an argument to the function along with multiple similarity features of the pair. This UDF predicts whether the pair is a match or not along with their similarity score. Since the features are different for different datasets, we do not know beforehand the shape of the feature columns. Hence we construct an array over the multiple features so that we can resolve different types of entities. In a good bad way, there was no code change we could do to optimize this function. Our code was concise and to the point, and there wasn’t anything to shave off here. So we focused on the Snowflake invocation of this UDF. We learned that Snowflake advocates vectorizing UDFs, especially if they involve matrices and other machine learning workloads. The vectorized UDFs are passed batches of rows, and return batches with results. This made immediate sense as invoking the UDF row by row was surely not going to help with the inherent matrix manipulations. Though we wanted to do this right away, there were quite a few challenges.

It was difficult for us to define the function signature as the column number and types were dynamic. Earlier we used to pass an array, so it had worked. After some thought, we were able to build the signature and define the function on the fly using the metadata we already had while creating the similarity features. Passing the model became tricky however and we were not able to deserialize it in the function no matter what we tried. It seemed there were some extra bytes added when the model was passed as a column received by the UDF as part of a Pandas Dataframe. The same code had been working earlier row by row, so the conversion of Snowframe Dataframe rows to Pandas Dataframe internally seemed to be the likely cause. There was another complexity. We were invoking the Python function through Java Snowpark API, so there were too many layers to unravel here. We went over the open-source code of Snowpark, we tried to add function logging and tracing, we tried to debug calling the UDF from Python, and we read all that we could..we pretty much did anything and everything we could think of!

After struggling for a few anxious days, we changed our approach and began persisting the model to a Snowflake stage. Our UDF would import that location and read and use the model. This finally solved the serde issue we were seeing earlier, along with reducing the data sent to the function. In hindsight, we should have tried this approach first, but at that time the thought was to only change what was absolutely necessary. The code for passing the model had been working for a while, and while tuning performance, it is important to control the changes. So we weren’t too wrong either.

It got a bit late in the day when we ran the build with these changes, and when the test dataset ran in half an hour, we were sure something was crazily wrong. Maybe we had left the bulk of our processing code commented out in the run by mistake? We saw the output table, the results were there, and the predictions looked accurate. It was hard to believe. Too weary to think further, we called it a day.

The next day we double-checked the output and everything was in order. We cross-checked. Yes, it looks fine. No, we are not sure. Not yet. Let us run it once more. Boom, the job was finished by the time we finished our daily catchup + coffee.

28 minutes. From 12 hours. From half a day to half an hour! 24x.

On an x-small warehouse resolving identities across multiple columns for 500,000 rows. Wow! High Five! Now we need a break. Mission Impossible Dead Reckoning, here we come!

It was now time to run the customer data set in their warehouse. Brimming with confidence, we fire the job and monitor it from time to time. Umm..it is not working. The predict function is still a bottleneck. For some reason, the batch size Snowflake has automatically chosen on this dataset is 10, while it was sending 1000 rows at a time to the function on our test data. Possibly because there are way too many columns here compared to our test data? The batch size hint is just a hint, but that seems to be the only ammunition we have right now. Will it work? So we modify the UDF and add the max batch size annotation. This has the desired effect, and we have some great results for our customers.

There were a few other changes we had to make to get here. For example, the Dataframe.show() function would cause an out of memory error on the client machine if the data was big. While working on performance optimization, it is critical that functional parts of the code work flawlessly. We had used show() liberally in our code to verify the logic, assuming that it would limit the records sent to the Snowpark client by limiting rows in the warehouse itself. But show() works differently. It fetches everything to the client and then truncates to 10 rows. This was a bit counterintuitive, but an easy fix.

Overall we are super excited about the gains we have made and so happy that we have come this far. We hope the lessons above will help folks who are building on Snowflake and Snowpark. Feel free to talk to us if you are struggling with Snowpark/Snowflake performance issues, we would love to help and share our learnings! And if you know someone who needs entity resolution on Snowflake, please do send them our way :-)

Ciao!

I have very exciting news to share - we have signed our first customer for Zingg Enterprise Snowflake! It feels magical, almost surreal. I call it magical not because I did not believe it will happen, but because of the pace and ease of closing the deal. If you have been doing B2B enterprise sales, you probably know what I mean :-)

Zingg open source has always supported Snowflake as a data source using Apache Spark as the connector. As we talked with our users, we realized that many of them had only SQL or warehouse workloads, and we could really simplify their stack by using Snowflake directly. Over the last year, we watched Snowpark evolve and did multiple experiments to see how viable it would be to build Zingg over Snowpark. This post will detail some of the things we did and learned as part of the process.

Our first challenge was building our extensive ML pipeline for entity resolution in Snowpark. We have our own blocking algorithm but rely on Spark’s MLLib for classification. Snowpark does not have a built in machine learning library but facilitates ML through third party packages like scikit-learn. This means that Snowpark Dataframes can not be passed directly for training models, but a conversion to Pandas Dataframe becomes necessary. This has two issues. Firstly, this limits the size of data one can train on. Fortunately for us, as we use active learning to build training sets efficiently, our training data size is quite small. The second issue was the real one for us. Our code is in Java and Scala so introducing a Python flow for training and prediction became challenging. Zingg does substantial preprocessing and feature engineering enriching the input data with various signals before training and prediction. Using Python meant our in-memory enriched Dataframe could not be passed directly for training. Eventually, we decided to persist the features into a temporary table, and built a stored procedure in Python to read it to train a classifier. We persist the classifier into a Snowflake table. and load it in a named Python user defined function, which is invoked from our codebase during inference. Thankfully, a named user defined function in one language can be easily invoked from another, which is the strategy we have used extensively in our code.

Snowpark also misses a graph library, like GraphX or Graphframes in Apache Spark. Luckily the Snowflake team helped us here and provided us with a Scala library for graph processing. We had some work cut out to integrate this into our code, especially since the Snowpark Scala Dataframes are not interchangeable with the Snowpark Java ones.

Building a cross-domain ML product introduces a great deal of generalization in the flow. We do not know beforehand the shape of the input dataset - it could be product matching, it could be customer matching, or it could be a B2B account with domain names and other details. Many row-wise operations become tricky without a map equivalent in Snowpark. Hence we built our own version of map, passing all the columns into a VARIANT type array and then getting individual values as necessary. This was a bit of a tough problem, and we were pretty proud when we could get it to work.

There were some other minor hiccups like temporary udfs not getting cleaned up, no way to access data from dataframes using column names but having to pass column index, etc. I am sure these will get sorted out as the Snowpark API matures further.

One of our biggest concerns was that our codebase assumed Apache Spark as the defacto processing engine, meaning that the dependencies were all over the code. Apache Spark is an integral part of Zingg, and is intricately weaved into all Zingg operations. So our first thought was to have two code bases, one for Apache Spark and then the same logic ported for Snowpark. All functionality would exist independently in each. There would be some common code like reading user configuration that would be shared between the two. As we were planning the Snowflake version of the product to be closed source, the code would anyways have to be in a different repository than the open source ones. Hence, why not? This would also let us build the product in less time, as the open source is extensively validated and already in production. Ship faster, they say! We spent a day or two doing global search and replace operations on dependencies and imports, to see how this would shape out and realized soon enough that this was going to be a mess. If we had to fix a bug, let alone add a new feature, we would have to do it in two places. Clearly, this was going to be a recipe for a lot of overwork and technical debt and I am glad we abandoned this soon enough.

The next approach was to build up generic classes and then templatize them with Spark or Snowpark equivalents. All the logic remains centrally located in one place, and the processing engine equivalents have their own extensions. We can easily add another processing engine if a new Dataframe API comes up. This meant touching every single bit of our code, which was going to be a lot of work on both the development and the testing fronts. Not a decision we wanted to jump into, but there wasn’t a lot of choice here. I have to let you in a secret here. Besides the user discussions, we had another motive for the Snowflake product. We wanted to enter the Snowflake challenge. Some cash and publicity can never hurt a frugal startup! We wanted to apply with a running application, which left us with roughly 1.5 months to get everything running, and that is the deadline we set for ourselves at the beginning of the year. It was a lot of work, and more work, and more work. It was tiring, but it was fun. We finished a week before our self-induced deadline, without any shortcuts or hacks :-)

Snowpark has been recently open-sourced, so it is going to be even more fun building around it. Zingg is surely one of the more complex applications being built around it, and we can’t wait to add more stuff to our identity resolution product and push Snowpark further.

In the end, we did not win the challenge, but had some nice interactions with the Snowflake team and have learned a lot of stuff that will help us as we now put Zingg Identity Resolution on Snowflake into production.

We are really excited about how this journey is shaping up!

What good is a song composed but not hummed by the audience? The musician may love it as their best song, but it will soon fade away from the memory of its listeners. While the pleasure of building and innovating is its own reward, creators need external validation to generate the ammunition to make even bolder bets.

As people who pour our hearts and souls into building Zingg, we too take a pause at times to reflect if we are indeed creating value for our users. There are two main things we are super curious to learn at this stage.

1. To learn how people are discovering us

2. To understand how they are using Zingg

Signals for these come from multiple sources. The important ones are direct user and customer communication, Slack, website traffic, product downloads and product usage statistics.

Direct interaction with users and customers is the richest form of learning. Many of our open source users join our slack and chat with us and we learn about the problems they are solving with Zingg. Some of them write to us on email, or DM us on Slack or LinkedIn, sharing what they are building with and around Zingg. They want to check the best way to use Zingg for their scenario. We ask them about their stack and use cases, and how we can help them better. Learning more about our users guides us in shaping the roadmap for Zingg. Zingg’s docker and the Zingg Python API are a direct result of these interactions. We built them because we realized that they will help users generate identity resolution results faster and will enable them to use Zingg as a first class citizen in their transformation pipelines.

While interacting directly, we often ask users where they learnt about Zingg. This is augmented with the signals from traffic information from Google Analytics. Github also provides repository traffic analysis of the previous fortnight.

Often, the discovery is word of mouth. For us, having a great product and providing an even better support is the best form of engaging people. It suits us as developers, it helps our users drive value faster and it builds a strong relationship with the user. A check at the testimonials on the Zingg website reveals we are doing a nice job there.

And we should still aim to get better here. Like identity resolution, support is never going to be perfect, so we always have things to improve upon.

Many times the discovery is through the tutorials and guides we have written. Or talks and events in select data communities. Instead of a lot of content, we have focused on deep and long articles that help the user understand the problem of identity and entity resolution, why it is needed, the challenges in resolving entities and how Zingg can be run on different platforms like Databricks and Snowflake. Early adopters of a product know the problem and are looking for a solution. Hence, we owe it to them to make using Zingg easy so that they can focus on solving the bigger puzzle - Customer 360, Segmentation, Fraud, Risk, Compliance etc. Discoverability is the side benefit of our content, helping our users adopt Zingg is the main goal.

Broadly, these data points are sufficient for us to understand and improve discoverability at this stage. Besides this, we track open source adoption and product usage anonymously. We gather information about

1. Github downloads of our releases

2. Docker pulls

3. PyPi installs

4. Slack users

5. Documentation views

Together, the above give us a fair idea of our adoption and how we are growing. For now, we get these values manually every couple of weeks and track them in an excel file, charting out a time series to understand our growth rate. We do have a column for github stars which we populate but dont think much about as we have richer data to get signals from.

Coming to product analytics, the aim here is to learn the different aspects of how people are using Zingg-

1. Which platform do they run Zingg on?

2. Is the product scaling? How many records do they match?

3. How much time does it take to run?

4. Do they run Zingg multiple times or is it a one time deduplication?

For any product analytics to function, we ought to have a way to identify unique installations and trigger usage events from the installation to a place where we can analyze them. Our asks are fairly simple questions, but the novel nature of our product as well as deployment makes it tricky to get the answers.

Often, Zingg is deployed on cloud resources like Databricks or EMR or on private enterprise networks which are blocked from sending information outside the VPN. So if a user downloads and uses Zingg under such environments, we have no way of knowing how they are using Zingg.

Another issue is that it is impossible to identify installs through cloud systems, as virtual machines change with every run. Even when we don’t want to identify the user but only the install, there is no way to track a unique installation by writing a random uuid on disk. This problem exists on our docker environment as well.

Some of the features we are building now will allow users to trigger cloud jobs from their machines, so we will think about the install identification then. Till then, we already have the download stats, so thats a good enough proxy for us. Instead, we gather usage data. We respect user privacy, so we limit data collection to reflect product usage. We also provide a flag to turn data collection off.

We use Google Analytics as our event server and post custom events to it from the code every time Zingg is run. These events are sent to our GA account, which has been configured to pipe these events into BigQuery. For every run of Zingg, we collect the data and output formats, time it took to run, number of records and a few other stats. Every few weeks, a handful of SQL queries help us understand the data volumes we are processing, the number of daily jobs run and their trends, the time taken to complete each run etc. Mostly, it is a brief glance on the numbers to make sure we are on the right track.

Many times, the usage data has an unheard story to tell. We discovered that there were a good amount of users using Zingg to resolve entities on Snowflake. A Spark deployment is not ideal for them, as they are not using it anywhere in their stack. Could we leverage Snowpark and build a lighter, tighter product for such users? Seems we can, and I have a lot of exciting stories to tell about that effort and where it is leading us!

Coming back to our simplistic data stack.

Sometimes we do detective work ;-) If we see events of the same format and number of fields triggered at regular time intervals through the day, we treat it as Zingg running in production. Once in a while, it is fun to look into individual events and extrapolate the usage to different scenarios.

Churning out aggregate stats is a revelation too - like Zingg has resolved at least 2 billion records in the last two months. This is a huge number, and it has grown rapidly over the last few months.

When I say “at least”, it is with the awareness that we are not collecting all the data points. We have talked with many users who are running production loads but they do not show up in our telemetry. So our analysis tends to be conservative. Which is absolutely fine.

Overall, the aim of this stack is to help us get a sense of how we are doing, what we can do to help our users and what more we should build. We are on free plans on Slack as well as GA. The frugality is in the cost, and also in the implementation. As a small team, we needed something that is easy to setup and will answer our basic questions without much effort. Also, the trend in the data is more important than the individual data points at this stage.

We may miss some dots, but the path is clear. Which is what matters!

It feels great to complete an entire year! Days have passed fairly quickly, spread between writing, lots of reading, speaking, giving demos, and meeting people. We have done a lot of development work as well as spent time on incorporation and funding. I have thoroughly enjoyed interacting with users and learning about the different use cases enabled by identity resolution. We have more than 200 members on our Slack now, with representation from Fortune 500s, digital natives, data platforms, consulting companies and startups. Besides the Customer 360, personalisation, AML, KYC, GDPR, master data management use cases, the trend towards composable customer data platforms is fuelling a lot of interest in Zingg which is very promising. The data stack has proven patterns for data ingestion, transformation and activation, but identity is a core missing piece there. Zingg fits in very well into this space.

One of the fundamental issues with identity resolution is scalability. While building Zingg, we used example datasets and tested our algorithms on them. To test scalability, we copy pasted an example multiple times to build a dummy dataset of roughly 15 million rows. Most enterprise master data is sub 10m rows. Zingg scaled well on this. However, dummy datasets with the same rows perform differently from actual data with real distribution of values. So there were some lingering doubts in the back of my mind. These were soon resolved when we consulted with a Fortune 10 having 12.5m records where Zingg ran like a breeze. Hence, I was confident of scalability while releasing the open source, but I wasn’t sure how far it would go beyond 15m records without tuning the knobs. Hence, it is extremely uplifting to see our users resolve identities on 70-80m records in a few hours with Zingg. Due to the quadratic nature of the comparisons, this is at least 25 times more complex than our 15m endeavours, and seeing it work well is a great confidence boost! Another measure for Zingg is the accuracy of predictions, and users have done head to head comparisons with their multi-million dollar MDM tools to confirm our results. Yay!

One very important aspect for us is arming our users with the power of machine learning and distributed systems in a self-serve, no-code way. Many of our users are first timers with Java, Spark or ML. Zingg abstracts away the technical complexity, and gives the superpower of identity resolution at scale directly to the end user. Now that our Python release is out, we are actively working on a native Snowflake implementation. This opens up a new way for Snowflake users to use Zingg and identify their customers and personalize their journeys and experiences.

There have been some mistakes last year, and I plan to do better in the coming year. Due to my flow state, I missed applying to some conferences but the one I am most disappointed about missing is Coalesce from dbt Labs. I would have met some amazing data practitioners there and made many friends. This time I will clear my calendar and attend virtually. If you are joining, do let me know.

There have been some other disappointments, but they were in the making since some time. Can’t complain.

On a philosophical note, identity resolution is the discovery of self as well as building relationships. Working on Zingg enables me to learn a lot of things around communication, positioning, customer needs, technology, hiring and sales. It pushes my boundaries and makes me discover myself. It also helps me talk to a lot of people and build relationships with customers, data leaders, practitioners, evangelists, contributors, team, investors and mentors. Many of us feel lost at the thought of networking - I am not sure how I would do if I had the charter to meet x new people every week! But when the baseline is common ground on data or ML, it is much easier to connect.

Zingg is made with meraki. It has been a lot of hard work, but it doesn’t feel that way at all. Last year has been nice, and we are having a good time. This coming year will decide how things shape up for us as a project and a startup.

Thanks to all of you for investing your time and effort on Zingg. We are incredibly grateful and motivated to do even more this year.

Wish us luck!

As you might have guessed by now, we just released a new version of Zingg! This release marks a complete shift from the command line training and matching jobs to a full programmatic interface of building applications with Zingg. With Python.

After Zingg became open source, we found most of our early users came from a non-Java, non-Spark background. Though it was extremely encouraging to see a broader user base for Zingg than originally anticipated, it had been a poor assumption at our end that everyone had the JVM running on their laptops to run Zingg. Hats off to the early adopters who still went ahead and resolved their entities with Zingg, we had definitely not made working with Zingg easy for them :-(

This realization led to our urgent Docker release, and it felt that we had crossed a big hurdle! Life and work moved on, and as interactions with users increased, one common pattern emerged - the use of Python. A lot of our users are writing Python data pipelines to get data from multiple sources. They are wrangling data in Python before putting it through Zingg. They are consuming the Zingg output results in Python based machine learning pipelines for risk modeling, customer segmentation, and product recommendations. They are on Databricks notebooks. Wouldn’t it be cool if such users could use Zingg through Python itself?

With this in mind, we set about defining a Python interface for Zingg. Our biggest challenge with Python was that the majority of our source code and all our core algorithms are in Java/Scala. Due to the computationally intensive nature of the identity resolution problem, we keep a very tight and performance-focused implementation. Rewriting everything would not make sense, though we did think about it for a fleeting moment. After some tension, we realized that Apache Spark was structured similarly, and realized we could borrow the best practices from there. This turned out to be a really good decision. In fact, we referred to the Apache Spark codebase a lot to build a Py4J-based interface to Zingg. We would invoke our Java code from Python, all the Java interfaces would be replicated on the Python side. Things started to look up again.

There was one more hurdle though. Something we realized much later as we got closer to shipping - Python packaging. Unlike a usual Python application, our interface is dependent on the compiled Java code and hence we had to figure out a way to include this as part of our package. Luckily, Spark is doing something similar so the packaging there became our big inspiration. Again!

With these major hurdles crossed, we got to the grind and were finally ready to put our work out. Honestly, this release has been a real big one - not only did we build the whole Python interface, but we have also added lots of cool things to core Zingg. Stopwords - stuff to ignore while matching attributes for example. Or special match types for emails or pin codes. More diagnostic messages while running Zingg - getting better there slowly and steadily!

In the end, once the release was done, it felt like a triumph! A mini celebration, and having a different version of fun - rest, recharge and laze around a bit. I haven’t heard any user complaints so far so it is probably working as planned ;-)

Among other things, I am very excited about the ability to finally have a good integration point with dbt. Besides Python, SQL through dbt is something we frequently see (of course! :-)). With dbt supporting Python, we have a cool new way for Zingg to work with dbt. I can’t wait to try this out….or maybe someone is working on this already! Are you The One?