Celebrating 3 Years of Open Source Entity Resolution!
Celebrating 3 Years of Open Source Entity Resolution!
Miles travelled, and miles to go...
“The Data Weaver’s Tale”
In fields of data, vast and wide,
Where records dance and entities hide,
A weaver works with threads so fine,
To link and match and intertwine.
Deduplication clears the way,
As fuzzy matching comes to play.
Record linkage, strong and true,
Brings scattered pieces into view.
Entity linking, precise and keen,
Connects the dots once unforeseen.
The knowledge graph, a tapestry,
Reveals hidden identity.
With AI’s wisdom as our guide,
We stitch and resolve, side by side.
In this grand quest for truth and light,
Entity resolution shines so bright.
When I open sourced Zingg three years ago, Entity Resolution felt like an esoteric problem. While I had personally experienced the issue of unharmonized records, it was hard to say if I was an outlier or if this was a much more common problem. Sure, master data management systems have existed since the beginning of the data industry. Or even before it. Customer data platforms have also had their share of the spotlight. Yet, for all of the modern data stack maps out there, identity and entity resolution have been largely omitted.
For someone who had burnt their life’s savings in working on Zingg, I was treading unknown territory. The encouragement I had received from people who understood the problem truly meant a lot to me. Yet, the way forward was unclear. Starting and building a company is terribly hard. It was tempting to think of a cushioned job which would save me from a lot of hassles, guarantee a handsome package and possibly some other interesting problems.
However, I could not bring myself to do anything else - entity resolution felt like a problem I could not forget. If I could help a few practitioners like myself who had struggled with entity resolution, I would be satisfied. If I think back, it is likely that there was a builder bias in my thinking. Working on Zingg is fun! The challenges we see are not common place. They force you to think, they push you to the drawing board and scratch your head again and again in the quest to find a solution. It is wonderful to apply tools and techniques learnt over multiple years, yet see oneself falling short and researching and learning more and more. Working on Zingg feels like an endless maze of hard computational puzzles thrown our way. As a professional, one could not ask for more!
Also, somewhere deep down, I was not too worried about the selling bit. I was convinced that it was worth a try. One big reason was that the existing tooling in entity resolution is outdated. MDM and CDP tools aim to do too much, and the core problem that they try to solve - entity resolution, is something Zingg specializes in. Zingg’s identity resolution capabilities make it a drop in replacement for an MDM system and a key component in composable CDP stacks. Zingg can work with large datasets and resolve different kinds of entities with ease. We are privacy first, with no data ever leaving customer premises. Our warehouse/lakehouse native approach enables an elegant data architecture, letting people leverage their existing investment into data tooling and running Zingg AI as part of their data pipeline. Without any specialized AI skills, one can straight off use Zingg results in marketing attribution, risk or personalisation models. Or pipe Zingg resolved entities to the RAG of an LLM model or to a knowledge graph.
I have surely made my share of mistakes in life, but turns out I was right on betting wholeheartedly on Zingg. Adoption by open source users has been motivating. Due to on premise and multicloud environments along with the proliferation of SAAS tools, there is a growing need for trusted and enriched entity data. The race to deploy large language models, enterprise knowledge graphs and RAG based systems is pushing the demand for entity resolution even more. An enterprise data leader recently told me that Customer 360 is among the top 3 problems their team worries about. We are repeatedly hearing single customer view, SCV, Customer 360, C360 and variants in sales conversations. A very healthy and growing revenue from Zingg Enterprise is giving us the lever to go deeper into the problem, innovate even further and build simple yet delightful experiences for end users.
We are onto something special, and it is an experience worth having!
This would not have been possible without some really solid support and I am extremely grateful for that.
A big thank you to each of our users for trying Zingg, mentioning us to friends and coworkers and for joining our Slack. We hope that Zingg will continue to help you in your data journey. Special thanks to our investors and paying customers of Zingg Enterprise. We could not have reached here without you. Thank you for your early confidence and bet on us. We have built a lot of valuable features like ZINGG_ID, incremental flows, and determinstic matching based on customer asks and our understanding of customer needs. The product is so much richer due to your feedback and feature asks.
A lot of ground has been covered in the last 3 years, and a lot more remains. In the coming months, we plan to grow the team to build more cool stuff we have been aching to, and continue to dabble in interesting problems at the intersection of technology and company building. Wish us luck!
Thank you for reading, and if this story resonates with you, do say hello!