History of the Data Vault Engine & Studio
The original Data Vault Engine (DVE) project began life on Source Forge where Edwin Weber and a few other smart developers from the Netherlands created the original the project.
Edwin & Co. did a brilliant job of both the VMware example and the implementation using Pentaho Data Integration (PDI). The project was known to be servicing Data Vault needs across the globe. Millersoft had it operating in at least 2 client sites including a leading freight logistics company using CargoWise as the ERP source.
Despite early successes, a few issues transpired to slow Data Vault development in general and the Source Forge DVE in particular:
-
The IT world went Data Lake/Lakehouse mad (literally). This approach was seen (wrongly) as a cheaper/faster means of producing data warehouses. The vendors all started pushing Data Lakes and the DV approach became a niche interest.
-
Getting data into a Data Vault was always much easier than getting data out. Complex joins were the norm and few could write the SQL consistently. Vendor tools helped with the automation but they were/are very expensive.
-
Document Databases and Big Data technology like Hadoop also seemed much sexier to use and learn. Budgets seldom stretched to the 6 months needed for a Data Vault and skilled engineers were scarce. There was also no standard pattern for building/refreshing the needed business layer of the Data Vault.
-
Hitachi taking over Pentaho meant falling interest in PDI and the technology eventually went closed source (in parts).
-
Edwin's Data Vault Engine arrived before the 2.0 standard became popular (an update to the DV 2.0 standard may have appeared after Millersoft started a conversion). It was out of date and only supported MySQl and Postgres options. The code base was complex to amend even for experienced PDI developers. Adding new database types was difficult and it did not support new streaming workflows. There was no GUI and complex configuration was all in a spreadsheet.
So what happened next?
- Millersoft added DV2.0 support to the original Data Vault Engine.
- Apache Hop was launched and it was possible for Millersoft to convert Edwin's DVE to a new open source platform. Thank you Matt Casters, Bart Maertens and the rest of Team Hop.
- Postgres Foreign Data Wrappers arrived to enable the DVE to support any data base (works a treat for example on Actian/Ingres X100 tables).
- Artificial Intelligence (AI) is driving the need for believable data. Data Lake staging areas are not mature enough to support interpolation by AI analysis. Ontologies are needed, and guess what, Data Vaults are ontologies by design (thanks again Dan Linstedt and AI engines understand them (with a little context).
- Suddenly AI can write all the complex Data Vault queries. Suddenly meta-data driven Data Vaults can be created with AI. Suddenly Edwin Weber's idea has come of age.
- Millersoft added a Data Vault Docker container layer for orchestration and enterprise scale-out, we even have multi-tenancy.
- Millersoft (using AI) has added a much needed GUI, the Data Vault Studio is born.
- Millersoft created a Data Vault in day against Hubspot, suddenly a new data warehouse market opens up.
- AI Analytics over believable data is a game changer but one layer is still missing over the raw vault...the Semantic Layer. We're integrating that next into the Data Vault Studio. We want to help automate the construction and population of the [Dan's]((https://www.linkedin.com/in/dlinstedt/) Business Vault.
- AI Analytics, all powered by the Data Vault architecture, fully open source engine and a semantic query layer is the new data frontier.