3 posts tagged with "Data Vault"

“Great things are done by a series of small things brought together.” — Vincent van Gogh

Millersoft introduces a metadata-driven Data Vault automation platform with a visual Studio for workflow-driven delivery, AI analytics readiness, and open source extensibility.

Edinburgh, Scotland — 20th of July, 2026 — Millersoft today announced the upcoming open source release of the Millersoft Data Vault Engine and Studio, a metadata-driven platform designed to help data teams build, manage, and evolve Data Vault architectures with greater consistency, transparency, and speed.

Released under the Apache License, the Millersoft Data Vault Engine is designed to give organisations, consultants, and data engineers an open foundation for Data Vault automation. The accompanying Millersoft Data Vault Studio provides a visual, workflow-driven interface for modelling, configuration, orchestration, and guided interaction with the Engine.

As organisations prepare for AI analytics, semantic modelling, data products, and enterprise-scale governance, the need for trusted, historised, auditable data foundations has never been greater. Millersoft believes the Data Vault pattern, combined with open source engines and metadata-first automation, represents a new frontier for modern analytics delivery.

“AI analytics, all powered by the Data Vault architecture and open source engines, is the new data frontier,” said Calum Miller, Founder/Owner of Millersoft. “We want teams to explore it, build on it, and adapt it to their own environments. The Engine and Studio are designed to make Data Vault delivery more repeatable, more transparent, and more accessible.”

A Metadata-Driven Approach to Data Vault Delivery​

The Millersoft Data Vault Engine is built around a metadata-first philosophy. Instead of hard-coding every pattern by hand, users define structures, relationships, rules, and workflows that the Engine can use to support repeatable Data Vault generation and operation.

The Studio adds a visual layer that helps teams interact with the Engine through guided workflows, making the platform suitable not only for engineers, but also for architects, analysts, and delivery teams who need a clearer view of how Data Vault components are defined and managed.

Key capabilities include:

• Metadata-driven Data Vault modelling and automation
• Visual Studio interface for workflow-driven interaction
• Support for repeatable Data Vault delivery patterns
• Open source extensibility under the Apache License
• A foundation for analytics, AI readiness, governance, and lineage-aware delivery
• Professional services support for organisations seeking expert guidance

Open Source Foundation, Professional Experience Available​

Millersoft is releasing the platform as open source to encourage experimentation, learning, contribution, and innovation across the Data Vault community.

However, Millersoft also recognises that successful Data Vault delivery requires more than tooling. The Data Vault pattern can appear deceptively simple, but real-world implementations often expose complex challenges around business keys, source-system behaviour, historical change, data quality, integration semantics, and governance.

“The Engine and Studio can help accelerate delivery, but they do not replace Data Vault experience,” added Calum Miller. “Our goal is to give teams a powerful open source foundation while also making experienced support available for organisations that want to avoid common pitfalls and deliver with confidence.”

Millersoft’s consultants bring practical experience building live Data Vaults both manually and using the Millersoft Data Vault Engine. Professional services are available for architecture review, implementation support, modelling validation, delivery acceleration, and best-practice guidance.

Availability​

The Millersoft Data Vault Engine and Studio will be made available as an open source project under the Apache License.

Further details, documentation, community resources, and release information will be published via Millersoft’s official channels.

Organisations interested in early access, implementation support, or professional services can contact Millersoft directly.

About Millersoft​

Millersoft helps organisations design, build, and operate modern data platforms using Data Vault architecture, metadata-driven automation, and pragmatic engineering practice.

With experience across hand-built and automated Data Vault delivery, Millersoft supports teams seeking scalable, governed, and AI-ready analytics foundations.

History of the Data Vault Engine​

More details here on the original history of the Data Vault Engine and the Millersoft refinements.

Videos of Data Vault Studio in Action​

Watch Data Vault Studio use AI to build a Data Vault.

Contact​

Millersoft
LinkedIn: https://www.linkedin.com/in/millersoft
Website: https://millersoft.co
Email: calum+datavault@millersoftltd.com
GitHub: https://github.com/millersoft/datavault

---

The original Data Vault Engine (DVE) project began life on Source Forge where Edwin Weber and a few other smart developers from the Netherlands created the original the project.

Edwin & Co. did a brilliant job of both the VMware example and the implementation using Pentaho Data Integration (PDI). The project was known to be servicing Data Vault needs across the globe. Millersoft had it operating in at least 2 client sites including a leading freight logistics company using CargoWise as the ERP source.

Despite early successes, a few issues transpired to slow Data Vault development in general and the Source Forge DVE in particular:

• The IT world went Data Lake/Lakehouse mad (literally). This approach was seen (wrongly) as a cheaper/faster means of producing data warehouses. The vendors all started pushing Data Lakes and the DV approach became a niche interest.

• Getting data into a Data Vault was always much easier than getting data out. Complex joins were the norm and few could write the SQL consistently. Vendor tools helped with the automation but they were/are very expensive.

• Document Databases and Big Data technology like Hadoop also seemed much sexier to use and learn. Budgets seldom stretched to the 6 months needed for a Data Vault and skilled engineers were scarce. There was also no standard pattern for building/refreshing the needed business layer of the Data Vault.

• Hitachi taking over Pentaho meant falling interest in PDI and the technology eventually went closed source (in parts).

• Edwin's Data Vault Engine arrived before the 2.0 standard became popular (an update to the DV 2.0 standard may have appeared after Millersoft started a conversion). It was out of date and only supported MySQl and Postgres options. The code base was complex to amend even for experienced PDI developers. Adding new database types was difficult and it did not support new streaming workflows. There was no GUI and complex configuration was all in a spreadsheet.

So what happened next?​

• Millersoft added DV2.0 support to the original Data Vault Engine.
• Apache Hop was launched and it was possible for Millersoft to convert Edwin's DVE to a new open source platform. Thank you Matt Casters, Bart Maertens and the rest of Team Hop.
• Postgres Foreign Data Wrappers arrived to enable the DVE to support any data base (works a treat for example on Actian/Ingres X100 tables).
• Artificial Intelligence (AI) is driving the need for believable data. Data Lake staging areas are not mature enough to support interpolation by AI analysis. Ontologies are needed, and guess what, Data Vaults are ontologies by design (thanks again Dan Linstedt and AI engines understand them (with a little context).
• Suddenly AI can write all the complex Data Vault queries. Suddenly meta-data driven Data Vaults can be created with AI. Suddenly Edwin Weber's idea has come of age.
• Millersoft added a Data Vault Docker container layer for orchestration and enterprise scale-out, we even have multi-tenancy.
• Millersoft (using AI) has added a much needed GUI, the Data Vault Studio is born.
• Millersoft created a Data Vault in day against Hubspot, suddenly a new data warehouse market opens up.
• AI Analytics over believable data is a game changer but one layer is still missing over the raw vault...the Semantic Layer. We're integrating that next into the Data Vault Studio. We want to help automate the construction and population of the [Dan's]((https://www.linkedin.com/in/dlinstedt/) Business Vault.
• AI Analytics, all powered by the Data Vault architecture, fully open source engine and a semantic query layer is the new data frontier.

Now go fill your boots and send us a postcard of the view!​

Team Millersoft​

Millersoft has open sourced a powerful Data Vault Studio editor for easy maintenance of the Data Vault Engine. This studio enables AI driven data vault development at scale.

General Data Vault Studio Architecture​

Watch Data Vault Studio in Action​

Watch Data Vault Studio AI Insights in Action​