Over two days, we explored how an open-source platform guarantees digital sovereignty.
Today, the most important decision when building an AI-ready data platform is no longer choosing the right LLM. Rather, it is this: How do we build a platform that remains open, sovereign, and adaptable for years to come? This is precisely the question we explored during our internal DEVnet Hackathon.
Five teams. Two days.
The goal was not merely to experiment with the latest technologies. Instead, we wanted to determine if a modern, AI-ready Lakehouse platform could be built entirely on open source. Without vendor lock-in. Without hyperscalers. And without sensitive data ever leaving the company’s own infrastructure.
Our result: Apache Iceberg, Trino, Spark, dbt, Superset, MLflow, and OpenMetadata, combined with an AI agent that answers questions in natural language via MCP. Powered by a locally hosted open-source language model, the entire processing chain remains within our infrastructure.
Instead of focusing on individual technologies, we centered our efforts on five architectural decisions:
1. Does every data platform require a cluster?
Not necessarily. DuckDB and Polars write to the same Iceberg REST catalog as Spark. Compute has become interchangeable. Today, appropriate sizing is an architectural decision.
2. How do you build governance without a dedicated governance project?
Lineage, ownership, and data contracts are automatically generated from dbt artifacts. Governance becomes part of the platform rather than a costly retrofitting project.
3. How do you avoid lock-in in the analytics layer?
Superset generates the SQL. Trino executes it via pushdown. The BI tool becomes interchangeable. The business logic remains within the platform.
4. How do you make machine learning reproducible?
Git versions the code. Iceberg versions the data via snapshots. MLflow connects the two. Reproducibility becomes a platform capability.
5. When is an AI agent production-ready?
Not when it impresses in a demo, but when it delivers reliable results. Smaller open-source models often struggle with tool selection. Therefore, in our view, production-ready AI requires exclusive read access, resource limitations, and a human-in-the-loop from the start.
Our most important takeaway:
The open-source Lakehouse ecosystem has come of age. The limiting factor today is no longer technology. The deciding factors are the right architectural decisions, as they determine flexibility, AI readiness, governance, cost efficiency, and digital sovereignty.
This is the difference between a platform that works today and a platform that will continue to create genuine value in the future.
