r/software • u/GuidanceBrief612 • 20d ago
News I built a Python reference implementation for Data Mesh governance. Are data contracts enough to prevent downstream failures?
I've been exploring a problem that keeps coming up in enterprise data engineering: Data Mesh gives us domain ownership, but how do we actually enforce governance across domain boundaries?
Consider a Customer domain publishing profiles consumed by Analytics and ML pipelines. What happens when an upstream schema changes, a duplicate record slips through, or stale data enters a downstream feature?
I recently published research on domain-driven data and AI platforms and built a small open-source Python reference implementation to explore this.
The implementation includes:
- Four synthetic domain-owned data products
- Versioned data contracts with schema, uniqueness, and freshness checks
- Publication gates that reject invalid data before replacing published outputs
- Simulated cross-domain consumption permissions
- Product-level lineage and downstream impact analysis
- Reproducible tests covering failure scenarios
Everything runs using standard Python without external dependencies.
One interesting takeaway: enforcing data quality at the publication boundary is relatively straightforward. Understanding the consequences of a contract change across downstream products is considerably harder.
The project is deliberately lightweight. It's not a production-grade governance framework, and the access controls are illustrative rather than real security enforcement.
GitHub: https://github.com/akshaysharma1088/domain-driven-ai-research-companion
I'm particularly interested in how others approach this in production.
Do you rely primarily on data contracts, tools like dbt/Great Expectations, or a metadata/lineage platform? And how do you handle semantic changes that don't technically break the schema?
Would appreciate technical feedback, especially on scenarios this reference implementation should test next.
2
u/NBALive1995 20d ago
Slop