The current method of providing data services using data lakes often leads to highly specialized, siloed teams of data engineers and business analysts. The result is complexity, limited flexibility, and a long time-to-market for data products. This can be prevented by allowing decentralized DevOps teams to develop business-driven functionality on top of central data services.
The Problem: Data Silos
In many organizations, three distinct silos emerge:
- Source Teams: These teams provide operational data.
- Data Platform Teams: They handle the processing and “plumbing” of the data.
- Domain-Driven Business Teams: The end-users who actually need the data for insights.
Implementing changes in this structure is complicated and requires constant coordination between these three groups. This often leads to a bottleneck where the central data team cannot keep up with the diverse needs of different business units.
A New Approach: Data Mesh
The article proposes a shift toward a “Data Mesh” architecture. This approach redefines the roles:
- Platform Teams: Their job is to provide the underlying data infrastructure and tools as a service.
- Business DevOps Teams: These are cross-functional teams consisting of experts from operational systems, data engineers, and business analysts.
In this model, the Business DevOps teams are responsible for building their own domain-specific data products. Because they understand both the source of the data and the business requirements, they can respond quickly and flexibly to changes in operational systems or new business questions.
Conclusion
An agile, business-driven way of working is essential for the success of modern data strategies. By shifting responsibility to the business domains (the “Data Mesh” approach), organizations can ensure that data remains a valuable asset rather than a technical burden.
Key Takeaways:
- Decentralization: Move away from one massive, centrally managed data lake toward decentralized “data products.”
- Ownership: The business units (domains) should own their data lifecycle, not just the IT department.
- Agility: This structure reduces the time-to-market for new insights because the people who need the data are the ones building the tools to analyze it.