data contracts between upstream and downstream teams
Reddit r/dataengineering1d5 min read
Hi, we use Databricks, and a couple of teams produce tables, views, and table functions that other teams depend on. Within a team, its easy enough to define a contract. We codify it and always check that the data matches. But what about the upstream artifacts my team depends on? Do w define a source spec ourselves and check that its met on every run? What do we require from the upstream team? Who owns the contract, and how do you enforce it when multiple teams depend on the same thing? I want to keep this stupidly simple and enforce it automatically. How do you do this in practice? submitted b