How are you orchestrating dbt, Airbyte, and Spark together without it becoming a mess ?
Reddit r/analytics1mo4 min read
Our data stack is Airbyte for ingestion, Spark for heavy transforms and dbt for the modeling layer. Right now each tools runs on its own schedule aand we coordinate them with a slack message that says : airbyte finished you can trigger dbt now. Yes, I'm embarrassed writing this. I want one place where I can define: Airbyte sync finishes, Spark job runs and dbt models build a and Slack notification if anything fails. Tried wiring this through Airflow but writing Python DAGs for what is essentially run these 4 things in order with retries felt like massive overkill. What are you guys using? subm
