Data Engineering

The latest data engineering stories, curated continuously from hundreds of sources.

Christophe Pettus: All Your GUCs in a Row: row_security

Data EngineeringPostgreSQL Planet3h

Vadim Yatsenko: Splitting the Read-Only Transaction: Replica Reads Inside an Open Transaction

Data EngineeringPostgreSQL Planet9h

How are you monitoring Debezium/CDC?

Data EngineeringReddit r/dataengineering/u/Particular-Idea-178616h

Radim Marek: Getting rid of a 4 TB table in PostgreSQL

Data EngineeringPostgreSQL Planet19h

Lakebase branching for CI/CD: lessons from 100 PR databases

Data EngineeringDatabricks Blog20h

Is incremental batch processing of SCD type 2 with multiple sources, actually that hard to solve?

Data EngineeringReddit r/databricks/u/AggravatingAvocado3621h

This Halloween, I'm building a horror game where the only way to survive is to migrate to Databricks

Data EngineeringReddit r/databricks/u/Significant-Guest-1422h

Need help to get Databricks Voucher

Data EngineeringReddit r/databricks/u/Beautiful-Wash34521d

VM choice - D8ads_v7 vs F4amds_v7

Data EngineeringReddit r/databricks/u/Lumiaman881d

Snowflake Intelligence-> CoWork -> ???

Data EngineeringReddit r/snowflake/u/Global-War1811d

Read-Only Apache Iceberg Catalog for Kafka

Data EngineeringReddit r/apachekafka/u/GuidanceAggressive501d

Does agentic analytics need a separate semantic layer if the data is already well modeled?

Data EngineeringReddit r/databricks/u/Apprehensive_Tie24941d

GCP(Dataproc) - When do you run Spark jobs in Cluster Mode on Dataproc?

Data EngineeringReddit r/dataengineering/u/SinkNeither15271d

Genie App Builder

Data EngineeringReddit r/databricks/u/hubert-dudek1d

Databricks and SAP unleash end-to-end data & AI capabilities to level up enterprise AI transformation

Data EngineeringDatabricks Blog1d

Wealth Management Client Onboarding Needs More Than AI

Data EngineeringDatabricks Blog1d

Deliberate decisions and enterprise AI value

Data EngineeringDatabricks Blog1d

The Lakehouse Isn’t Enough. Agents Need an Intelligence Layer of Their Own.

Data EngineeringStarburst Blog1d

Heads up: Snowflake Backups now skip transient tables without warning (BCR-2360)

Data EngineeringReddit r/snowflake/u/Spiritual-Kitchen-792d

Igor Suhorukov: pg_vexec one month on: ahead of ClickHouse on Parquet, close behind on MergeTree

Data EngineeringPostgreSQL Planet2d

Robert Haas: Postgres 19 is late, and it's AI's fault

Data EngineeringPostgreSQL Planet2d

Interesting Links in Data Engineering: October 2026

Data EngineeringReddit r/dataengineering/u/rmoff2d

The hawkish pause, Lula's market meltdown, and the Anthropic IPO bombshell: The financial week in five charts

Data EngineeringReuters2d

Interesting Kafka Links: October 2026

Data EngineeringReddit r/apachekafka/u/rmoff2d

OpenAI’s latest discoveries spark a backlash + Help! My kid hates AI : Machine Gods

Data EngineeringNPR2d

unit tests

Data EngineeringReddit r/databricks/u/ptab02112d

Reading order is the part of PDF parsing that quietly breaks downstream NLP

Data EngineeringReddit r/LanguageTechnology/u/Sussybaka8008692d

Starting out on Databricks

Data EngineeringReddit r/databricks/u/adwaithpn2d

[Project] Kawa 0.1.0 - An open-source Kafka protocol gateway

Data EngineeringReddit r/apachekafka/u/eniac_g2d

desflow: realistic Kafka event streams from a simulation

Data EngineeringReddit r/apachekafka/u/jaehyeon-kim2d

dbt users on Databricks: do your snapshots handle files that arrive out of order?

Data EngineeringReddit r/databricks/u/Alone-Security70442d

Does anyone use Streams and Tasks anymore?

Data EngineeringReddit r/snowflake/u/SinkNeither15272d

You don't need to build a new reliability system for every AI Framework

Data EngineeringTemporal BlogDavid HydeOct 30

Regina Obe: PostGIS 3.7.0

Data EngineeringPostgreSQL PlanetOct 30

Announcing General Availability of Real-Time Mode for Spark Declarative Pipelines on Databricks

Data EngineeringDatabricks Blog2d

Anyone using StreamNative

Data EngineeringReddit r/apachekafka/u/Help-pichu2d

Table update triggers now support up to 25 tables per job (higher coming soon)

Data EngineeringReddit r/databricks/u/saad-the-engineer2d

Introducing Apache Kafka 4.4.0

Data EngineeringConfluent BlogOmnia Ibrahim2d

Daily Co-Co credits

Data EngineeringReddit r/snowflake/u/HeartAndCircuits3d

Is Databricks the right solution for a team of 3 data scientists?

Data EngineeringReddit r/databricks/u/Mother_Seaweed89873d