Data Engineering

The latest data engineering stories, curated continuously from hundreds of sources.

Renting a moving truck to grab a pizza? Feedback on my OSINT pipeline architecture

Data EngineeringReddit r/ETL/u/iaxsofia13h

ClickHouse Cloud vs. Snowflake: What drives the real-time performance-per-dollar gap

Data EngineeringClickHouse BlogSep 10

Protected Multi cloud data plane

Data EngineeringReddit r/ETL/u/Realistic_Cod_234714h

Dave Stokes: Structured Query Language 101 at Texas Linuxfest

Data EngineeringPostgreSQL Planet18h

Turns out "Iceberg is open" doesn't mean every engine can actually read your table

Data EngineeringReddit r/apachespark/u/urban-pro19h

Five AI Questions We're Hearing from Financial Services Leaders

Data EngineeringDatabricks Blog19h

Build declarative ETL pipelines with AWS Glue 6.0

Data EngineeringAWS Big Data BlogSyed Humair20h

Announcing ClickHouse Managed Postgres on Google Cloud

Data EngineeringClickHouse Blog20h

Jimmy Angelakos: 100,000 Lines of C Later: pgSafe at PGDay UK 2026

Data EngineeringPostgreSQL Planet21h

A practical approach to end-to-end Solvency II reporting in Databricks

Data EngineeringDatabricks Blog21h

Ingestion and replication: moving from detection in hours to detection in minutes.

Data EngineeringAzure Data BlogMarbricks21h

Beyond embedding: How to secure AI/BI Dashboards for every viewer

Data EngineeringDatabricks Blog23h

Introducing Consort: Test-driven development on a branching database

Data EngineeringDatabricks Blog1d

Adaptive Instructed-Retriever: Frontier-Quality Search at 2x Lower Latency

Data EngineeringDatabricks Blog1d

Readpanda: TUI to manage kafka clusters

Data EngineeringReddit r/apachekafka/u/sonirico1d

Why does the same Spark job suddenly get much slower when the data shape changes even if total volume barely moved?

Data EngineeringReddit r/apachespark/u/Far-Diamond81841d

What if large-data analysis pipelines could be inspected and resumed?

Data EngineeringReddit r/ETL/u/panagos_stathis1d

How do you attribute 1M calls if the gateway strips headers?

Data EngineeringReddit r/ETL/u/OtherwiseWeekend22221d

Diskless Kafka: Object Storage, KIP-1150, and Kafka’s Future

Data EngineeringReddit r/apachekafka/u/mmatloka1d

What is Column-Level Security?

Data EngineeringStarburst Blog1d

Christophe Pettus: All Your GUCs in a Row: maintenance_work_mem

Data EngineeringPostgreSQL Planet1d

Automated Worker Versioning with GitHub Actions

Data EngineeringTemporal BlogAdam QuanSep 9

Webinar: What Application Owners Need to Know About SQL Server

Data EngineeringSQL Server CentralPress Release1d

Formal Support for Windows?

Data EngineeringReddit r/apachespark/u/SmallAd36971d

Azure SQL DB HS Data Load Using Microsoft JDBC Driver Instead of Apache Spark Connector for SQL

Data EngineeringAzure Data BlogSandip-Khandelwal1d

gabrielle roth: The SCaLE 24x CfP is open – get your Postgres talks in!

Data EngineeringPostgreSQL Planet1d

Beyond centralization: Why APAC retail and CPG teams need dbt | Blog | Fivetran

Data EngineeringFivetran Blog1d

How to solve ecommerce analytics without manual reporting

Data EngineeringFivetran Blog1d

Joshua Drake: Two features just left PostgreSQL 19.

Data EngineeringPostgreSQL Planet1d

SQL Data Types: Reference and Best Practices

Data EngineeringDatabricks Blog1d

Stop Moving Data. Start Using It.

Data EngineeringStarburst Blog1d

We open-sourced Filament, a data movement engine in Go (full loads, incremental, CDC). Apache 2.0

Data EngineeringReddit r/ETL/u/mitchbregs1d

Build durable agents with Temporal and Lakebase

Data EngineeringDatabricks Blog1d

Spark plus Neon db, how much JDBC parallelism is too much ?

Data EngineeringReddit r/apachespark/u/sqlink21d

Hardware accelerated ETL for real time crypto analytics just got me roasted by the trading team

Data EngineeringReddit r/bigdata/u/Foreign-Fondant43122d

This founder is making cheaper, cleaner steel

Data EngineeringMIT Technology Review AIBridget Reed Morawski2d

Netflix Moves Toward Open Source Flink Autoscaler for 30,000+ Streaming Jobs

Data EngineeringReddit r/dataengineering/u/rgancarz2d

Ever tried to quickly upload a file to a Fabric Lakehouse?

Data EngineeringReddit r/MicrosoftFabric/u/DUKOfData2d

Measuring real-time performance per dollar under continuous load: CostBench’s first end-to-end results

Data EngineeringClickHouse Blog2d

Spark max Spark lifetime API

Data EngineeringReddit r/MicrosoftFabric/u/Joppepe2d