Data Engineering

The latest data engineering stories, curated continuously from hundreds of sources.

Indexing the Data Lake for Online Point Queries

Data EngineeringSpotify Engineering BlogSpotify Engineering4d

Apache Iceberg Metadata Tables: Querying the Internals

Data EngineeringDremio BlogAlex Merced5d

Handling Row filters and column masks accross multiple engines

Data EngineeringReddit r/dataengineering/u/daibam_und_koode5d

Data Engineering Weekly #280

Data EngineeringData Engineering WeeklyAnanth Packkildurai5d

放弃XStream吧，聊聊我们怎么用自研的TLA组件搞定Oracle日志解析

Data EngineeringReddit r/ETL/u/Independent_Sky26585d

From Zero to Self-Built Oracle Transaction Log Analytics Engine: How TLA Breaks the Performance Shackles of LogMiner, XStream, and OGG

Data EngineeringReddit r/ETL/u/Independent_Sky26585d

Christophe Pettus: All Your GUCs in a Row: gin_fuzzy_search_limit

Data EngineeringPostgreSQL Planet5d

ClickHouse docs relaunch

Data EngineeringClickHouse BlogJul 27

How data modeling in Palantir Foundry actually differs from traditional SQL Data Warehousing

Data EngineeringReddit r/ETL/u/Palantir-foundry5d

Are pipelines the correct analogy to use?

Data EngineeringReddit r/dataengineering/u/marketlurker5d

Andrei Lepikhov: Why your pioneering Postgres feature should start in a fork

Data EngineeringPostgreSQL Planet6d

PDF Data Extraction

Data EngineeringReddit r/dataengineering/u/Every-Whereas57936d

VecFuzz: SymSpell speed without the heavy RAM footprint

Data EngineeringReddit r/dataengineering/u/Shaweyy6d

Lance table format explained simply (updated)

Data EngineeringReddit r/dataengineering/u/TonTinTon6d

Apache Spark 4.2: what actually matters and what to ignore

Data EngineeringReddit r/databricks/u/vicio356d

Best Approach for Integrating Separate Data Sources

Data EngineeringReddit r/dataanalysis/u/Technical-Bus-14236d

Spark 4.2: a visual breakdown of what actually changed

Data EngineeringReddit r/apachespark/u/vicio351w

Christophe Pettus: All Your GUCs in a Row: full_page_writes

Data EngineeringPostgreSQL Planet1w

SQL Data Sync Retirement: Migration Insights and Modern Alternatives

Data EngineeringAzure Data BlogKarunakar_Kotha1w

New research: The high cost of stale ERP data | Blog | Fivetran

Data EngineeringFivetran Blog1w

What happens to SCD2 tables if the source pipeline performs a Full Refresh?

Data EngineeringReddit r/databricks/u/Livid_Connection64291w

Apache Flink Agents 0.3.1 Release Announcement

Data EngineeringApache Flink BlogJul 25

How OpenAI's Security Team Turned a Streaming Pipeline Into Its Own Latency Watchdog with SDP ForEachBatch

Data EngineeringReddit r/databricks/u/Much-Neat-62731w

How we built a DuckDB transpiler

Data EngineeringReddit r/dataengineering/u/Aeronautical-You49171w

Introducing Lightstream: Measured faster than Apache Arrow Flight (gold standard) on every axis in open 50gbps EC2 network benchmarks whilst producing a single fully ordered stream off parallel data exchange.

Data EngineeringReddit r/dataengineering/u/peterxsyd1w

Building a workflow migration engine: Harness/Pi agents or LangChain/LangGraph?

Data EngineeringReddit r/ETL/u/imsuryya1w

Went to use Airflow and found it stuck at version 2.0

Data EngineeringReddit r/MicrosoftFabric/u/MonkeyDDataHQ1w

Automate creating AWS Glue Data Catalog views with AWS SDK for data mesh use case

Data EngineeringAWS Big Data BlogAarthi Srinivasan1w

How we pushed CDC into Postgres — and turned replication into clockwork

Data EngineeringReddit r/snowflake/u/craigkerstiens1w

Orchestrator: Rivers v0.4.0 – added OIDC and forward auth support

Data EngineeringReddit r/dataengineering/u/elgreco_141w

The Hidden Cost of Postgres Constraints at Scale

Data EngineeringTimescale BlogNanoHertz Communications1w

Is Databricks and Fabric overtaking Snowflake?

Data EngineeringReddit r/dataengineering/u/Throwaway0819202311w

What orchestration platform supports multi tenancy with proper isolation between business units?

Data EngineeringReddit r/ETL/u/fatassoo1w

Shaun Thomas: Looking Forward to Postgres 19: Autovacuum Tweaks

Data EngineeringPostgreSQL Planet1w

Airbyte Kotlin rewrite of connectors has terrible QA

Data EngineeringReddit r/dataengineering/u/Adrien06231w

DuckGQL : An ISO GQL extension to run graph queries and algorithms on DuckDB

Data EngineeringReddit r/dataengineering/u/randomchickibum1w

How do you monitor ETL pipelines to catch failures before users notice them?

Data EngineeringReddit r/ETL/u/Effective_Ocelot_4451w

Building a star schema in SDP? Use identity columns with Streaming Tables!

Data EngineeringReddit r/databricks/u/minibrickster1w

SQL Server Data Warehouse Concepts, Schema Design, and Implementation

Data EngineeringSQL Server CentralAdditional Articles1w

SQL Agent Job Automated Change Capture Process

Data EngineeringSQL Server CentralJames Greaves1w