Azure Data Factory: How do you handle late-arriving records in incremental loads?
Azure Data Blog2d4 min read
I am looking at approaches for implementing incremental loads in Azure Data Factory when records can arrive late or existing records can be updated after the initial load. A watermark based on a source timestamp works for many scenarios, but a strict timestamp > last_watermark filter can miss records if multiple rows share the same timestamp or if updates arrive outside the expected window. For a reliable pipeline, would you recommend: Using an overlapping extraction window and deduplicating records in the target? Tracking a source-specific change version or change tracking mechanism? Using a