How do you handle fan-in per timestamp in micro-batching?
Reddit r/dataengineering5d4 min read
Our device sends 3 separate records for each timestamp: data points, frames, and metadata. They arrive independently and in any order. Each timestamp can be processed on its own as soon as all 3 parts are there, without waiting for the batch window to close. How do you handle this fan-in, so that each timestamp is processed exactly once as soon as it's complete, with a timeout for sets that never complete? submitted by /u/Expensive_Break_6163 [link] [comments]