Data Science Wire

How do you move consumers to a new topic without losing your place?

Reddit r/apachekafka1mo4 min read

Every few weeks we have to move consumers from one topic to another. In our case it's pretty much always topic renames (naming convention cleanup). The tricky part is the ordering: producers don't switch to the new topic until consumers are already on it. Our consumer switching over is what triggers them to migrate, and even then they move on their own schedule. So the old topic keeps getting messages for who knows how long after we've switched. That means we can't do the comfortable thing (stop producers, let consumers drain the old topic, then move everyone). We have to switch consumers to t

Read the full story at Reddit r/apachekafka

More in Data Engineering