I’m writing this at 2:30 AM after spending roughly six hours chasing a mysterious bottleneck in my integration tests. While implementing the Kafka adapter for @anabranch/eventlog, I encountered a consistent 5-second delay when using admin.setOffsets for manual cursor commits. This latency was particularly noticeable in integration tests and administrative tasks where the consumer group was often idle.

The delay is a side effect of how KafkaJS implements setOffsets. When you see admin.setOffsets, you might expect a fast, lightweight admin function like I did. However, the library actually spins up a transitive consumer that joins the group, attempts a fetch to synchronize state, and then performs the update. Crucially, this internal consumer uses the default maxWaitTimeInMs of 5000ms:

// kafkajs/src/admin/index.js
const consumer = createConsumer({
  logger: rootLogger.namespace('Admin', LEVELS.NOTHING),
  cluster,
  groupId,
})

// Which eventually calls createConsumer with a hardcoded default:
({
  // ...
  maxWaitTimeInMs = 5000,
  // ...
}) => {
  // ...
}

In idle groups or topics with no new data, the consumer blocks for the full duration of maxWaitTimeInMs before the fetch request times out and the operation continues. While this is a safe default for high-throughput production consumers, it is unnecessarily slow for administrative updates.

Fitting Kafka into a Clean Interface

The @anabranch/eventlog abstraction defines a simple commitCursor operation that should, in theory, be a straightforward metadata update. Mapping this "obvious" requirement onto Kafka proved surprisingly difficult due to how Kafka manages group state and offsets. Unlike a simple key-value store, Kafka's offsets are tightly coupled to consumer group coordination, which introduces numerous edge cases - especially when trying to commit offsets "out-of-band" while a group might be rebalancing or empty.

A Configurable Mimic

To resolve the latency issue while respecting Kafka's requirements, we can mimic KafkaJS’s internal logic but with explicit control over the timing parameters. Instead of using the high-level admin.setOffsets, we implement a more surgical commit by spinning up a temporary consumer with a significantly lower maxWaitTimeInMs (e.g., 100ms).

Our implementation handles the following steps in a single commitCursor task:

  1. Initialize a temporary consumer with a configurable dummyMaxWaitTimeInMs.
  2. Connect and subscribe to the relevant topic.
  3. Use seek to point the consumer to the exact target offsets for each partition.
  4. Perform a standard commitOffsets and immediately disconnect.

By manually controlling the lifecycle and timing of this transitive consumer, we maintain the correctness of the group offset commit while reducing the latency from 5000ms to ~150ms per call. It’s still not fantastic - it remains the slowest integration test in the suite - but Kafka is bulky by nature, and 150ms is a far better baseline than five seconds.

The source for this adapter and the rest of the Anabranch monorepo is available on GitHub and JSR.