Configuration and tuning recommendations for RabbitMQ when used with Senzing.
RabbitMQ performs best when queues are shallow. At Senzing, we've observed severe bottlenecks when queues grow to 40M+ messages as RabbitMQ struggles to manage the index overhead. Use RabbitMQ as a buffer, not deep storage—if you need to queue hundreds of millions of messages, consider a system designed for that (e.g., NATS JetStream, Pulsar, or AWS SQS).
This guide applies to both RabbitMQ 3.x and 4.x. The queue/exchange setup, consumer patterns, and producer backpressure handling below are identical across both. Two things differ by major version and are called out inline where relevant:
- Classic queue storage mode (see Queue Storage Mode)
- Classic queue mirroring / HA (see High Availability)
If you're on RabbitMQ 3.13 with Khepri enabled, note that in-place upgrade to 4.x is not supported — that path requires a fresh-cluster migration, not an upgrade. RabbitMQ 4.x requires Erlang/OTP 26 or 27.
If you're deploying via Docker: Bitnami has removed bitnami/rabbitmq from Docker Hub entirely
(no tags remain as of this writing). Use the official rabbitmq image instead
(e.g. rabbitmq:4.3-management) — same env var conventions for default user/pass, different data
directory (/var/lib/rabbitmq vs Bitnami's /bitnami/rabbitmq/mnesia).
RabbitMQ's default consumer_timeout (30 minutes in recent versions) will close connections if a message takes too long to acknowledge. This can happen with huge entities.
For Senzing workloads with variable processing times, increase or disable this limit:
# rabbitmq.conf
# 30 minutes (in milliseconds)
consumer_timeout = 1800000
# Or disable entirely for very long processing
# consumer_timeout = infinityPrevent RabbitMQ from accepting messages when resources are constrained:
# rabbitmq.conf
vm_memory_high_watermark.relative = 0.6
disk_free_limit.relative = 1.5This is a separate mechanism from per-queue disk paging below — don't conflate the two. The watermark is a broker-wide safety valve: it blocks all publishing once total broker memory/disk crosses the threshold. It does not prevent an individual backlogged queue from paging its own messages to disk well before that threshold is ever reached.
Classic queues page message bodies to disk once a queue's backlog grows past a small in-RAM
window, independent of delivery mode (persistent vs. transient) and independent of how much
broker memory is actually available. We measured this directly on a live queue capped at
max-length: 50000: at 38,886 messages queued, 37,101 (95%) were paged to disk while the broker
was using 0.52 GB of a 53.9 GB memory high watermark — nowhere near memory pressure. Raising the
watermark or adding RAM does not change this; it's routine per-queue behavior for classic queues
(CQv2), not a reaction to load.
- RabbitMQ 3.x: you can opt a queue into the older
x-queue-version: 1storage mode, which stays RAM-resident until a real memory alarm rather than paging proactively. Given typical Senzing queue depths (thousands, capped bymax-length), this comfortably fits in memory on any modern host.{"arguments": {"x-queue-version": 1, "...": "..."}} - RabbitMQ 4.x: classic queue v1 has been removed. A 4.0+ node automatically migrates any v1 queue to v2 on startup (RabbitMQ's own benchmark: ~9s for a queue with 1M 100-byte messages). There is no way to opt out of the paging behavior above on 4.x.
This only matters for small messages. Any message larger than
queue_index_embed_msgs_below (default 4096 bytes) is always written to the on-disk message
store, regardless of delivery mode, queue depth, or classic queue version — this is fundamental
classic-queue architecture, not a paging behavior you can tune around. Senzing records can be
100MB+ (large entities with extensive relationship/alias data). For those, disk writes are
unavoidable by design; keeping max-length low does not help. queue_index_embed_msgs_below is
itself configurable, but raising it to cover 100MB messages means embedding huge blobs directly in
the queue index rather than the message store — we have no evidence this is a good idea and are
not recommending it. If your workload's message sizes are consistently small (under a few KB),
the RAM-residency options above are worth using; if they're consistently large, budget disk
throughput for the message store instead of trying to avoid it.
A destination for rejected messages during processing. Messages responded to with basic_reject(requeue=False) route here. Do not size-limit this queue.
Queue Name: senzing-dlx-queue
Durable: yes
Exchange Name: senzing-dlx-exchange
Type: direct
Routing Key: senzing.deadletter
Queue Name: senzing-rabbitmq-queue
Durable: yes
max-length: 50000
overflow: reject-publish
x-dead-letter-exchange: senzing-dlx-exchange
x-dead-letter-routing-key: senzing.deadletter
Exchange Name: senzing-rabbitmq-exchange
Type: direct
Routing Key: senzing.records
Why 50,000 max-length?
RabbitMQ performance degrades significantly with deep queues. This cap keeps the broker responsive. When the queue is full, reject-publish causes the broker to reject new messages—producers must handle this via publisher confirms and implement backpressure (pause/retry).
For multi-threaded consumers, set prefetch to match your thread count:
ch.basic_qos(prefetch_count=num_threads)Use an executor pool for parallel processing, but ack from the main thread only:
1. Main thread: fetch messages via consume() with inactivity_timeout
2. Executor pool: submit processing tasks
3. Main thread: poll futures, ack/reject on completion
This avoids Pika thread-safety issues and keeps the channel state consistent.
with concurrent.futures.ThreadPoolExecutor(max_workers) as executor:
ch.basic_qos(prefetch_count=max_workers)
futures = {}
while True:
# Poll for completed work
done, _ = concurrent.futures.wait(
futures, timeout=10, return_when=concurrent.futures.FIRST_COMPLETED
)
for fut in done:
msg = futures.pop(fut)
try:
fut.result()
ch.basic_ack(msg.delivery_tag)
except (SzBadInputError, SzRetryTimeoutExceededError):
ch.basic_reject(msg.delivery_tag, requeue=False) # -> DLQ
# Fetch more work
while len(futures) < max_workers:
msg = next(ch.consume(queue, inactivity_timeout=10))
if not msg or not msg[0]:
break
futures[executor.submit(process, msg)] = msgRecords that take too long to process should be rejected to the DLQ to prevent blocking other work and to avoid consumer_timeout disconnects.
LONG_RECORD = 300 # 5 minutes - log warning
REJECT_THRESHOLD = 600 # 10 minutes - reject to DLQ
for fut, msg in futures.items():
if not fut.done():
duration = time.time() - msg.start_time
if duration > REJECT_THRESHOLD and not msg.acked:
ch.basic_reject(msg.delivery_tag, requeue=False)
msg.acked = True # mark so we don't double-ack later
elif duration > LONG_RECORD:
print(f"Still processing ({duration/60:.1f} min): {msg.record_id}")This pattern:
- Warns about slow records at 5 minutes
- Rejects to DLQ at 10 minutes
- Prevents double-ack/reject by tracking state
- Keeps threads from stalling on one slow record
- RabbitMQ 3.x: classic queue mirroring (
ha-mode/ha-paramspolicies) replicates a classic queue across nodes. - RabbitMQ 4.x: classic queue mirroring is removed. Any mirroring policy becomes a no-op after upgrading — queues silently drop to a single replica with no error raised. If you rely on HA, migrate to quorum queues before or during the upgrade; see RabbitMQ's own migration guide. Note quorum queues have different storage semantics than classic queues — every message goes through a replicated, disk-backed log regardless of delivery mode, so they trade the RAM-residency option above for durability + HA.
When overflow: reject-publish triggers, the broker will NACK new publishes. Producers should:
- Enable publisher confirms
- Handle NACKs by pausing/retrying with backoff
- Monitor for sustained backpressure (indicates consumers can't keep up)
channel.confirm_delivery()
try:
channel.basic_publish(
exchange='senzing-rabbitmq-exchange',
routing_key='senzing.records',
body=message,
mandatory=True
)
except pika.exceptions.NackError:
# Queue full—backoff and retry
time.sleep(backoff_seconds)See sz_rabbit_consumer for a complete example implementing these patterns.