Technologies

Async Job Architectures: DB Polling vs RabbitMQ vs Transactional Outbox

H
Harshit Sharma
·

An API request takes 20–30 seconds to complete. The user is staring at a loader, the server is holding the connection open, and a slow downstream service can delay the entire response.

At some point, the question becomes: Does this work really need to finish before we respond to the user?

This is where asynchronous jobs come in. Instead of keeping the request open until everything is finished, the API can create a job, return a response, and let a separate worker handle the processing in the background.

For example, consider an API like:

typescript
POST /reports/export

But once jobs are asynchronous, a few questions come up:

Where do we store jobs? How do workers know when work is available? What happens when something fails?

There are several ways to solve this. A common progression is:

DB Polling → DB + RabbitMQ → Transactional Outbox + RabbitMQ

Let's look at how each approach works, where it starts falling short, and why the next one becomes useful.

1. DB Polling

The simplest way to implement asynchronous processing is to store jobs in a database and let workers periodically check for pending work.

The architecture looks like this:

text
API
 ↓
Database
 ↓
Scheduler
 ↓
Worker

When a request arrives, the API creates a job in the database and immediately returns a response. A scheduler periodically checks for pending jobs and assigns them to workers.

For example:

  1. A client calls POST /reports/export.
  2. The API creates a job with the status PENDING.
  3. The API returns the job ID to the client.
  4. A worker picks up the job when the scheduler finds it.
  5. The worker generates the report and updates the status to COMPLETED or FAILED.

The client doesn't need to wait for the report to be generated. It can check the job status separately.

Advantages

  • Simple to implement.
  • No additional messaging infrastructure.
  • Easy to inspect and manage jobs directly from the database.

Limitations

  • Repeated database polling creates additional queries.
  • Database load increases as the number of jobs and workers grows.
  • Job claiming, retries, and stuck jobs need careful handling.

Where Does It Start Hurting?

The main issue is that workers don't know exactly when a new job arrives. They discover it during their next polling cycle.

If workers poll every five seconds, a job might wait almost five seconds before processing even begins. Polling more frequently reduces this delay but increases database traffic.

With multiple workers, job claiming also has to be atomic. Otherwise, two workers may pick up the same PENDING job.

In PostgreSQL, a query like this can help workers claim different jobs without waiting on rows already locked by other workers:

sql
SELECT *
FROM jobs
WHERE status = 'PENDING'
ORDER BY created_at
FOR UPDATE SKIP LOCKED;

This is a simplified example. The job must be claimed and its status updated within the same transaction to prevent multiple workers from processing it.

DB polling isn't inherently bad. It works well for many applications, especially when job volumes are moderate and a small processing delay is acceptable.

The trade-off changes when database load, dispatch latency, or sudden workload spikes become significant constraints. That's when introducing a message broker starts making sense.

2. DB + RabbitMQ

Instead of having workers repeatedly check the database, we can introduce a message broker.

RabbitMQ allows the application to publish messages that workers can consume when they're ready to process them.

The architecture now looks like this:

text
API
 ↓
Database
 ↓
RabbitMQ
  ↙  ↓  ↘
Worker Worker Worker

When a request arrives, the API creates a job and publishes a message containing the job ID to RabbitMQ. Workers consume messages from the queue and process the corresponding jobs.

For example:

  1. A client requests a report export.
  2. The API creates a job in the database.
  3. The API publishes a message containing the job ID to RabbitMQ.
  4. A worker consumes the message and generates the report.
  5. The worker updates the job status in the database.

Multiple workers can process different jobs in parallel. The queue also acts as a buffer when requests arrive faster than workers can handle them.

Imagine 10,000 jobs arriving within a short period. If workers can process only 1,000 jobs at a time, RabbitMQ can hold the remaining work until workers become available.

This helps separate the rate at which jobs arrive from the rate at which they are processed.

Advantages

  • Workers can scale independently.
  • The queue buffers workload spikes.
  • Supports acknowledgements, redelivery, and dead-letter handling.

RabbitMQ also provides features like acknowledgements, redelivery, dead-letter exchanges, durable queues, and publisher confirms to help handle messages reliably.

But the application still decides how many times to retry, when to stop, and what happens to failed jobs.

This is a natural next step when DB polling starts becoming a bottleneck.

Limitations

  • Adds messaging infrastructure.
  • The database and broker can still get out of sync.
  • Retry policies and failure handling require application-level decisions.

The Dual-Write Problem

There is still a reliability issue: the application updates the database and publishes to RabbitMQ as separate operations.

If one operation succeeds and the other fails, the two systems can become inconsistent. The exact problem depends on the order of the writes.

For example, the database might save a job successfully, but publishing the message might fail. Now the job exists, but no message tells a worker to process it.

Alternatively, a message might be published even though the corresponding database transaction fails. A worker could receive a message for a job that doesn't exist.

In simple terms:

Two systems need to stay in sync, but there is no single transaction covering both.

This is the dual-write problem.

And this is where the Transactional Outbox pattern helps.

3. Transactional Outbox + RabbitMQ

The Transactional Outbox pattern addresses the dual-write problem by storing the business change and the event that needs to be published in the same database transaction.

Instead of writing directly to RabbitMQ as part of the request flow, the application first records an event in an Outbox table. A separate publisher sends that event to RabbitMQ.

The architecture becomes:

API → DB + Outbox → RabbitMQ → Worker

When a report export is requested, the application creates both the job and its Outbox event in a single transaction.

Conceptually:

text
BEGIN TRANSACTION

    Save job
    Save outbox event

COMMIT

If the transaction succeeds, both records are committed. If it fails, neither record is committed.

The complete flow looks like this:

text
              ┌─ Job
API → Database ┤
              └─ Outbox Event
                     ↓
              Outbox Publisher
                     ↓
                 RabbitMQ
                  ↙  ↓  ↘
              Worker Worker Worker

Here's how it works:

  1. The API receives a request to export a report.
  2. The application creates the job and an Outbox event in the same database transaction.
  3. A separate Outbox publisher reads pending events.
  4. The publisher sends the events to RabbitMQ.
  5. A worker consumes the message and processes the job.
  6. The publisher marks the Outbox event as published after successful delivery.

If RabbitMQ is temporarily unavailable, the event remains in the Outbox table. The publisher can retry later instead of losing the event simply because the broker was unavailable when the API request arrived.

The important difference is that the job and the intention to publish its event cannot be committed independently.

Advantages

  • Job and Outbox event are committed together.
  • Failed publishing can be retried from the Outbox.
  • Reduces the risk of losing events between the database and broker.

Limitations

  • Requires an Outbox table and a separate publisher.
  • Adds retry, cleanup, and monitoring responsibilities.
  • Duplicate messages still require idempotent consumers.

The Trade-off

The Outbox pattern makes event publication more reliable, but it doesn't guarantee that a message will be delivered or processed exactly once.

Consider this scenario:

  1. The publisher sends a message to RabbitMQ.
  2. RabbitMQ accepts the message.
  3. The publisher crashes before marking the Outbox event as published.
  4. After restarting, the publisher retries the same event.

The message may be published twice.

This is expected behavior in systems that retry delivery after uncertain outcomes.

Transactional Outbox provides a reliable path toward at-least-once delivery, not exactly-once processing.

Consumers must be idempotent, meaning that processing the same message more than once shouldn't repeat the business operation.

For example, if a message triggers a payment or updates an order, the consumer needs a way to recognize that the operation has already been applied.

The Outbox pattern introduces more components and operational work, so it isn't necessary for every background job. It's most useful when a database change must reliably result in an event being published.

4. What Production Adds

Regardless of which architecture you choose, a production system needs more than just a queue and a worker.

A few things become especially important as workloads grow:

  • Retries and backoff: Retry temporary failures without repeatedly overwhelming a failing dependency.
  • Dead-letter handling: Isolate messages that cannot be processed successfully after the allowed retries.
  • Idempotency: Prevent duplicate deliveries from repeating the same business operation.
  • Monitoring: Track queue depth and processing latency to identify growing backlogs.
  • Alerting: Detect stuck jobs and Outbox events that remain unpublished for too long.

For example, a queue might contain thousands of jobs, but that doesn't necessarily mean the system is healthy. Workers could have stopped progressing, or jobs might be failing repeatedly.

Monitoring helps distinguish a normal backlog from a processing problem before it affects users.

5. When Should Each Approach Be Used?

DB Polling

Choose DB polling when workloads are relatively small, polling delays are acceptable, and simplicity is the priority.

It is often a good starting point for scheduled tasks, report generation, and basic background processing.

Think: Simple background processing.

DB + RabbitMQ

Introduce RabbitMQ when workers need to scale independently, jobs arrive in bursts, or frequent polling becomes expensive.

The queue helps absorb sudden increases in workload while allowing workers to process jobs at their own pace.

Think: Scale and decoupling.

Transactional Outbox + RabbitMQ

Use the Transactional Outbox pattern when a database change must reliably trigger an asynchronous event and losing that event is not acceptable.

It is particularly useful in event-driven systems where services depend on changes made by other services.

Think: Reliable event publication.

6. Comparing the Three Approaches

DB Polling

  • Complexity: Low; no additional messaging infrastructure.
  • Latency: Depends on the polling interval.
  • Burst handling: Limited by database capacity and polling frequency.
  • Consistency: Jobs are stored directly in the database, without a separate broker.
  • Best for: Simple background jobs with moderate workloads.

DB + RabbitMQ

  • Complexity: Medium; requires a message broker.
  • Latency: Usually lower because workers receive messages as they arrive.
  • Burst handling: The queue buffers incoming jobs while workers catch up.
  • Consistency: Separate database and broker operations introduce a dual-write risk.
  • Best for: Growing workloads that need independent worker scaling.

Transactional Outbox + RabbitMQ

  • Complexity: Higher; requires an Outbox table and publisher.
  • Latency: Depends on how frequently the Outbox publisher runs.
  • Burst handling: RabbitMQ buffers jobs after publication.
  • Consistency: The job and Outbox event are committed in the same database transaction.
  • Best for: Workflows where reliable event publication is important.

No single approach is best for every workload. The right choice depends on the latency, scale, and reliability the system needs.

7. The Progression

DB Polling → Gets You Started

Simple background processing without additional infrastructure.

RabbitMQ → Helps You Scale

Decoupled workers, workload buffering, and better handling of unpredictable traffic.

Outbox + RabbitMQ → Makes the Handoff More Reliable

A safer way to connect database changes with asynchronous events, while accounting for duplicate delivery.

The important part isn't choosing the most advanced architecture from day one.

It's understanding where the current approach starts hurting and what the next architectural step actually solves.

Architecture isn't about adding more and more components. It's about introducing the right complexity at the right time while keeping scalability, reliability, and future growth in mind.