Redis and Valkey backends

Redis and Valkey store queue records in a shared service and use pub/sub hints to wake workers. Choose the client that matches the service your application operates.

pip install "litestar-queues[redis]"
# or: pip install "litestar-queues[valkey]"

from litestar_queues import QueueConfig, WorkerConfig
from litestar_queues.backends.redis import RedisBackendConfig

queue_config = QueueConfig(
    queue_backend=RedisBackendConfig(
        url="redis://localhost:6379/0",
        key_prefix="myapp:queues",
        worker_wakeups=True,
    ),
    execution_backend="local",
    worker=WorkerConfig(placement="external"),
)

Use ValkeyBackendConfig from litestar_queues.backends.valkey for Valkey. Both accept the same URL-shaped connection syntax, but Valkey uses the Valkey client and does not require Redis as an import side effect.

Note

Performance

Install hiredis alongside redis for a compiled response parser: pip install "redis[hiredis]", or add hiredis to your dependencies directly. redis-py uses it automatically when it is importable; no configuration change is required. The backend requires a client that supports pipelines and EVAL; both redis-py and valkey-py provide this.

Typical pickup time

On a local development machine with an idle worker already running, 95 out of 100 tasks started within about 4.3 ms with Redis and 4.6 ms with Valkey. These are useful expectations, not guarantees: network distance, machine load, task volume, and service configuration can all increase pickup time.

Payloads and key isolation

Task arguments, keyword arguments, metadata, results, and errors must be JSON serializable. Give each application and environment a distinct key_prefix; do not use FLUSHALL for test cleanup on shared infrastructure.

Maintenance index upgrade

Positive maintenance limits use versioned indexes so each run examines only the requested number of records. A new or empty key_prefix initializes these indexes automatically. Index version 3 includes the dispatch-repair indexes used to revisit missing Cloud Tasks deliveries fairly.

When upgrading a populated prefix created before this release, ordinary backend opening fails closed until the indexes are rebuilt. Stop every Redis or Valkey queue writer using that prefix, then run the rebuild once as a standalone script. queue_config is the configuration from the example above:

import asyncio

async def rebuild_indexes() -> None:
    backend = queue_config.get_queue_backend()
    try:
        rebuilt = await backend.rebuild_maintenance_indexes()
        print(f"reindexed {rebuilt} queue records")
    finally:
        await backend.close()


asyncio.run(rebuild_indexes())

The return value is the number of queue records examined. The rebuild is explicit, unbounded, and idempotent, so an interrupted call is safe to repeat. Run it before opening a QueueService: entering that service would hit the old-version guard before the rebuild could run. The rebuild invalidates the old marker first and publishes version 3 only after rebuilding every index. Restart the writers only after it completes.

Worker wakeups

worker_wakeups=True publishes non-durable worker hints. Workers still poll the stored queue state. wakeup_channel and queue key prefixes belong to queue operations, not browser Channels.

Event history and live delivery

Backend-managed event history is supported. You choose how long Redis or Valkey keeps history, what it backs up, and when it removes old records. The library cannot make an otherwise temporary service durable.

History becomes eligible for live delivery after the record hash and its query indexes are acknowledged. Retrying the same event ID preserves the first record and repairs missing index entries; conflicting content is rejected. This does not add an fsync or replication guarantee to the service’s own configuration. See Event history for buffering and failure behavior.

A Redis or Valkey queue backend does not automatically send events to browsers. For standalone workers or multiple web processes, configure a shared Channels backend with its own key prefix. Redis pub/sub is temporary; Redis Streams can keep a backlog. Protect stream access at the Litestar route.