Litestar Queues¶
Async task queues, supervised workers, cron schedules, realtime SSE & WebSocket event streams, and decoupled storage & execution backends for Litestar.
Decoupled Storage & Execution Architecture¶
litestar-queues separates where task state lives (BaseQueueBackend) from how execution is triggered (BaseExecutionBackend). External transports and cloud dispatchers carry only an ID-only {"version": 1, "task_id": "<uuid>"} envelope—never serializing task arguments, retry counters, or results onto the wire.
Platform Capabilities¶
-
Decoupled Task Storage
Store authoritative task records in SQLSpec (11 database dialects), Advanced Alchemy (SQLAlchemy 2.0), Redis 8.1+, Valkey 6+, cross-process Ephemeral SQLite, or in-memory storage.
-
Serverless & Broker Transports
Route interactive tasks to Google Cloud Tasks OIDC webhooks, heavy batch jobs to Cloud Run Jobs, or distribute work through Pub/Sub, AWS SQS, RabbitMQ, and Apache Kafka.
-
Flexible Worker Placements
Start with zero-config CLI-supervised workers (
placement="server"), run in-process for tests (placement="asgi"), or scale dedicated external worker fleets (placement="external"). -
Realtime SSE & WebSocket Streams
Emit structured progress percentages, stage updates, and logs from inside running tasks to durable
QueueEventLoghistory and live/queues/eventsstreams. -
Cron & Interval Schedules
Declare recurring tasks with 5-field cron expressions or fixed intervals, timezone awareness, catch-up policies, and fenced leader claims.
-
Fenced Leases & Uniqueness
Prevent duplicate execution and zombie worker overwrites with
(expected_retry_count, expected_execution_ref)CAS fencing andunique_untildeduplication windows.
Install¶
Quickstart Preview¶
Define a background task with @task, register QueuePlugin, and inject QueueService into any Litestar route handler:
"""Minimal runnable Litestar Queues quickstart application."""
from uuid import UUID
from litestar import Litestar, get, post
from litestar.di import NamedDependency
from litestar.exceptions import NotFoundException
from litestar.params import FromPath, FromQuery
from litestar_queues import QueueConfig, QueuePlugin, QueueService, task
@task(queue="reports")
async def render_report(report_id: str, requested_by: str) -> dict[str, str]:
"""Render an analytics report in the background."""
return {"report_id": report_id, "requested_by": requested_by, "status": "ready"}
@post("/reports/{report_id:str}")
async def start_report(
report_id: FromPath[str],
queue_service: NamedDependency[QueueService],
requested_by: FromQuery[str] = "team@example.com",
) -> dict[str, str]:
"""Enqueue a background report job and return its tracking ID."""
handle = await queue_service.enqueue(render_report, report_id, requested_by=requested_by)
return {"task_id": str(handle.id), "status": handle.status or "pending"}
@get("/tasks/{task_id:uuid}")
async def get_report_status(task_id: FromPath[UUID], queue_service: NamedDependency[QueueService]) -> dict[str, object]:
"""Inspect the status and return value of a queued task."""
record = await queue_service.get_task(task_id)
if record is None:
raise NotFoundException(detail=f"Task {task_id} not found")
return {
"task_id": str(record.id),
"status": record.status,
"is_terminal": record.is_terminal,
"result": record.result,
}
app = Litestar(route_handlers=[start_report, get_report_status], plugins=[QueuePlugin(config=QueueConfig())])
Run your application with the Litestar CLI—QueuePlugin automatically supervises a background worker alongside your web server: