Cloud Run Deployment#

This guide shows a general Cloud Run setup for litestar-queues. Replace the project, region, service, job, and import-path placeholders with your values.

Choosing a topology#

There are four ways to run queued work on Cloud Run. They differ in what has to stay running, how long a task may take, and what happens when nothing is happening. Picking one is worth a few minutes: changing later means changing both your deployment and your task timeouts.

Topology	What stays running	Task length	Use it when
In-process worker	The API service, warm and unthrottled	Bounded by the instance	Low volume, few services, and you accept work stalling whenever the service scales down.
External dispatcher with Cloud Run Jobs	A dedicated dispatcher, always on	Hours	Tasks are long, heavy, or need their own image and resources.
Cloud Tasks direct to a private Cloud Run service	Nothing	Up to 30 minutes	Work is HTTP-shaped and finishes in minutes. Everything scales to zero between deliveries. See Cloud Tasks Deployment.
Eventarc Advanced direct to a Cloud Run Job	Nothing	Hours	Not implemented, and not planned as part of this backend. Listed only so it is not mistaken for a supported option.

The first two are covered below. The third has its own guide, because it is a different execution backend rather than a different way to deploy this one.

The core model#

A Cloud Run deployment has three separate responsibilities:

        flowchart LR
    web["Web service<br/>(your API)"]
    store[("Queue store<br/>(shared DB)")]
    dispatcher["Dispatcher<br/>worker loop"]
    job["Cloud Run Job<br/>executes task"]

    web -- "enqueue() writes a pending row" --> store
    store -- "a running worker reads pending rows" --> dispatcher
    dispatcher -- "run_job" --> job
    dispatcher -. "dispatch()" .-> store

Each part has one job:

enqueue() writes a pending queue record.
A running worker loop calls dispatch().
The Cloud Run Job container executes the task.

The dispatcher is the worker loop that starts Cloud Run Jobs. Without a running dispatcher, records routed to cloudrun remain pending.

Because the web service and dispatcher are separate processes, live browser events need a shared Channels backend configured in both. Queue wakeups do not carry SSE or WebSocket events. See SSE and WebSockets.

Topology and security#

Component	Required shared state	Security boundary
Web service	Persistent queue backend	Authenticate enqueue routes and protect queue credentials.
Dispatcher service	Same queue backend and Cloud Run configuration	Grant only `run.jobs.runWithOverrides`-equivalent access.
Cloud Run Job	Same queue backend and task modules	Use a dedicated service account and least-privilege data access.
Browser event replicas	Explicit shared Channels transport	Authorize stream scopes; queue wakeups are not browser events.

Recommended topology#

Use one web service, one always-on dispatcher service, and one Cloud Run Job that runs the queued task.

The web service should use WorkerConfig(placement="external") so it only enqueues records. The dispatcher service owns the worker loop.

from sqlspec.adapters.asyncpg import AsyncpgConfig

from litestar_queues import QueueConfig, WorkerConfig, task
from litestar_queues.backends.sqlspec import SQLSpecBackendConfig
from litestar_queues.execution.cloudrun import CloudRunExecutionConfig

# Any SQLSpec adapter config works; this one points at Cloud SQL for PostgreSQL.
sqlspec_config = AsyncpgConfig(
    connection_config={"dsn": "postgresql://queue@127.0.0.1:5432/appdb"}
)


@task("reports.render", execution_backend="cloudrun", execution_profile="heavy")
async def render_report(report_id: str) -> None:
    print(f"rendering {report_id}")


queue_config = QueueConfig(
    queue_backend=SQLSpecBackendConfig(sqlspec_config=sqlspec_config),
    execution_backend=CloudRunExecutionConfig(
        project_id="my-project",
        region="my-region",
        job_name="my-worker-job",
        profiles={"heavy": "my-worker-job-heavy"},
    ),
    worker=WorkerConfig(placement="external"),
    task_modules=("myapp.tasks",),
)

The web service, dispatcher, and Cloud Run Job must all reach the same queue backend. The dispatcher needs CloudRunExecutionConfig to choose the Cloud Run Job for each record.

Example deployment commands:

# Web service: accepts requests and enqueues records
gcloud run deploy my-api \
  --image REGION-docker.pkg.dev/my-project/my-repo/my-api:TAG \
  --region my-region \
  --add-cloudsql-instances my-project:my-region:my-db

# Dedicated dispatcher: keeps polling and dispatching records
gcloud run deploy my-queue-dispatcher \
  --image REGION-docker.pkg.dev/my-project/my-repo/my-api:TAG \
  --region my-region \
  --command litestar \
  --args queues,run \
  --min-instances 1 \
  --no-cpu-throttling \
  --add-cloudsql-instances my-project:my-region:my-db

The dispatcher must keep CPU available while it waits for tasks. Therefore, the recommended command uses both --min-instances 1 and --no-cpu-throttling.

In-app worker alternative#

You can run the dispatcher in the API process. This uses fewer services, but background polling works only while the service stays warm and has CPU.

from sqlspec.adapters.asyncpg import AsyncpgConfig

from litestar_queues import QueueConfig, WorkerConfig
from litestar_queues.backends.sqlspec import SQLSpecBackendConfig
from litestar_queues.execution.cloudrun import CloudRunExecutionConfig

sqlspec_config = AsyncpgConfig(
    connection_config={"dsn": "postgresql://queue@127.0.0.1:5432/appdb"}
)

queue_config = QueueConfig(
    queue_backend=SQLSpecBackendConfig(sqlspec_config=sqlspec_config),
    execution_backend=CloudRunExecutionConfig(
        project_id="my-project",
        region="my-region",
        job_name="my-worker-job",
    ),
    worker=WorkerConfig(placement="asgi"),
)

gcloud run deploy my-api \
  --image REGION-docker.pkg.dev/my-project/my-repo/my-api:TAG \
  --region my-region \
  --min-instances 1 \
  --no-cpu-throttling \
  --add-cloudsql-instances my-project:my-region:my-db

Use this only for low-volume deployments. A dedicated dispatcher is the safer default because web and background capacity can scale separately.

Cloud Run Job worker#

The Cloud Run Job executes the queued task. Run it with litestar queues run-task. The command reads the task id and QUEUES_CONFIG_FACTORY from the container environment, resolves the shared queue service, re-fetches the saved record, claims it, runs the task, and exits with a defined code.

The command resolves the config factory before reading the prefixed task id, so a custom env_prefix works throughout the process.

Environment contract#

The following names use the default env_prefix value, LITESTAR_QUEUES. If you change CloudRunExecutionConfig.env_prefix, the prefix changes everywhere.

Env var	Required	Meaning
`QUEUES_CONFIG_FACTORY`	Yes	`module:attr` or dotted path returning the shared-DB `QueueConfig` or `QueueService`. Without it, the consumer cannot rebuild the queue service from the job container.
`LITESTAR_QUEUES_TASK_MODULES`	Recommended	Comma-separated modules to import so `@task` registrations exist. The consumer merges these with `config.task_modules`.
`QUEUES_TASK_ID`	Injected	The id (UUID) of the queued record to run. The dispatcher sets this automatically through container overrides. The consumer re-fetches the live record by this id, which stays the source of truth for its args, result, and retry state.

Run one job locally#

The run-task options override the environment so you can run a single saved record by hand – useful for reproducing a failure or draining a stuck job. Pass the record id and a config factory that returns the same shared-DB service; no environment variables are required:

litestar queues run-task \
  --task-id 0f9c1e2a-7b34-4c56-8d90-1a2b3c4d5e6f \
  --config-factory myapp.queues:create_config

--task-modules imports extra modules before the task runs. The command exits with the same deterministic codes as the injected Cloud Run invocation.

Profile-based job selection#

Use execution_profile to send different task families to different Cloud Run Jobs. The dispatcher chooses the job name in this order:

profiles[record.execution_profile]
job_name
profiles["default"]

If no job name can be resolved, dispatch fails before any Cloud Run execution is created.

Example job command:

gcloud run jobs create my-worker-job \
  --image REGION-docker.pkg.dev/my-project/my-repo/my-worker:TAG \
  --region my-region \
  --command litestar \
  --args queues,run-task \
  --set-cloudsql-instances my-project:my-region:my-db \
  --set-env-vars QUEUES_CONFIG_FACTORY=myapp.queues:create_config,\
  LITESTAR_QUEUES_TASK_MODULES=myapp.tasks \
  --task-timeout 3600s

Set the timeout for the longest expected task. The Cloud Run Job timeout must be at least as long as the queue timeout. Using the same value for both is the simplest option.

If you need a different prefix for the worker environment variables, set it on CloudRunExecutionConfig.env_prefix and use the matching prefix when you deploy the Job.

IAM#

The dispatcher or API service account must be allowed to run the worker Job with container overrides. The simplest grant is roles/run.developer on that Job because it includes run.jobs.runWithOverrides.

gcloud run jobs add-iam-policy-binding my-worker-job \
  --region my-region \
  --member serviceAccount:my-dispatcher-sa@my-project.iam.gserviceaccount.com \
  --role roles/run.developer

If you use a custom IAM role instead, make sure it includes run.jobs.runWithOverrides.

To cancel a running execution, the dispatcher’s service account additionally needs ``run.executions.cancel`` on the job’s executions. It is not implied by the run-job permission. The predefined role carrying it is Cloud Run Admin (roles/run.admin); a least-privilege deployment grants a custom role holding run.jobs.run, run.executions.get (already required for reconciliation) and run.executions.cancel.

If this permission is missing, cancellation maps the permission error to retryable, so cancel_task() returns False and the record is deliberately left running rather than being marked cancelled while the container keeps running. The warning log and the litestar_queues.execution.cancel{queue.execution.status="retryable"} counter are the signal.

The resource name used for cancellation is the stored execution reference, so cancellation targets the exact execution this package created — including one created by a different execution profile’s job.

Database connectivity#

The web service, dispatcher, and Job must all reach the same queue database. Creating the Cloud Run Job does not provide that connection.

If you use Cloud SQL:

attach the instance to the web service,
attach the same instance to the dispatcher service, and
attach the same instance to the Job.

Example:

gcloud run deploy my-api \
  --image REGION-docker.pkg.dev/my-project/my-repo/my-api:TAG \
  --region my-region \
  --add-cloudsql-instances my-project:my-region:my-db

gcloud run deploy my-queue-dispatcher \
  --image REGION-docker.pkg.dev/my-project/my-repo/my-api:TAG \
  --region my-region \
  --command litestar \
  --args queues,run \
  --add-cloudsql-instances my-project:my-region:my-db

gcloud run jobs create my-worker-job \
  --image REGION-docker.pkg.dev/my-project/my-repo/my-worker:TAG \
  --region my-region \
  --set-cloudsql-instances my-project:my-region:my-db

Use the connection format required by your driver. All three processes must point to the same persistent queue store.

Dispatch failure behavior#

By default, Litestar Queues reports dispatch failures in logs and events. The record remains routed to cloudrun. The default fallback backend is None.

If you want the older reroute behavior, set fallback_execution_backend="local" explicitly and run a local worker that can consume the fallback queue.

A temporary Cloud Run status-check failure is treated as still running. This prevents a false final state. If the remote execution is missing, the status check marks the record failed so retryable work can run again.

Scheduling#

Scheduled work still needs a dispatcher. Choose one of these options:

use @task(schedule=...) so the dispatcher creates the pending records on a schedule, or
use Cloud Scheduler to call an endpoint that enqueues the task.

In both cases, enqueueing only writes the record. The dispatcher is what turns that record into a Cloud Run Job execution.

Troubleshooting#

Jobs never trigger - check the sequence in order: * the dispatcher service is running, * the record’s execution backend is cloudrun, * the dispatcher uses a typed CloudRunExecutionConfig, * the job name resolves from execution_profile or job_name, * the dispatcher service account can run the Job with overrides, and * the web service, dispatcher, and Job all point at the same database.
``Extension litestar_queues not found`` appears during SQLSpec startup - this is usually a SQLSpec extension-discovery warning. Verify that the queue migration path is registered correctly, and do not assume the warning means the queue schema is absent.
The worker exits with a non-zero code - the exit codes are deterministic. The most useful ones are 4 for missing record, 6 for claim lost, 7 for cancellation, and 8 for missing CONFIG_FACTORY.