Retries and background work

Networks retry. Clients time out after the server commits. Workers crash after an external service accepts a call. Design commands and deferred work around those facts instead of treating retries as exceptional.

Make commands idempotent

Use run_idempotently() around commands that callers or infrastructure can retry. Scope the caller-provided key to an actor or tenant, then store three values in the same transaction as the command:

the scoped key;
a stable fingerprint of the validated input;
enough of the original success to reproduce its body, status, and headers.

Tenchi applies these outcomes through the application-supplied IdempotencyStore:

Existing record	Result
No record	Claim the key, perform the command, store the response, and commit together
Same key and fingerprint	Return the original successful response
Same key and different fingerprint	Raise the declared `IDEMPOTENCY_CONFLICT` error
Same key and matching work still active	Raise the declared `IDEMPOTENCY_IN_PROGRESS` error

Build the fingerprint from validated data:

python

from app.features.tasks.schemas import CreateTask
from tenchi.idempotency import fingerprint


request_fingerprint = fingerprint(request, annotation=CreateTask)

The repository must claim the key with a unique constraint and create the resource in one transaction. A process-local lock or cache does not protect multiple workers and loses history on restart.

Decide and document a retention period. Do not delete an idempotency record while callers can still legitimately retry its key.

See Make operations safe to retry for the store protocol, typed replay, expiration, cancellation, contract errors, and client retry workflow.

Commit deferred effects with state

Do not send email, publish a message, or call a webhook between a database write and commit. If the process fails in that gap, application state and the external effect disagree.

Declare the message and keep the outbox port transport-shaped:

python

from typing import Protocol

from tenchi.jobs import job


member_added_job = job(
    "projects.member_added",
    request=MemberAdded,
    result=None,
)


class Outbox(Protocol):
    async def enqueue(self, *, job: str, payload_json: bytes) -> None: ...

The concrete outbox adapter writes through the same transaction as the domain repositories. The use case changes state and enqueues the work before returning:

python

from tenchi.jobs import job_message


saved = await context.projects.save(updated)
message = job_message(
    member_added_job,
    MemberAdded(
        project_id=saved.id,
        project_name=saved.name,
        user_id=request.user_id,
    ),
)
await context.outbox.enqueue(
    job=message.name,
    payload_json=message.payload_json,
)
return saved

job_message() validates the producer value and serializes compact JSON before the transaction stores it. The payload carries the facts needed for delivery; a worker should not have to re-read mutable state merely to reconstruct what happened.

An outbox does not make an external call exactly once

The outbox makes the state change and work record atomic. Delivery is normally at least once: a worker can crash after the external system accepts the call but before the outbox acknowledgement commits. Use an idempotency key at the downstream service or make the consumer idempotent.

Validate work at the worker boundary

Bind job declarations to plain async use cases at the composition root:

python

from app.features.projects.jobs import member_added_job
from app.features.projects.use_cases.notify_member_added import (
    notify_member_added,
)
from tenchi.jobs import create_job_dispatcher, job_group, job_handler


jobs = job_group(job_handler(member_added_job, notify_member_added))
dispatcher = create_job_dispatcher(jobs=jobs)


await dispatcher.dispatch(
    entry.job,
    payload_json=entry.payload_json,
    context=context,
)

job_handler() checks request and return annotations at composition. dispatch() rejects unknown names, validates stored JSON, invokes the handler, and validates its result. The worker still owns acknowledgement and retry behavior; the dispatcher deliberately does not choose a queue, backoff policy, or dead-letter store. See Background jobs for the complete producer, composition, dispatch, and observer API.

Classify failures before retrying

Use three terminal paths for each delivery attempt:

Outcome	Transaction and queue action
Delivered	Commit application writes and acknowledgement together
Deterministic failure	Roll back partial writes, then dead-letter with a bounded error
Transient infrastructure failure	Roll back the claim, apply bounded backoff, and retry

Unknown job names, ValidationError, and JobResultError are deterministic for the same record. Classify each AppError code by its semantics: a rejected business rule may be permanent, while a declared rate-limit error may be retryable after its advertised delay. Retrying permanent failures forever blocks useful work behind a poison message. Connection failures and dependency timeouts may also be transient.

Set a maximum attempt count or age for transient failures, add jitter to backoff, and alert on queue age, repeated retries, and dead-letter growth. Preserve enough metadata to replay a corrected handler safely.

Instrument every non-HTTP entrypoint

Pass use_case_observers= to create_job_dispatcher() to report the operation, duration, result category, and stable application error code. Job outcomes use the "job" entrypoint. Workers still own attempt counts, queue latency, and correlation metadata because those values belong to the transport. Carry trace or request correlation in the message envelope, not inside business payloads that should remain stable.

See Workers and scripts for the execute() failure taxonomy and context rules, and Observability and audit for telemetry conventions.