Observability and audit

Observe every boundary with stable operation names, trusted correlation IDs, bounded-cardinality fields, and explicit redaction. Keep operational telemetry separate from durable audit records.

Export outcomes with OpenTelemetry

Install Tenchi's optional API integration and an OpenTelemetry SDK:

uv add "tenchi[otel]" opentelemetry-sdk

Configure the SDK, resource, readers, span processors, and exporter for your deployment before creating the observers. Tenchi deliberately does not choose an exporter or set process-global providers. The OpenTelemetry Python exporter guide shows console and OTLP configurations.

from tenchi.client import Client
from tenchi.opentelemetry import create_opentelemetry_observers
from tenchi.server import create_app


telemetry = create_opentelemetry_observers()

app = create_app(
    routes=routes,
    context_factory=create_context,
    observers=(telemetry.request,),
    use_case_observers=(telemetry.use_case,),
)

async with Client(
    base_url="https://inventory.example.com",
    observers=(telemetry.client,),
    attempt_observers=(telemetry.client_attempt,),
) as client:
    inventory = await client.call(get_inventory_contract)

The factory uses the process-global providers by default. Pass tracer_provider= or meter_provider= when the application keeps providers explicit instead:

telemetry = create_opentelemetry_observers(
    tracer_provider=tracer_provider,
    meter_provider=meter_provider,
)

Use the same telemetry.use_case observer with every non-HTTP entrypoint:

from tenchi.execution import execute
from tenchi.jobs import create_job_dispatcher
from tenchi.tasks import create_task_runner


dispatcher = create_job_dispatcher(
    jobs=jobs,
    use_case_observers=(telemetry.use_case,),
)

runner = create_task_runner(
    tasks=tasks,
    context_factory=create_context,
    use_case_observers=(telemetry.use_case,),
)

await execute(
    rebuild_index,
    context=create_context,
    use_case_observers=(telemetry.use_case,),
)

The integration emits the following instruments. Each duration histogram also provides an event count through its histogram count; the explicit counters make throughput queries independent of histogram configuration.

Metric dimensions are limited to static contract or use-case names, declared path templates, HTTP methods and statuses, bounded outcome classifications, entrypoint names, error ownership, and the retry decision. Application error codes, attempt numbers, maximum attempts, and selected retry delays are useful on individual spans but are intentionally excluded from metrics.

The observer spans are completed INTERNAL logical-operation spans. They inherit the current trace context and use each outcome's UTC completed_at timestamp plus Tenchi's measured duration, but they are not active while application or transport work runs. Use standard ASGI, httpx, database, and queue instrumentation for active transport spans and automatic context propagation. The active transport spans and Tenchi's finalized logical spans then describe complementary parts of the same trace.

The integration depends only on opentelemetry-api. Without an application-configured SDK, its providers are no-ops. The application owns provider flushing and shutdown in its process lifespan.

Write a custom outcome observer

Register a synchronous or asynchronous observer with create_app():

import logging

from tenchi.client import Client, ClientOutcome
from tenchi.execution import UseCaseOutcome
from tenchi.server import RequestOutcome, create_app


logger = logging.getLogger("app.requests")


def observe_request(outcome: RequestOutcome) -> None:
    logger.info(
        "request.complete",
        extra={
            "operation": outcome.request.contract.name,
            "method": outcome.request.method,
            "status_code": outcome.status_code,
            "duration_seconds": outcome.duration_seconds,
            "completed_at": outcome.completed_at.isoformat(),
            "error_source": outcome.error_source,
            "request_id": outcome.request.request_id,
        },
    )


def observe_use_case(outcome: UseCaseOutcome) -> None:
    operation = str(
        getattr(outcome.use_case, "__name__", type(outcome.use_case).__name__)
    )
    logger.info(
        "use_case.complete",
        extra={
            "operation": operation,
            "entrypoint": outcome.entrypoint,
            "status": outcome.status,
            "duration_seconds": outcome.duration_seconds,
            "completed_at": outcome.completed_at.isoformat(),
            "error_code": outcome.error_code,
        },
    )


def observe_client(outcome: ClientOutcome) -> None:
    logger.info(
        "client.complete",
        extra={
            "operation": outcome.contract.name,
            "status": outcome.status,
            "status_code": outcome.status_code,
            "duration_seconds": outcome.duration_seconds,
            "completed_at": outcome.completed_at.isoformat(),
            "error_code": outcome.error_code,
        },
    )


app = create_app(
    routes=routes,
    context_factory=create_context,
    observers=[observe_request],
    use_case_observers=[observe_use_case],
)

The outcome arrives after the request context has closed and the response is finalized. It therefore describes the status the caller receives and includes time spent committing or rolling back the request scope. completed_at records when that measured boundary finished even if an earlier observer delays delivery.

Observer failures are logged and isolated from the response and later observers. Keep observers fast: they still run before the ASGI endpoint returns. Export through a non-blocking logging or telemetry pipeline.

Observe use cases across entrypoints

UseCaseOutcome reports every use case that was actually invoked through HTTP, a background job, an operational task, an application tool, or execute():

Pass the observer to create_app(use_case_observers=...) for HTTP, execute(use_case_observers=...) for workers, schedules, scripts, and direct calls, or create_task_runner(use_case_observers=...) for operational tasks. Background dispatchers accept the same observer through create_job_dispatcher(use_case_observers=...). Application tool runners use create_tool_runner(use_case_observers=...). It runs after the surrounding context scope closes, so transactional cleanup finishes before telemetry is exported. Input values, return values, and exception objects are never included.

Keep use-case observers fast too. HTTP, tasks, tools, and execute() all wait for the observer chain after scope cleanup.

Validation, hook, and context-acquisition failures that happen before invocation do not produce a use-case outcome. Conversely, a use case that returns successfully still reports succeeded if later response validation or transaction cleanup makes the HTTP request fail. Compare it with RequestOutcome to distinguish application behavior from the complete HTTP boundary.

app_error means the use case raised AppError; it does not bypass Tenchi's error honesty rule. An undeclared application error still becomes a framework-owned HTTP 500 in RequestOutcome.

When an HTTP deadline cancels the function, the use-case status is cancelled and the request status is 504. If the function catches that cancellation and returns, the use-case status is succeeded, but the expired HTTP deadline still owns the response and returns 504.

Observe outbound contract calls

ClientOutcome reports one finalized outcome for every typed client call:

async with Client(
    base_url="https://service.example.com",
    observers=[observe_client],
) as client:
    inventory = await client.call(get_inventory_contract)

The outcome is emitted for local validation failures before any request leaves the process as well as for completed transport attempts. A response whose status or wire data violates the contract reports unexpected_response. failed covers failures without a response that are neither an httpx transport error nor cancellation, such as invalid local input or contract configuration.

Client outcomes contain no input, URL, header, body, response object, or exception payload. Use outcome.contract.name, status classes, and the bounded status classification as metric dimensions. Keep full network spans in httpx or OpenTelemetry instrumentation, with an explicit redaction policy.

Client observers run in declaration order before the call returns or raises. Their failures are logged and isolated from later observers and from the caller.

When a call uses an explicit retry policy, pass attempt_observers= to receive one payload-safe ClientAttemptOutcome per attempt. It adds the attempt number, selected retry delay, will_retry decision, and the attempt's UTC completed_at. Use the logical outcome for availability and latency; use attempt outcomes for retry pressure.

Use bounded labels

Use the contract name, HTTP method, status class, and error source as metric dimensions. Do not use raw paths, resource IDs, user IDs, idempotency keys, or error messages as labels; their unbounded values can overwhelm a metrics backend.

error_source distinguishes declared application failures from framework-owned boundary failures. Record the exact application error code in structured logs when it is available at the application boundary, but keep the metric label set controlled.

Propagate correlation safely

Tenchi accepts a valid inbound x-request-id or creates one, exposes it as outcome.request.request_id, and returns it on the response. Configure the trusted edge so callers cannot inject unrelated tracing headers past your policy.

When HTTP enqueues work, copy the trusted request or trace correlation into message metadata. A worker should start or continue a trace from that metadata and attach its attempt and queue measurements around dispatcher.dispatch(). The shared use-case observer reports the application call; the consumer still owns transport-specific metadata and acknowledgement.

Choose the right instrumentation seam

HTTP request and use-case observers run only for matched Tenchi routes. ASGI middleware is the correct layer for 404s outside the composed route group and for work that continues while a streaming response body is sent.

Redact by allowlist

RequestInfo.headers is read-only, but it can contain authorization, cookies, and other credentials. Never log the complete mapping. Select known safe fields and redact values before serialization.

ClientOutcome omits headers and payloads entirely. Instrumentation below the typed client does not inherit that guarantee.

Apply the same rule to:

• request and response bodies;
• AppError.details;
• database statements and bound parameters;
• external service URLs and headers;
• job payloads and dead-letter errors;
• settings and process environment values.

Prefer stable identifiers that help correlate an incident without exposing personal data. Define retention and access controls for every telemetry sink.

Record audit events transactionally

When a business action requires an audit record, define an AuditLog port and write through the same request transaction as the state change:

from typing import Protocol


class AuditLog(Protocol):
    async def record(
        self,
        *,
        actor_id: str,
        action: str,
        subject_id: str,
    ) -> None: ...

Use stable action names and the minimum necessary subject identifiers. Do not store secrets or unrestricted before-and-after payloads. If audit records must leave the primary database, publish them through the transactional outbox pattern after the local record commits.

Alert on user-visible failure

A useful initial alert set covers:

• elevated framework-owned 5xx responses;
• latency by contract and status class;
• readiness failures;
• database pool exhaustion and transaction errors;
• worker queue age, retry rate, and dead-letter growth;
• OpenAPI or application-tool compatibility gate failures before deployment.

Page on symptoms that require action. Keep lower-level diagnostic events in logs and traces so an alert links to evidence instead of duplicating it.