> ## Documentation Index
> Fetch the complete documentation index at: https://docs.lerian.studio/llms.txt
> Use this file to discover all available pages before exploring further.
# Fetcher observability
> Fetcher health and readiness endpoints, parallel dependency probes, drain behavior on SIGTERM, circuit-breaker reporting, Prometheus metrics, and OpenTelemetry tracing.
Fetcher answers three operational questions through HTTP: is the process alive, can it serve traffic right now, and which dependency is at fault. This page covers each surface and what it reports.
## Endpoints
***
| Endpoint | Answers | Manager | Worker |
| -------------------- | ------------------------------------------------ | ------- | ------ |
| `/health` | Is the process alive and did it start cleanly? | Yes | Yes |
| `/readyz` | Can this instance serve traffic right now? | Yes | Yes |
| `/readyz/tenant/:id` | Can it serve one tenant? Multi-tenant mode only. | Yes | Yes |
| `/metrics` | Prometheus exposition. | Yes | Yes |
| `/version` | Build version. | Yes | — |
The Manager serves all of these on `SERVER_ADDRESS`. The Worker has no API server, so it runs a health micro-server on `HEALTH_PORT`, which defaults to `4007`.
All of them mount before authentication. Kubernetes and load-balancer probes need no token.
## `/health` and the startup self-probe
***
`/health` is not a static 200. At boot, Fetcher runs every dependency probe once, in parallel, then flips a process-wide flag on the result. Until that self-probe succeeds, `/health` returns 503.
**The kubelet restarts a pod whose dependencies failed at boot.** It does not send it traffic. The flag starts false, so a process that crashes mid-probe never reports healthy by accident. Point your liveness probe at `/health`.
Each dependency also emits its self-probe outcome as a metric. A repeated boot failure therefore shows on a dashboard, not only in the logs.
## `/readyz` and the dependency probes
***
`/readyz` runs every registered probe on every request, in parallel, one goroutine per dependency. The handler holds no cache and no background state. A cached answer opens a window where Kubernetes keeps routing to a degraded pod.
A healthy aggregate returns 200. Anything else returns 503. The response body reports each dependency by name, with its status, its latency, and its TLS posture.
### What each service probes
| Dependency | Manager | Worker |
| -------------------- | ----------------- | ----------------- |
| `mongodb` | Yes | Yes |
| `rabbitmq` | Yes | Yes |
| `redis` | Yes | — |
| `multi_tenant_redis` | Multi-tenant only | Multi-tenant only |
| `tenant_manager` | Multi-tenant only | Multi-tenant only |
| `s3` | — | Yes |
In multi-tenant mode, the shared MongoDB and RabbitMQ entries report `n/a` with the reason `multi-tenant: see /readyz/tenant/:id`, and the per-tenant probes move to that endpoint.
### Per-dependency timeouts
Each probe runs under a fixed deadline. The values are not configurable, so every Lerian service has the same readiness latency envelope and one dashboard threshold works across the fleet.
| Dependency class | Deadline |
| ----------------------------------------------------------------- | -------- |
| Databases (`mongodb`, `postgres`, `mysql`, `oracle`, `sqlserver`) | 2s |
| Cache (`redis`, `valkey`) | 1s |
| Queue (`rabbitmq`) | 2s |
| Storage (`s3`, `seaweedfs`) | 2s |
| HTTP upstreams (`tenant_manager`, `upstream_*`) | 1s |
A probe that ignores its deadline does not block the response. The handler substitutes a `down` result for it.
## Circuit breaker state
***
Tenant resolution runs behind a circuit breaker. The `MULTI_TENANT_CIRCUIT_BREAKER_THRESHOLD` variable sets how many consecutive failures open it, and it defaults to 5. The `MULTI_TENANT_CIRCUIT_BREAKER_TIMEOUT_SEC` variable sets how long it stays open, and it defaults to 30 seconds.
When the breaker is open, the per-tenant probe reports the dependency as `down` with the error `circuit breaker open` and a breaker state of `open`. That distinguishes a tripped breaker from an ordinary connection failure, which is the difference between waiting for a reset and paging a database owner.
## Drain on SIGTERM
***
On `SIGTERM` or `SIGINT`, both services enter a drain before they tear down connections.
1. `/readyz` short-circuits to 503 for `READYZ_DRAIN_DELAY_SEC` seconds, which defaults to 12 and has a minimum of 1.
2. Kubernetes removes the pod from Service endpoints while it still serves in-flight work.
3. Only then do connections close.
The service skips the real probes during the drain. The response carries one synthetic dependency named `draining` with status `down`, and it emits metrics like any other.
**Alerts keep rating through a rolling deploy.** The synthetic `draining` dependency keeps the metric series alive, so a dashboard shows a drain rather than a gap. Set your termination grace period above the drain window.
## Metrics
***
`/metrics` serves Prometheus exposition, including the Go runtime and process collectors.
| Metric | Type | Labels | Reports |
| -------------------------- | --------- | --------------- | --------------------------------------------------------- |
| `readyz_check_duration_ms` | histogram | `dep`, `status` | Duration of each readiness probe, in milliseconds. |
| `readyz_check_status` | counter | `dep`, `status` | Outcome count per dependency and status. |
| `selfprobe_result` | gauge | `dep` | Last startup self-probe result: `1` for up, `0` for down. |
The histogram buckets run from 1 ms to 5,000 ms. Metric names, labels, and buckets are a platform contract, and dashboards across the fleet depend on them.
The duration histogram records the wall-clock time the probe contributed to the handler, not the latency the probe reported for itself. That is the number that explains a slow `/readyz`.
## Tracing
***
Set `ENABLE_TELEMETRY=true` and point `OTEL_EXPORTER_OTLP_ENDPOINT` at your collector. Fetcher then exports OpenTelemetry traces and metrics over OTLP.
Four resource attributes shape what you see: `OTEL_RESOURCE_SERVICE_NAME` (`fetcher` on the Manager, `fetcher-worker` on the Worker), `OTEL_RESOURCE_SERVICE_VERSION`, `OTEL_RESOURCE_DEPLOYMENT_ENVIRONMENT`, and `OTEL_LIBRARY_NAME`.
The engine emits its own spans through a one-method port. A host that supplies no tracer gets a no-op, and behavior does not change.
## What to alert on
***
1. **`/readyz` 503 outside a deploy window.** One dependency name in the response body tells you which.
2. **A rising dead-letter queue.** Each message there is a job the Worker could not process. See [Deployment](/en/fetcher/fetcher-deployment).
3. **`selfprobe_result` at 0 for any dependency.** A pod restarted into a broken dependency.
4. **A breaker state of `open` on `tenant_manager`.** Tenant resolution fails, and every tenant-scoped request depends on it.
## Next steps
***
Dependencies, queues, scaling, and startup checks.
Every environment variable, per component.
Keys, signing, encryption at rest, and host validation.
Job lifecycle, terminal states, and events.