Skip to main content
Fetcher answers three operational questions through HTTP: is the process alive, can it serve traffic right now, and which dependency is at fault. This page covers each surface and what it reports.

​
Endpoints

The Manager serves all of these on SERVER_ADDRESS. The Worker has no API server, so it runs a health micro-server on HEALTH_PORT, which defaults to 4007. All of them mount before authentication. Kubernetes and load-balancer probes need no token.

​
/health and the startup self-probe

/health is not a static 200. At boot, Fetcher runs every dependency probe once, in parallel, then flips a process-wide flag on the result. Until that self-probe succeeds, /health returns 503.
The kubelet restarts a pod whose dependencies failed at boot. It does not send it traffic. The flag starts false, so a process that crashes mid-probe never reports healthy by accident. Point your liveness probe at /health.
Each dependency also emits its self-probe outcome as a metric. A repeated boot failure therefore shows on a dashboard, not only in the logs.

​
/readyz and the dependency probes

/readyz runs every registered probe on every request, in parallel, one goroutine per dependency. The handler holds no cache and no background state. A cached answer opens a window where Kubernetes keeps routing to a degraded pod. A healthy aggregate returns 200. Anything else returns 503. The response body reports each dependency by name, with its status, its latency, and its TLS posture.

​
What each service probes

In multi-tenant mode, the shared MongoDB and RabbitMQ entries report n/a with the reason multi-tenant: see /readyz/tenant/:id, and the per-tenant probes move to that endpoint.

​
Per-dependency timeouts

Each probe runs under a fixed deadline. The values are not configurable, so every Lerian service has the same readiness latency envelope and one dashboard threshold works across the fleet. A probe that ignores its deadline does not block the response. The handler substitutes a down result for it.

​
Circuit breaker state

Tenant resolution runs behind a circuit breaker. The MULTI_TENANT_CIRCUIT_BREAKER_THRESHOLD variable sets how many consecutive failures open it, and it defaults to 5. The MULTI_TENANT_CIRCUIT_BREAKER_TIMEOUT_SEC variable sets how long it stays open, and it defaults to 30 seconds. When the breaker is open, the per-tenant probe reports the dependency as down with the error circuit breaker open and a breaker state of open. That distinguishes a tripped breaker from an ordinary connection failure, which is the difference between waiting for a reset and paging a database owner.

​
Drain on SIGTERM

On SIGTERM or SIGINT, both services enter a drain before they tear down connections.
  1. /readyz short-circuits to 503 for READYZ_DRAIN_DELAY_SEC seconds, which defaults to 12 and has a minimum of 1.
  2. Kubernetes removes the pod from Service endpoints while it still serves in-flight work.
  3. Only then do connections close.
The service skips the real probes during the drain. The response carries one synthetic dependency named draining with status down, and it emits metrics like any other.
Alerts keep rating through a rolling deploy. The synthetic draining dependency keeps the metric series alive, so a dashboard shows a drain rather than a gap. Set your termination grace period above the drain window.

​
Metrics

/metrics serves Prometheus exposition, including the Go runtime and process collectors. The histogram buckets run from 1 ms to 5,000 ms. Metric names, labels, and buckets are a platform contract, and dashboards across the fleet depend on them. The duration histogram records the wall-clock time the probe contributed to the handler, not the latency the probe reported for itself. That is the number that explains a slow /readyz.

​
Tracing

Set ENABLE_TELEMETRY=true and point OTEL_EXPORTER_OTLP_ENDPOINT at your collector. Fetcher then exports OpenTelemetry traces and metrics over OTLP. Four resource attributes shape what you see: OTEL_RESOURCE_SERVICE_NAME (fetcher on the Manager, fetcher-worker on the Worker), OTEL_RESOURCE_SERVICE_VERSION, OTEL_RESOURCE_DEPLOYMENT_ENVIRONMENT, and OTEL_LIBRARY_NAME. The engine emits its own spans through a one-method port. A host that supplies no tracer gets a no-op, and behavior does not change.

​
What to alert on

  1. /readyz 503 outside a deploy window. One dependency name in the response body tells you which.
  2. A rising dead-letter queue. Each message there is a job the Worker could not process. See Deployment.
  3. selfprobe_result at 0 for any dependency. A pod restarted into a broken dependency.
  4. A breaker state of open on tenant_manager. Tenant resolution fails, and every tenant-scoped request depends on it.

​
Next steps

Deployment

Dependencies, queues, scaling, and startup checks.

Configuration

Every environment variable, per component.

Security

Keys, signing, encryption at rest, and host validation.

Extraction jobs

Job lifecycle, terminal states, and events.