About Observability

​

Who configures what?

​

Client side

• Processors: batching, memory limits, filtering, obfuscation, sampling, etc.
• Exporters: for example, dual routing to Prometheus
• API key authentication secrets

​

Lerian side

• Central Collector
• Prometheus
• Loki
• Tempo
• Grafana

​

How the data flows

• Prometheus for metrics
• Loki for logs
• Tempo for traces

​

Stack components

​

Client Collector

​

Central Collector

​

Prometheus

​

Loki

​

Tempo

​

Grafana

​

Embedded Collector

otel-collector-lerian:
  enabled:true

​

Editing the Client Collector

1. Edit components/infra/grafana/otelcol-config.yaml.
2. Add or adjust the processors or exporters blocks.
3. Restart the stack:
   Copy

   Ask AI

   make down
   make up
   

​

Client Collector Processors

• What it is: Groups multiple telemetry signals (metrics, logs, or traces) into batches before sending them to the next stage.
• Why it matters: Improves compression efficiency, reduces network requests, and enhances overall pipeline performance.
• Configuration:

processors:
  batch: {}

• What it is: Monitors the collector’s memory usage and drops data if it approaches a defined threshold.
• Why it matters: Prevents the collector from being OOMKilled by Kubernetes.
• Configuration:

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 75
    spike_limit_percentage: 15

• What it is: Generates metrics directly from trace data.
• Why it matters: Produces “RED” metrics (Rate, Errors, Duration) automatically.
• Configuration:

processors:
  spanmetrics:
    metrics_exporter: prometheus
    dimensions:
      - name: http.method
      - name: http.status_code
      - name: service.name
      - name: client.id

• What it is: Removes sensitive span attributes using regex.
• Why it matters: Keeps identifiers but strips headers, bodies, or other sensitive request data.
• Configuration:

processors:
  transform/remove_sensitive_attributes:
    trace_statements:
      - context: span
        statements:
          - delete_matching_keys(attributes, "^app\\.request\\.(?!request_id$).*")

• What it is: A sampling strategy applied after spans are received.
• Why it matters: Keeps only high-value traces (errors, specific clients) and reduces storage costs.
• Configuration:

processors:
  tail_sampling:
    policies:
      - name: keep_firmino_traces_policy
      - name: http_server_errors_policy
      - name: drop_all_other_traces_policy

• What it is: Filters out node-level metrics.
• Why it matters: Reduces noise and focuses on app-level telemetry.
• Configuration:

processors:
  filter/drop_node_metrics:
    metrics:
      exclude:
        match_type: regexp
        metric_names:
          - ^k8s\\.node\\..*$

• What it is: Keeps only metrics from midaz and midaz-plugins.
• Why it matters: Eliminates irrelevant Kubernetes workloads.
• Configuration:

processors:
  filter/include_midaz_namespaces:
    metrics:
      include:
        match_type: regexp
        resource_attributes:
          - key: k8s.namespace.name
            value: '^(midaz|midaz-plugins)$'

• What it is: Adds Kubernetes metadata to telemetry.
• Why it matters: Enables richer context in Grafana queries.
• Configuration:

processors:
  k8sattributes:
    auth_type: "serviceAccount"
    passthrough: false
    extract:
      metadata:
        - k8s.pod.name
        - k8s.deployment.name
        - k8s.namespace.name
        - k8s.node.name

• What it is: Inserts or updates client.id in telemetry.
• Why it matters: Critical for multi-tenancy.
• Configuration:

processors:
  resource/add_client_id:
    attributes:
      - key: client.id
        value: "Firmino"
        action: upsert

• What it is: Removes log body content.
• Why it matters: Prevents sensitive or PII data from persisting in logs.
• Configuration:

processors:
  transform/remove_log_body:
    log_statements:
      - context: log
        statements:
          - set(body, "")

• What it is: Obfuscates selected attributes.
• Why it matters: Protects sensitive values (like legalDocument or accountAlias) before data leaves your cluster.
• Configuration (otelcol-config.yaml):

processors:
  transform/obfuscate_attributes:
    trace_statements:
      - context: span
        statements:
          - replace_pattern(attributes["legalDocument"], ".*", "***")
          - replace_pattern(attributes["accountAlias"], ".*", "***")
    log_statements:
      - context: log
        statements:
          - replace_pattern(attributes["legalDocument"], ".*", "***")
          - replace_pattern(attributes["accountAlias"], ".*", "***")

• Customizing the fields:
  • Defaults: legalDocument, accountAlias
  • Add or remove fields as needed
  • Restart required: make down && make up

​

In short

​

Protecting sensitive data

Client vs Lerian responsibilities in the observability pipeline

• Client Collector (you configure): Runs in your cluster. Apply processors such as transform/remove_sensitive_attributes, transform/remove_log_body, and transform/obfuscate_attributes.
• Central Collector (Lerian managed): Receives only the filtered, sanitized telemetry streams and routes them to Prometheus, Loki, and Tempo.

​

Telemetry flow

1. Application starts and detects OpenTelemetry configuration.
2. Telemetry is exported to the local Client Collector.
3. Client Collector enriches data with Kubernetes metadata and your client_id.
4. Processors enrich, filter, and transform the data.
5. Data is forwarded to the Central Collector.
6. Central Collector processes and routes data:
   • Metrics → Prometheus
   • Logs → Loki
   • Traces → Tempo
7. Grafana lets you query it all, correlating across signals.

sum(rate(http_server_duration_seconds_count{
  k8s_pod_name=~"checkout-.*",
  client_id="client-name"
}[5m]))

​

Authenticating collector requests

​

How to set it up

1. Create the Kubernetes Secret to store your API token:

kubectl create secret generic otel-api-key \
  --from-literal=OTEL_API_KEY='YOUR_TOKEN_HERE' \
  -n midaz

2. Reference the secret in your Helm values file to inject it as an environment variable:

extraEnvs:
  - name: OTEL_API_KEY
    valueFrom:
      secretKeyRef:
        name: otel-api-key
        key: OTEL_API_KEY

3. Telemetry is securely sent to Lerian’s telemetry endpoint over HTTPS, with the API key included in the headers.

<https://telemetry.lerian.io:443>

​

Data encryption in transit

• The communication between the Client Collector and the Central Collector is fully encrypted.
• Data in transit is protected against interception, tampering, or unauthorized access.
• Even if network traffic is inspected, the contents remain unreadable without the proper cryptographic keys.

​

Dual routing

​

Example

exporters:
  otlphttp/server:
    endpoint: "<https://telemetry.lerian.io:443>"
    headers:
      x-api-key: "${OTEL_API_KEY}"
  prometheus/local:
    endpoint: prometheus-server-example:8889

​

Glossary