Midaz production best practices

Midaz runs in production at high volume. This guide gives you the deployment, high-availability, and observability patterns Lerian recommends. Follow them to keep downtime low and protect your data.

​

Best-fit setup

---

Start a production deployment with these choices:

• Deploy across multiple availability zones.
• Run at least 3 worker nodes with autoscaling.
• Separate application and database workloads.
• Use managed services such as RDS, ElastiCache, and MongoDB Atlas.
• Apply Kubernetes patterns for resilience, security, and observability.
• Automate backups and alerting from day one.

​

Infrastructure planning

---

​

Cluster architecture

Plan the cluster for resilience and performance:

• Deploy across multiple availability zones.
• Run at least 3 worker nodes for high availability.
• Enable node autoscaling to absorb workload spikes.
• Separate application and database workloads when possible.

​

Resource sizing

• Match node sizes to the expected workload.
• Give critical services enough resources first.
• Apply resource quotas to prevent contention.
• Monitor usage and tune the sizing over time.

​

Storage

• Use SSD-backed storage for all database components.
• Define a storage class for each cloud provider.
• Provision volumes with headroom for growth.
• Use replicated or durable storage for critical data.

​

Database architecture and high availability

---

Midaz uses CQRS (Command Query Responsibility Segregation) to separate reads from writes. This design lets you scale each path on its own.

​

PostgreSQL

• Use a dedicated primary for writes and replicas for reads.
• Enable synchronous replication for critical data.
• Configure automatic failover with Patroni or AWS RDS.
• Monitor replication lag and consistency.
• Prefer managed services such as AWS RDS or GCP Cloud SQL.

​

Redis / Valkey

• Deploy one primary with replicas across multiple zones. Midaz does not support Redis Cluster mode for transactions.
• Enable automatic failover with a Sentinel topology via REDIS_MASTER_NAME, or with a managed service with cluster mode disabled. Set REDIS_HOST to one address, or set REDIS_MASTER_NAME for Sentinel. Without REDIS_MASTER_NAME, a comma-separated REDIS_HOST selects Cluster mode. A managed service (ElastiCache, Memorystore) is the safer default.

​

MongoDB

• Use replica sets with members across zones.
• Monitor role transitions and lag.
• Schedule regular backups.
• Use managed services such as MongoDB Atlas or AWS DocumentDB.

​

Messaging infrastructure

---

Midaz runs two messaging surfaces. Kafka-compatible streaming and asynchronous transaction processing are off by default. The legacy RabbitMQ overdraft and audit publishers are enabled at runtime unless you explicitly set their flags to false. The bundled example configuration sets those flags to false:

• RabbitMQ carries the optional write-behind pipeline that stores confirmed transactions in PostgreSQL and MongoDB (RABBITMQ_TRANSACTION_BALANCE_OPERATION_*, enabled with RABBITMQ_TRANSACTION_ASYNC=true) plus legacy outbound overdraft and audit exchanges. Use a managed RabbitMQ service such as AWS MQ or CloudAMQP in production.
• RedPanda is the event backbone. Set STREAMING_ENABLED=true, STREAMING_BROKERS, and STREAMING_CLOUDEVENTS_SOURCE set to exactly ledger for ledger, CRM, and Fees, or tracer for Tracer. Facts publish to lerian.streaming.ledger or lerian.streaming.tracer. Route by CloudEvents headers. Transaction-lifecycle facts publish through this surface.

Optional PostgreSQL read replicas (DB_ONBOARDING_REPLICA_* and DB_TRANSACTION_REPLICA_*) serve reads. Transactional reads go to the primary by default.

​

High availability strategies

---

​

Service redundancy

• Deploy multiple replicas for every service.
• Use anti-affinity rules to spread services across zones.
• Apply Pod Disruption Budgets to limit downtime during updates.

​

Load balancing

• Use ingress controllers with health checks.
• Avoid session affinity unless a service requires it.
• Enable connection draining for smooth rollouts.

​

Security considerations

---

​

Network security

• Apply Kubernetes network policies to control traffic.
• Give each service account minimal permissions.
• Secure external access with TLS.
• Restrict admin interfaces with IP allowlists.

​

Secret management

• Use Kubernetes Secrets for credentials and tokens.
• Rotate secrets on a regular schedule.
• Never hardcode secrets in containers or config files.
• Use an external secret manager for a stronger posture.

​

Monitoring and observability

---

​

Metrics

• Monitor the key application and infrastructure KPIs.
• Set alert thresholds that lead to action.
• Use dashboards for real-time visibility.

​

Logging

• Centralize logs across all services.
• Use a structured format for easier filtering.
• Apply log retention and rotation policies.
• Define log-based alerts for critical events.

​

Tracing

• Enable distributed tracing across services.
• Sample traces to balance performance and cost.
• Correlate traces with logs and metrics for full visibility.

​

Alerting

• Create clear, reliable alerts.
• Tune thresholds to reduce noise.
• Route each alert through the right channel.
• Keep runbooks for recurring issues.

​

Backup strategy

---

• Automate regular backups for critical systems.
• Store backups in more than one location or region.
• Test the restore procedure on a regular schedule.
• Keep the backup documentation current and accessible.

​

Idempotency

---

Protect critical operations against duplicate processing in production:

• Send an idempotency key on every transaction create request with the X-Idempotency header.
• Use explicit, deterministic keys tied to your business IDs (order IDs, payment references), not auto-generated keys.
• Read the X-Idempotency-Replayed response header to tell a new transaction from a cached replay.
• Set the X-TTL header in seconds to match your retry window. The default is 300. Use a shorter value for synchronous flows and a longer one for asynchronous flows.

​

Aligning infrastructure to the Midaz architecture

---

Align your infrastructure to the Midaz architecture and you gain:

• Clean read/write separation with CQRS.
• Compatibility with managed cloud services.
• A clear path to observability, failover, and secure operations.

Review your setup on a regular schedule.

​

What’s next?

---

• Read the Midaz deployment guide.
• Contact our team for tailored support.