Skip to content

OpenSearch Centralized Logging

Overview

All application and database logs ship to a single managed OpenSearch cluster (opensearch-logs) and are viewed in OpenSearch Dashboards. There are two streams:

  • App logs — DigitalOcean App Platform forwards each app's stdout/stderr to a per-app index (log_destination in the app spec).
  • DB logs — DigitalOcean managed PostgreSQL clusters forward their server logs (connections, slow queries, errors) via a logsink to a per-cluster index.

Infra source of truth (sinks, ACLs, index templates, retention): the infra-do repo — docs/db-unification-runbook.md and docs/observability/RUNBOOK.md. This page is the developer-facing view: how to name a log stream so it lands in the right dashboard.

The one rule that matters

Every log index name must contain logs, and the name's tokens decide everything downstream — write permission, which dashboard shows it, and how long it's kept. Get the name right and the dashboards "just work" with zero extra setup.

Stream Index name convention Example Time field
App logs <app>-<env>-logs anatomi-backend-prod-logs @timestamp
DB logs postgresql-<cluster>-logs postgresql-anatomi-prod-logs timestamp
  • DB log indices must start with postgresql- — that prefix is what matches the postgres_template mappings (so timestamp is a date and PRIORITY is numeric) and the logs-db dashboard. pg-… will silently break both.
  • The <env> token (-dev- / -staging- / -prod-) drives retention (below).
  • DB logs use the journald schema: MESSAGE, HOSTNAME, SYSTEMD_UNIT, PRIORITY (syslog severity — PRIORITY <= 3 = error/critical), time field timestamp (not @timestamp).

Index patterns the dashboards expect

These index patterns and saved objects already exist in Dashboards — match their naming and your data appears automatically.

Dashboard / pattern Matches index pattern Time field Purpose
logs-dev *-dev-logs* @timestamp dev app logs
logs-staging *-staging-logs* @timestamp staging app logs
logs-prod *-prod-logs* @timestamp prod app logs
logs-all *-logs* @timestamp everything
logs-db postgresql-*-logs-* timestamp all DB logs

Saved searches / dashboards:

  • "App logs overview" — log volume by do_app_name (chatty-app / disk early warning).
  • errors-dev / errors-staging / errors-prod — query log: *ERROR* or log: *FATAL* or log: *Exception*.
  • "DB logs overview" + errors-db — DB severity via PRIORITY <= 3.

DigitalOcean appends a date suffix, so an index is e.g. postgresql-anatomi-prod-logs-2026-06-15. When creating a Dashboards index pattern, always end it with * (e.g. postgresql-*-logs-*) — a pattern ending in -logs won't match the dated index.

Retention (ISM, automatic)

Retention is decided by the <env> token in the index name and auto-applies to new daily indices — no per-index action needed.

Index name contains Retention
-dev-logs 7 days
-staging-logs 14 days
-prod-logs 30 days
(any other *-logs*) 14 days (catch-all)

So postgresql-anatomi-prod-logs-* is kept 30 days; postgresql-shared-dev-logs-* 7 days — purely because of the -prod- / -dev- token.

Viewing logs

Log into OpenSearch Dashboards as dev-logs (read-only; password in the password manager — never use doadmin). Start from "App logs overview" or "DB logs overview", or open Discover on the relevant pattern above.

Adding a new stream

New app — add a log_destination to the app spec (infra-do envs/<env>/ app_platform.tf) with index_name = "<app>-<env>-logs". The existing logs-<env> pattern picks it up; nothing to create in Dashboards.

New DB cluster — add a digitalocean_database_logsink (infra-do) with index prefix postgresql-<cluster>-logs, authenticating as the applogs user. It lands in the existing logs-db dashboard automatically.

Both belong in Terraform (infra-do). Configuring a logsink only in the DO console works but is not reproducible — a cluster rebuild drops it silently.

Naming checklist (before you ship a log stream)

  • [ ] Name contains logs (else writes are denied by the applogs ACL — silent).
  • [ ] DB logs start with postgresql- (else no mappings, empty logs-db dashboard).
  • [ ] Name carries the right <env> token for the retention you want.
  • [ ] Dashboards index pattern (if you must create one) ends in *.