> ## Documentation Index
> Fetch the complete documentation index at: https://docs.reachcrew.space/llms.txt
> Use this file to discover all available pages before exploring further.

# Logs and dashboards (Grafana)

> Grafana with Loki and Grafana Alloy: every container's log searchable, plus dashboards for health, LinkedIn sends, replies and scheduled jobs.

`docker-compose.observability.yml` adds three services beside ReachCrew:

| Service | What it does |
| - | - |
| **Loki** | Stores container logs for 30 days. |
| **Grafana Alloy** | Ships every container's log on the host to Loki, labelled with container, service and stream (stdout or stderr). |
| **Grafana** | Three dashboards, provisioned from `docker/observability/grafana/dashboards/`. |

| Dashboard | Shows |
| - | - |
| **ReachCrew overview** (home) | The `/api/health` counters (engine tick age, unconfirmed and stuck sends, webhook errors), errors per container, failing scheduled jobs, recent errors. |
| **ReachCrew logs** | Every log line, filtered by container, component (`scheduler`, `crm/worker`, `webhook/…`), stream and text. |
| **ReachCrew outreach** | Invitations and messages sent, failures and why, replies by channel, a per-organization table and the scheduled job runs. Filter by organization. |

Errors are the lines on stderr: ReachCrew writes `console.error` and `console.warn` there.

> [!NOTE]
>
> **What you need**
>
> * A Docker Compose install ([Docker Compose](https://docs.reachcrew.space/self-hosting/docker-compose.md)) with the repository's `docker/observability/` folder beside `docker-compose.yml`, or a clone of the repository.
> * About 1 GB of RAM to spare.

## 1. The read-only database role

The outreach and health panels read the database through `grafana_reader`, which can see states, types, timestamps and organization names only: no lead details, no message text, no rendered invitations, no webhook bodies. Each query is cut off after 10 seconds.

Replace `CHANGEME-grafana-reader-password` in `docker/observability/grafana-reader.sql` with a password (`openssl rand -hex 16`), then run it once:

```sh theme={null}
docker compose exec -T db psql -U reachcrew reachcrew < docker/observability/grafana-reader.sql
```

With your own PostgreSQL, run it with `psql "$DATABASE_URL" -f docker/observability/grafana-reader.sql` as the database owner. Running it again is safe.

## 2. Turn it on

Add to `.env`:

```sh theme={null}
COMPOSE_FILE=docker-compose.yml:docker-compose.observability.yml
GRAFANA_ADMIN_PASSWORD=...     # openssl rand -hex 16
GRAFANA_DB_PASSWORD=...        # the grafana_reader password from step 1
```

Combine it with the other files as needed, for example `COMPOSE_FILE=docker-compose.yml:docker-compose.https.yml:docker-compose.observability.yml`. Then `docker compose up -d`.

## 3. Open Grafana

Grafana listens on `127.0.0.1:3001` only, because it shows delivery data for every organization. From your computer:

```sh theme={null}
ssh -L 3001:localhost:3001 you@your-server
```

then open [http://localhost:3001](http://localhost:3001) and sign in as `admin` with `GRAFANA_ADMIN_PASSWORD`. Anonymous access and sign-up are off.

To give it its own domain instead (for example `grafana.example.com`), route that name to `grafana:3000` in your proxy, set `GRAFANA_ROOT_URL=https://grafana.example.com`, and keep a strong admin password.

## Settings

| Variable | Default | What it does |
| - | - | - |
| `GRAFANA_ADMIN_PASSWORD` | required | The `admin` user's password. |
| `GRAFANA_DB_PASSWORD` | required | The `grafana_reader` password. |
| `GRAFANA_DB_HOST` | `db:5432` | The database as `host:port`, when it is not the bundled one. |
| `GRAFANA_DB_NAME` | `reachcrew` | Database name. |
| `GRAFANA_DB_USER` | `grafana_reader` | Database role. |
| `GRAFANA_DB_SSLMODE` | `disable` | `require` for a hosted PostgreSQL. |
| `GRAFANA_PORT` | `3001` | The local port Grafana listens on. |
| `GRAFANA_ROOT_URL` | `http://localhost:3001` | Grafana's public address, if you give it one. |
| `LOG_CONTAINERS` | every container | A regular expression on container names, to collect only some (for example `reachcrew-.*`). |

## On Dokploy

Create a separate **Compose** service in Dokploy from this repository with `docker-compose.observability.yml` as its file, and set the variables above under **Environment**. `GRAFANA_DB_HOST` must name the database (the Dokploy PostgreSQL service's internal host and port). Alloy reads the Docker socket, so it collects the ReachCrew app's log however Dokploy runs it. Set `LOG_CONTAINERS` to your app's container name pattern if the server runs other things. Add a domain for the `grafana` service, port 3000, under **Domains**.

## Alerts

Grafana can alert on the same queries. A good first set, each sent to your on-call channel (Grafana → Alerting → Contact points):

* **Minutes since LinkedIn engine tick** above 10.
* **Stuck LinkedIn actions** above 0 for 10 minutes.
* **Webhooks given up (dead)** above 0.
* **Oldest unconfirmed** above 24 hours.
* **Errors** above your normal rate.

Pair them with an uptime check on `/api/health` from outside the server: if the server itself goes down, Grafana goes down with it.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.