# systemd Service Templates and Verification

## Gateway Systemd Service

Save to `/etc/systemd/system/hermes-gateway.service`:

```ini
[Unit]
Description=Hermes Agent Gateway - Messaging Platform Integration
After=network-online.target
Wants=network-online.target
StartLimitIntervalSec=0

[Service]
Type=simple
User=root
Group=root
ExecStart=/usr/local/lib/hermes-agent/venv/bin/python -m hermes_cli.main gateway run --replace --accept-hooks
WorkingDirectory=/usr/local/lib/hermes-agent
Environment="HOME=/root"
Environment="USER=root"
Environment="LOGNAME=root"
Environment="PATH=/usr/local/lib/hermes-agent/venv/bin:/usr/local/lib/hermes-agent/node_modules/.bin:/root/.hermes/node/bin:/root/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
Environment="VIRTUAL_ENV=/usr/local/lib/hermes-agent/venv"
Environment="HERMES_HOME=/root/.hermes"
Environment="API_SERVER_ENABLED=true"
Environment="API_SERVER_KEY=<your-api-server-key>"
Environment="API_SERVER_PORT=8080"
Environment="API_SERVER_HOST=127.0.0.1"
Environment="API_SERVER_CORS_ORIGINS=*"
Restart=always
RestartSec=60
KillMode=mixed
KillSignal=SIGTERM
ExecReload=/bin/kill -USR1 $MAINPID
TimeoutStopSec=210
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target
```

### Installation commands

```bash
# Install as system service (root required)
sudo hermes gateway install --system --run-as-user root

# Manual alternative:
# 1. Write the unit file above
# 2. systemctl daemon-reload
# 3. systemctl enable hermes-gateway
# 4. systemctl start hermes-gateway
```

**Note:** `hermes gateway install --system` refuses root by default. Must pass `--run-as-user root` on VPS/container environments.

**Note:** Do NOT use `RestartMaxDelaySec` or `RestartSteps` — not supported on Ubuntu 22.04 systemd (v249).

## WebChat Systemd Service

Save to `/etc/systemd/system/hermes-webchat.service`:

```ini
[Unit]
Description=Hermes Web Chat Server
After=network-online.target hermes-gateway.service
Wants=network-online.target
StartLimitIntervalSec=0

[Service]
Type=simple
User=root
Group=root
WorkingDirectory=/root/.hermes/webchat
ExecStart=/usr/bin/python3 /root/.hermes/webchat/server.py
Restart=always
RestartSec=10
KillMode=mixed
KillSignal=SIGTERM
TimeoutStopSec=30
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target
```

### Installation commands

```bash
# Write the unit file, then:
systemctl daemon-reload
systemctl enable hermes-webchat
systemctl start hermes-webchat
```

**Note:** The `After=hermes-gateway.service` ensures WebChat starts after Gateway. If Gateway is down, WebChat will still start but chat requests will fail with connection refused.

## Verification Checklist

After server reboot or manual restart, verify:

```bash
# 1. All three services active
systemctl is-active hermes-gateway hermes-webchat nginx

# 2. All three services enabled (auto-start on boot)
systemctl is-enabled hermes-gateway hermes-webchat nginx

# 3. Ports listening
ss -tlnp | grep -E '80[^0-9]|8080|8888'

# 4. Full end-to-end test (login + chat)
curl -s -c /tmp/test.txt -X POST http://127.0.0.1:80/api/login \
  -H 'Content-Type: application/json' \
  -d '{"password":"YOUR_PASSWORD"}'
curl -s -b /tmp/test.txt -X POST http://127.0.0.1:80/api/chat \
  -H 'Content-Type: application/json' \
  -d '{"message":"hello"}'

# 5. Check logs
journalctl -u hermes-gateway --no-pager -n 20
journalctl -u hermes-webchat --no-pager -n 20

# 6. Check Gateway has API Server env vars
systemctl show hermes-gateway --property=Environment
```

## Common Issues

| Symptom | Cause | Fix |
|---|---|---|
| `hermes gateway install` fails | Root user, no `--run-as-user` | Add `--run-as-user root` |
| Chat returns "Connection refused" | Gateway not running or 8080 not bound | `systemctl restart hermes-gateway`, wait 5s, `ss -tlnp \| grep 8080` |
| systemd warnings about unknown keys | `RestartMaxDelaySec`/`RestartSteps` | Remove them, use only `Restart` + `RestartSec` |
| WebChat starts but can't reach Gateway | Race condition — WebChat up before Gateway | Normal; `Restart=always` with `RestartSec=10` handles it |
| API Server returns 401 | Wrong API_KEY in webchat/server.py | Must match `API_SERVER_KEY` in gateway systemd unit |
| Stale gateway processes after multiple restarts | Old PIDs not cleaned up | `pkill -f "hermes.*gateway"`, then `systemctl start hermes-gateway` |