Hermes — Self-Hosted AI Personal Assistant

This is an operator’s guide to standing up Hermes — a self-hosted, always-on AI personal assistant (as opposed to a coding agent like Claude Code or Mistral Vibe). Hermes lives in a chat group and/or an email inbox, holds long-term memory, manages a calendar, searches the web, and pushes urgent alerts to your phone. You talk to it like a chief-of-staff; it takes actions with real tools instead of just answering.

Everything here is provider-agnostic where possible and uses placeholder values (assistant@example.com, notify.example.com, 198.51.100.10). Substitute your own.

What is Hermes?

Hermes (the Nous “Hermes” agent lineage) is an agent runtime that connects a language model to gateway platforms (chat, email, etc.) and a set of tools (terminal, web search, calendar, memory, custom skills). The agent:

receives a message on a gateway (a chat group, a DM, an email),
reasons with its configured model,
calls tools to do things (run a command, send an email, add a calendar event, push an alert), and
replies with what it did.

It runs headless in a container, so it’s available 24/7 without your laptop.

Architecture

                       ┌─────────────────────────────────────────────┐
   Your phone / chat   │  Host (VM or LXC, e.g. on Proxmox)           │
   ───────────────►    │                                             │
   chat group, email   │   ┌───────────┐   model API (Mistral/OpenAI)│
                       │   │  Hermes   │──────────────────────────────────►  cloud LLM
   push alerts ◄────── │   │  agent    │   ┌──────────┐                │
   (ntfy app)          │   │ container │──►│ memory   │ (vector + LLM) │
                       │   └─────┬─────┘   └──────────┘                │
                       │         │  tools:                             │
                       │         ├─► email CLI  ──► your mailbox(es)    │
                       │         ├─► calendar (CalDAV sidecar)         │
                       │         ├─► web search (SearXNG)             │
                       │         └─► push (ntfy) ──┐                   │
                       │                            │                  │
                       │   Cloudflare Tunnel ◄──────┴── localhost only │
                       └───────────────│─────────────────────────────┘
                                       ▼
                              public HTTPS (no open ports)

Design principles

Bind services to localhost/private IPs; expose via Cloudflare Tunnel, not open ports. See Cloudflare Tunnels.
One container per concern, on a shared Docker network. The agent talks to sidecars (memory, calendar, search, push) by service name.
Secrets live server-side only — a gitignored .env (chmod 600), never committed, referenced from config as ${VAR}.

Prerequisites

A Linux host: a VM or Proxmox LXC container with Docker + Compose. 2 GB RAM is enough for the agent alone; 4 GB+ if you also self-host memory/search.
A baseline-hardened server — see Linux Server Initial Setup.
A model API key (Mistral and/or OpenAI), or a local model endpoint.
A Cloudflare Tunnel for any service the phone needs to reach (push, calendar).
A chat platform for the assistant to live in (a self-hostable messenger group works well) and/or a dedicated mailbox.

Install the core agent

Pull the agent image, give it a config directory, and a gitignored .env for secrets.

# compose.yml (excerpt)
services:
  hermes:
    image: ghcr.io/nousresearch/hermes:latest   # or your chosen agent image
    container_name: hermes
    restart: unless-stopped
    env_file: ./data/.env                        # secrets, chmod 600, gitignored
    volumes:
      - ./config:/config:ro                      # config.yaml, SOUL.md, skills
      - ./data:/home/hermes/.hermes              # runtime state, notes, memory
    networks: [assistant-net]
    user: "10001:10001"                          # never run the agent as root

networks:
  assistant-net:
    external: true

mkdir -p data config && chmod 700 data
cp .env.example data/.env && chmod 600 data/.env   # then fill in your keys
docker compose up -d
docker logs -f hermes

The no-clobber seed trick above only helps on a truly empty mount. Once the volume already holds a runtime, re-running the same cp -an on a new image version does nothing — no-clobber means the old files win — or, if you instead force-overwrite, you risk grafting a newer seed onto an older checkout and getting a mixed, half-upgraded tree. Neither is a real upgrade path. Do it properly: stamp a version marker in the baked seed (.runtime-release), and on a version change, stage the runtime directories (the agent binary/venv, not your config/secrets/state) somewhere temporary, validate the staged executable actually runs, then atomically swap old→a runtime-rollbacks/ folder and new→live. This gets you a real upgrade and a same-container rollback path if the new version misbehaves — never mutate the live runtime directory in place.

On hosts that default-DROP in the DOCKER-USER iptables chain (common on multi-service boxes), a container that auto-lands on a 192.168.x bridge gets no egress — DNS and API calls silently fail. Pin your assistant network into a range your firewall already allows (e.g. 172.16.0.0/12) so the agent can reach its model API:

networks:
  assistant-net:
    ipam:
      config:
        - subnet: 172.20.0.0/16

Model tiers (cheap/default/deep)

Don’t hardwire one model. Configure a default, automatic fallbacks, and a deep-reasoning tier you can switch to on demand. This controls both cost and quality.

# config.yaml (excerpt)
model:
  provider: mistral
  name: devstral-medium-latest        # default day-to-day brain
  fallback: [mistral-medium-latest, mistral-small-latest]
auxiliary:
  name: mistral-small-latest          # cheap model for vision/summarise/compression

For hard problems, wire a high-end model (e.g. a top OpenAI or Anthropic model) as a deep tier and toggle it with a small script that swaps the model block and restarts — keeping cheap models for routine traffic. Tip: give the deep tier its own rate-capped API key so a runaway loop can’t drain your main key.

Some providers gate function/tool calling to their Chat-Completions-style endpoint, where tool calls only work reliably with an effective reasoning_effort: none — reasoning plus tools on that same model instead requires their separate Responses-style API, which is a different integration, not a config flag. If you swap in a new model and the agent stops calling tools (silently falls back to describing what it would do), check whether your new model’s reasoning default is incompatible with tool calls on the transport you’re using — don’t assume a same-vendor model swap is a drop-in replacement. Always run a real tool-call probe (not just a text completion) after any model migration, before considering the cutover done.

Memory that actually persists

LLM agents have two memory problems: they forget between sessions, and their auto-summarised “memory” can drift into confidently wrong facts. Solve it in layers:

A long-term memory service (e.g. a vector-store memory backend) for general recall. These typically need a one-time DB migration before first start — if it crash-loops on boot, run the migration (alembic upgrade head or equivalent) first. They also often need a real embeddings key (don’t point an OpenAI-embeddings module at a Mistral key — it 401-loops).

Durable Markdown note pages under the runtime dir — the reliable second brain:

~/.hermes/notes/trips/<trip>.md      # itineraries, confirmation codes
~/.hermes/notes/projects/<x>.md
~/.hermes/notes/people.md
~/.hermes/expenses/<period>.md       # logged receipts

Email — give the agent its own inbox

Let the assistant send and receive as itself (e.g. assistant@example.com) and optionally manage your personal mailboxes. Two ways:

A gateway “email” platform: the agent polls an inbox and auto-replies to allowlisted senders — you email the assistant, it answers like an assistant.
A CLI email tool the agent drives from its terminal for read/reply/forward/move/ archive/delete (a TUI/CLI mail client is far more reliable here than most email MCPs, which frequently lack a move/archive verb or break under headless npx).

If your mail server’s ports are bound to a public IP (common for SMTP/IMAP) but the hostname is proxied, a container trying to reach it by hostname may fail to connect. Pin the hostname to the direct IP (compose extra_hosts) and/or join the mail server’s Docker network so the agent reaches it internally. Symptom: IMAP/SMTP timeouts from inside the container while the host itself connects fine.

These two fixes are often not alternatives — you may need both together. If the mail stack runs separate backend containers behind one TLS-covered hostname (e.g. IMAP and SMTP served by different containers, each with its own internal IP), simply joining the network gets you connectivity but not a hostname the TLS handshake will accept, since you’re connecting by internal IP with no matching certificate name. Join the network and add extra_hosts entries that map each certificate-covered hostname to that specific backend’s internal IP — connectivity from the network join, certificate validity from the hostname pin.

By default the email gateway replies to anything an allowlisted sender sends — including mail where the assistant is only CC’d, which you usually don’t want a reply on. But the stock adapter often parses only From/Subject/body, so the agent can’t even tell. Two changes make it behave: (1) parse To/Cc and inject them into the message the agent sees, with a rule — in To → reply (to the sender only, never reply-all); only in Cc → do NOT reply, just extract tasks/dates/facts and act, then send nothing; (2) give it a way to send no reply. An empty response does not suppress the email — many runtimes normalise "" into a ”⚠ no response generated” auto-reply — so honor an explicit [SILENT] sentinel in the adapter’s send() and have the agent emit it to stay quiet.

Calendar & contacts

Give the agent a calendar it fully controls via a CalDAV/CardDAV sidecar (e.g. Radicale) and a DAV tool/MCP. This sidesteps the common situation where your main mail/groupware stack has DAV locked behind an SSO/Bearer wall that rejects plain passwords. The agent gets working calendar + contacts immediately; you can subscribe to the same calendar from your phone by exposing the DAV port through the tunnel.

Put the exact calendar URL in SOUL.md — or the agent invents one

A DAV “create event” tool usually takes a calendar URL argument. If you don’t tell the agent the real collection URL, it will hallucinate a plausible one (e.g. https://cal.example.com/dav/calendars/<you>/calendar/), every call fails “calendar not found,” the MCP trips its circuit-breaker after a few misses, and the agent reports the calendar is “unreachable” — when the server was healthy the whole time. Fix: put the exact collection (e.g. http://radicale:5232/<user>/calendar/) in SOUL.md with a rule “never invent a calendar URL; if a call says unreachable, wait and retry, don’t switch to a guessed URL.” Verify the path independently with curl -u … -X PROPFIND (a 207 means it’s fine — the problem is the URL the agent chose, not the service).

Proactive monitoring & scheduled briefs

A chief-of-staff doesn’t only answer when asked — it surfaces things on a schedule. Use the agent runtime’s cron to push recurring briefs to your chat: a morning brief (today’s calendar + tasks due + urgent unread mail + today’s travel), an evening prep (tomorrow), a midday inbox triage (read-only — alert only if urgent), and a weekly review (open tasks + week ahead + un-filed expenses). Keep each brief short and tell it (in SOUL.md) to skip empty sections and never dump its memory profile.

Tasks & follow-ups — one Markdown backlog is the source of truth

Have the agent capture every actionable item (from chat, an email, a receipt) into a single ~/.hermes/notes/tasks.md table (| Task | Due | Source | Status |) and surface open/overdue items in the briefs. This file — not the DAV todo store (often broken, above) and not derived memory (drifts) — is authoritative. For time-bound tasks, also schedule a one-off reminder and, if it belongs on the calendar, add a calendar event.

Web search

Point the agent’s web tool at a self-hosted SearXNG instance for private, keyless search. Note: SearXNG returns results, not full page text — if you need article extraction, give the agent a dedicated fetch/extract tool and tell it (in SOUL.md) to use that, not a search tool, for “read this URL” requests.

Push alerts (“like a phone call”)

For time-critical pings (a delayed flight, a deadline, a security event) put a self-hosted ntfy server in the stack. The agent publishes internally (http://ntfy/<topic>); your phone subscribes over a public HTTPS URL via the tunnel. Give the agent a dead-simple wrapper so it reliably uses this channel:

# ~/.hermes/bin/alert  [--urgent] "message"
prio=high; [ "$1" = "--urgent" ] && { prio=urgent; shift; }
curl -fsS -H "Authorization: Bearer $NTFY_TOKEN" -H "Priority: $prio" \
     -d "$1" "$NTFY_URL/$NTFY_TOPIC"

Self-hosted ntfy publishes but WON’T notify the phone until you set upstream-base-url

This is the #1 “it’s not working” gotcha. A self-hosted server can’t wake a phone on its own — Android/iOS only deliver background push via Google FCM / Apple APNs. Without an upstream relay, messages sit in the topic and only appear when you open the app or pull-to-refresh (which looks like an auth problem but isn’t). Fix in server.yml:

upstream-base-url: "https://ntfy.sh"

Then restart. Your server now sends ntfy.sh a hashed wake-up trigger only — the message body never leaves your server — and ntfy.sh relays the push. The phone must remove and re-add the subscription so it registers for the upstream (hashed) topic. After that, urgent alerts ring through silent/DND (enable Critical alerts on the subscription). The F-Droid build can instead use the app’s Instant delivery toggle (a persistent connection, no upstream needed).

Exposing services via Cloudflare Tunnel

Bind everything to 127.0.0.1/private IPs and front it with a tunnel.

Adding a tunnel hostname is two operations with different permission requirements:

Ingress rule (which origin a hostname maps to) — needs Tunnel:Edit. For a token-mode tunnel run with --token and no local config file, the daemon picks up ingress changes from the remote/dashboard config automatically.
DNS record (so the hostname resolves to the tunnel) — needs DNS:Edit on that zone. A token scoped only to Tunnel:Edit + Zone:Read cannot create the DNS record. The simplest path is to run, on the host that holds the tunnel’s origin cert:
Terminal window
```
cloudflared tunnel route dns <tunnel-id> notify.example.com
```

Verify end-to-end: dig @1.1.1.1 notify.example.com resolves, and curl https://notify.example.com/v1/health returns healthy.

The single most important lesson: `SOUL.md`

Most agent runtimes load a persona/behavior file (often SOUL.md) fresh on every message. This — not the model, not the skills — is your reliable behavior channel.

Things worth encoding in SOUL.md:

“Use your tools — don’t describe.” Without this, agents reply with “here’s how you could do that” instead of doing it. Tell it: run the command first, answer from the result; never reply “I don’t have access” for something it has a tool for.
Take initiative (chief-of-staff). Map intent → action: a date → add a calendar event
- reminder; a receipt → log it to the expense file; a durable fact → write it to the right note page; then reply with what you did.
Notes/files override memory for must-be-correct facts (see Memory).
The exact API calls for any live data you want it to fetch (your own endpoints, with the auth header referenced as an env var, never the secret inline).
One alert channel. Name it explicitly (run the alert command), or the agent will improvise and try to alert you over chat/SMS/email instead.

Multi-message replies & chat formatting

When the agent drafts a message for you to forward (a note to a colleague, a reply to paste elsewhere), you want the draft in its own clean bubble — separate from the agent’s “want me to send this?” chatter — so you can copy it in one tap. Two problems bite:

Chat apps don’t render GitHub-flavored Markdown. Many messengers (SimpleX, for one) use single-character delimiters (*bold*, _italic_, `code`) and have no **bold** or triple-backtick fences — so an agent that wraps a draft in ``` fences just shows literal backticks, which also ruins copy-paste.
The adapter sends the whole reply as one message (it only length-chunks), so the draft, the preamble, and the follow-up question all land in one bubble.

The instinctive fix — a SOUL.md rule (“put the draft in its own message, plain text, separate parts with a [[SPLIT]] marker”) — is unreliable: a smaller model ignores the convention and keeps fencing the draft. The robust fix is deterministic, in the adapter’s send(), keyed on what the model reliably does (it always fences a draft): split the reply into separate messages on fenced blocks (delivering each with the fences stripped), also honor an explicit [[SPLIT]], and flatten **bold**/--- the app can’t render. Now a fenced draft automatically arrives as its own clean, copy-pasteable bubble — regardless of whether the model followed any instruction. General principle: for output hygiene, don’t depend on a small model to follow a formatting rule; transform deterministically at the I/O boundary. (Keep the SOUL.md rule too, as belt-and-suspenders.)

Security & hardening

Run the container as a non-root user; mount config read-only.
Secrets server-side only: gitignored .env (chmod 600), referenced as ${VAR} in config. Never commit keys, password hashes, or account tokens. Rotate any shared password the assistant uses.
Allowlist who can drive the agent. If it lives in a chat group, restrict commands to known senders rather than “anyone in the group.” For the email gateway, set an explicit allowed-senders list — otherwise anyone who emails the address can task your assistant.
no-new-privileges + AppArmor caveats: if you harden the container with no-new-privileges: true, some runtimes that spawn helper processes (Node, envoy) need apparmor: unconfined or they fail silently.
Minimise blast radius of the model key — cap/limit the deep-tier key separately.

Before a risky change: back up first

A model migration, a mail-routing change, or a runtime upgrade on a live assistant is a one-way door if it goes wrong mid-flight — the agent is holding real conversations, real memory, real credentials. Before any of these, in order:

Snapshot the runtime data volume (config, notes, memory, sessions, SimpleX/chat databases) to a timestamped, checksummed archive.
Tag a rollback image of the currently-running container (docker tag hermes hermes:rollback-<timestamp>) before rebuilding.
Copy the pre-change .env alongside the data snapshot — a secrets-rotation step inside the same change is common and easy to lose track of otherwise.
Verify the backup, not just that it ran — checksum it, or restore it to a scratch directory and diff.

This turns “the migration broke something and I have to rebuild from memory” into “revert the tag and restore the snapshot,” which takes minutes instead of hours.

Troubleshooting

Symptom	Likely cause	Fix
`HTTP 400 invalid model`	Alias instead of canonical id	Use the provider’s exact id (e.g. `…-latest`)
Agent has no internet / DNS fails	Container on a `DOCKER-USER`-dropped subnet	Pin network to an allowed range (e.g. `172.16/12`)
Memory service crash-loops on boot	Migrations not run	Run `alembic upgrade head` (or equivalent) first
Embeddings 401-loop	Wrong key for the embeddings module	Give the embeddings module a real OpenAI key
Agent ignores an installed skill	Skills aren’t auto-consulted	Put the must-know bits in `SOUL.md`
Agent recalls wrong trip/flight facts	Trusting stale injected memory	Note files override memory; have it `cat` the file
ntfy publishes but phone is silent	No `upstream-base-url`	Set it, restart, re-add the phone subscription
ntfy “not authorized”	Anonymous / wrong-server subscription	Log in per-server, re-add subscription
Tunnel DNS create fails (auth error)	Token lacks DNS:Edit on the zone	Use `cloudflared tunnel route dns …` on the host
Mail timeouts from container only	Public-IP hairpin	`extra_hosts` pin or join the mail Docker network
Email MCP can’t move/archive	MCP lacks the verb	Use a CLI mail client from the agent’s terminal
Calendar reported “unreachable” but server is healthy	Agent invented a calendar URL	Put the exact collection URL in `SOUL.md`; verify with `PROPFIND` (`207`)
`create_todo`/VTODO errors at runtime	DAV client doesn’t implement it	Use `create_event` + a Markdown task backlog
Key “works” but real requests fail	Key authenticates but account has no funded quota	Test with one real completion call, not just an auth probe
Model swap stops calling tools (agent describes instead of doing)	New model’s reasoning default incompatible with tool calls on this transport	Set effective `reasoning_effort: none`, or migrate to the Responses-style API
Post-upgrade: runtime crashes, weird half-old/half-new behavior	No-clobber seed copy can’t upgrade an existing runtime	Versioned atomic promotion (stage → validate → swap, keep old under `runtime-rollbacks/`)
Same rebuild produces a different runtime each time	Installer tracks `main` instead of a pinned release	Pin an explicit release tag; fetch the installer from that same tag
Scheduled brief never fires	`cron:` config block isn’t the live store	Create jobs via the runtime CLI (`cron/jobs.json`); confirm with the job-list cmd
Agent replies to mail it was only CC’d on	Adapter doesn’t parse To/Cc	Parse To/Cc + inject role; reply only when in To, `[SILENT]` otherwise
Draft arrives as one bubble with literal `**`/```	Whole reply sent as one msg; app ≠ GitHub markdown	Split deterministically on fences in `send()`, strip fences/`**`

Mistral Vibe — self-hosted agentic CLI; same model ecosystem
Claude Code — agentic coding assistant (different use case)
Agent Comparison — feature/benchmark matrix of agents
Docker — container runtime fundamentals
Proxmox LXC Setup — host platform
Linux Server Initial Setup — server baseline
Cloudflare Tunnels — expose services with no open ports
Authentik Installation — SSO if you add a dashboard
Self-Hosted Mail Server (2026) — give the agent a real mailbox
Hermes adapter patches & cron-seed (snippet) — runnable companion code for the patches in this guide

Hermes — Self-Hosted AI Personal Assistant

Hermes — Self-Hosted AI Personal Assistant

What is Hermes?

Architecture

Prerequisites

Install the core agent

Model tiers (cheap/default/deep)

Memory that actually persists

Email — give the agent its own inbox

Calendar & contacts

Proactive monitoring & scheduled briefs

Tasks & follow-ups — one Markdown backlog is the source of truth

Web search

Push alerts (“like a phone call”)

Exposing services via Cloudflare Tunnel

The single most important lesson: SOUL.md

Multi-message replies & chat formatting

Security & hardening

Before a risky change: back up first

Troubleshooting

Related Resources

The single most important lesson: `SOUL.md`