Hermes — Self-Hosted AI Personal Assistant

Hermes — Self-Hosted AI Personal Assistant

Quick Navigation

Jump to: What is Hermes? | Architecture | Prerequisites | Install | Model tiers | Memory | Email | Calendar | Proactive briefs & tasks | Search | Push alerts | Behavior tuning | Multi-message & formatting | Hardening | Troubleshooting

This is an operator’s guide to standing up Hermes — a self-hosted, always-on AI personal assistant (as opposed to a coding agent like Claude Code or Mistral Vibe). Hermes lives in a chat group and/or an email inbox, holds long-term memory, manages a calendar, searches the web, and pushes urgent alerts to your phone. You talk to it like a chief-of-staff; it takes actions with real tools instead of just answering.

Everything here is provider-agnostic where possible and uses placeholder values (assistant@example.com, notify.example.com, 198.51.100.10). Substitute your own.

Reference implementation

The non-obvious patches below (CC-aware email, deterministic multi-message replies, proactive cron-seed) have real, copy-pasteable code in a companion snippet: Hermes adapter patches & cron-seed.

Treat the whole thing as a credential

A personal assistant with your email, calendar, and memory is a high-value target. The chat group link, the email login, and the agent’s API keys are all credentials. If you let anyone in a group drive the agent, the group invite is effectively a password to your inbox. Plan an allowlist from day one (see Hardening).

What is Hermes?

Hermes (the Nous “Hermes” agent lineage) is an agent runtime that connects a language model to gateway platforms (chat, email, etc.) and a set of tools (terminal, web search, calendar, memory, custom skills). The agent:

• receives a message on a gateway (a chat group, a DM, an email),
• reasons with its configured model,
• calls tools to do things (run a command, send an email, add a calendar event, push an alert), and
• replies with what it did.

It runs headless in a container, so it’s available 24/7 without your laptop.

Architecture

                       ┌─────────────────────────────────────────────┐
   Your phone / chat   │  Host (VM or LXC, e.g. on Proxmox)           │
   ───────────────►    │                                             │
   chat group, email   │   ┌───────────┐   model API (Mistral/OpenAI)│
                       │   │  Hermes   │──────────────────────────────────►  cloud LLM
   push alerts ◄────── │   │  agent    │   ┌──────────┐                │
   (ntfy app)          │   │ container │──►│ memory   │ (vector + LLM) │
                       │   └─────┬─────┘   └──────────┘                │
                       │         │  tools:                             │
                       │         ├─► email CLI  ──► your mailbox(es)    │
                       │         ├─► calendar (CalDAV sidecar)         │
                       │         ├─► web search (SearXNG)             │
                       │         └─► push (ntfy) ──┐                   │
                       │                            │                  │
                       │   Cloudflare Tunnel ◄──────┴── localhost only │
                       └───────────────│─────────────────────────────┘
                                       ▼
                              public HTTPS (no open ports)

Design principles

• Bind services to localhost/private IPs; expose via Cloudflare Tunnel, not open ports. See Cloudflare Tunnels.
• One container per concern, on a shared Docker network. The agent talks to sidecars (memory, calendar, search, push) by service name.
• Secrets live server-side only — a gitignored .env (chmod 600), never committed, referenced from config as ${VAR}.

Prerequisites

• A Linux host: a VM or Proxmox LXC container with Docker + Compose. 2 GB RAM is enough for the agent alone; 4 GB+ if you also self-host memory/search.
• A baseline-hardened server — see Linux Server Initial Setup.
• A model API key (Mistral and/or OpenAI), or a local model endpoint.
• A Cloudflare Tunnel for any service the phone needs to reach (push, calendar).
• A chat platform for the assistant to live in (a self-hostable messenger group works well) and/or a dedicated mailbox.

Install the core agent

Pull the agent image, give it a config directory, and a gitignored .env for secrets.

# compose.yml (excerpt)
services:
  hermes:
    image: ghcr.io/nousresearch/hermes:latest   # or your chosen agent image
    container_name: hermes
    restart: unless-stopped
    env_file: ./data/.env                        # secrets, chmod 600, gitignored
    volumes:
      - ./config:/config:ro                      # config.yaml, SOUL.md, skills
      - ./data:/home/hermes/.hermes              # runtime state, notes, memory
    networks: [assistant-net]
    user: "10001:10001"                          # never run the agent as root

networks:
  assistant-net:
    external: true

mkdir -p data config && chmod 700 data
cp .env.example data/.env && chmod 600 data/.env   # then fill in your keys
docker compose up -d
docker logs -f hermes

Bind-mounting the runtime dir can hide the image’s baked defaults

If the image bakes starter files into ~/.hermes and you mount an empty data/ over it, those files disappear. Seed them: have the entrypoint copy baked defaults into the mount on first run (cp -an /seed/. ~/.hermes/), or initialise data/ from the image once.

Docker network egress trap

On hosts that default-DROP in the DOCKER-USER iptables chain (common on multi-service boxes), a container that auto-lands on a 192.168.x bridge gets no egress — DNS and API calls silently fail. Pin your assistant network into a range your firewall already allows (e.g. 172.16.0.0/12) so the agent can reach its model API:

networks:
  assistant-net:
    ipam:
      config:
        - subnet: 172.20.0.0/16

Model tiers (cheap/default/deep)

Don’t hardwire one model. Configure a default, automatic fallbacks, and a deep-reasoning tier you can switch to on demand. This controls both cost and quality.

# config.yaml (excerpt)
model:
  provider: mistral
  name: devstral-medium-latest        # default day-to-day brain
  fallback: [mistral-medium-latest, mistral-small-latest]
auxiliary:
  name: mistral-small-latest          # cheap model for vision/summarise/compression

Use the model provider’s EXACT model id

A gateway/proxy may accept a friendly alias (devstral-medium) that the raw provider API rejects with HTTP 400 invalid model. Always use the exact published id (devstral-medium-latest). When a model “works in one place and 400s in another,” an alias-vs-canonical mismatch is the usual cause.

For hard problems, wire a high-end model (e.g. a top OpenAI or Anthropic model) as a deep tier and toggle it with a small script that swaps the model block and restarts — keeping cheap models for routine traffic. Tip: give the deep tier its own rate-capped API key so a runaway loop can’t drain your main key.

Memory that actually persists

LLM agents have two memory problems: they forget between sessions, and their auto-summarised “memory” can drift into confidently wrong facts. Solve it in layers:

1. A long-term memory service (e.g. a vector-store memory backend) for general recall. These typically need a one-time DB migration before first start — if it crash-loops on boot, run the migration (alembic upgrade head or equivalent) first. They also often need a real embeddings key (don’t point an OpenAI-embeddings module at a Mistral key — it 401-loops).

2. Durable Markdown note pages under the runtime dir — the reliable second brain:

   ~/.hermes/notes/trips/<trip>.md      # itineraries, confirmation codes
   ~/.hermes/notes/projects/<x>.md
   ~/.hermes/notes/people.md
   ~/.hermes/expenses/<period>.md       # logged receipts

Files must OVERRIDE auto-injected memory for must-be-correct facts

A real failure mode: an assistant “lost” a flight itinerary (its built-in memory was wiped) and then recalled the wrong one (its vector memory held a stale version). The fix is a written rule (in SOUL.md, below): for trip/flight/expense/confirmation facts, the agent must read the note file and answer only from it — the file wins over injected memory. Persist anything that must be right; don’t trust derived memory alone.

Email — give the agent its own inbox

Let the assistant send and receive as itself (e.g. assistant@example.com) and optionally manage your personal mailboxes. Two ways:

• A gateway “email” platform: the agent polls an inbox and auto-replies to allowlisted senders — you email the assistant, it answers like an assistant.
• A CLI email tool the agent drives from its terminal for read/reply/forward/move/ archive/delete (a TUI/CLI mail client is far more reliable here than most email MCPs, which frequently lack a move/archive verb or break under headless npx).

Version-match your mail client config

CLI mail clients change their config schema between releases (e.g. backend.type = "imap" in one major version vs imap.server = … in another; subcommand flag ordering differs too). Match the docs to the installed version, not the latest on the web, or every command errors.

Container → public-IP “hairpin” for mail

If your mail server’s ports are bound to a public IP (common for SMTP/IMAP) but the hostname is proxied, a container trying to reach it by hostname may fail to connect. Pin the hostname to the direct IP (compose extra_hosts) and/or join the mail server’s Docker network so the agent reaches it internally. Symptom: IMAP/SMTP timeouts from inside the container while the host itself connects fine.

Make the auto-reply CC-aware: reply when in To, just ingest when CC’d

By default the email gateway replies to anything an allowlisted sender sends — including mail where the assistant is only CC’d, which you usually don’t want a reply on. But the stock adapter often parses only From/Subject/body, so the agent can’t even tell. Two changes make it behave: (1) parse To/Cc and inject them into the message the agent sees, with a rule — in To → reply (to the sender only, never reply-all); only in Cc → do NOT reply, just extract tasks/dates/facts and act, then send nothing; (2) give it a way to send no reply. An empty response does not suppress the email — many runtimes normalise "" into a ”⚠ no response generated” auto-reply — so honor an explicit [SILENT] sentinel in the adapter’s send() and have the agent emit it to stay quiet.

Calendar & contacts

Give the agent a calendar it fully controls via a CalDAV/CardDAV sidecar (e.g. Radicale) and a DAV tool/MCP. This sidesteps the common situation where your main mail/groupware stack has DAV locked behind an SSO/Bearer wall that rejects plain passwords. The agent gets working calendar + contacts immediately; you can subscribe to the same calendar from your phone by exposing the DAV port through the tunnel.

Put the exact calendar URL in SOUL.md — or the agent invents one

A DAV “create event” tool usually takes a calendar URL argument. If you don’t tell the agent the real collection URL, it will hallucinate a plausible one (e.g. https://cal.example.com/dav/calendars/<you>/calendar/), every call fails “calendar not found,” the MCP trips its circuit-breaker after a few misses, and the agent reports the calendar is “unreachable” — when the server was healthy the whole time. Fix: put the exact collection (e.g. http://radicale:5232/<user>/calendar/) in SOUL.md with a rule “never invent a calendar URL; if a call says unreachable, wait and retry, don’t switch to a guessed URL.” Verify the path independently with curl -u … -X PROPFIND (a 207 means it’s fine — the problem is the URL the agent chose, not the service).

create_todo is often broken — use calendar events, not VTODOs

Several DAV MCPs expose a create_todo/VTODO tool that fails at runtime (client.createTodo is not a function) because the underlying client/server doesn’t implement it. Don’t build task tracking on it. Keep tasks in a Markdown backlog (below) and, for anything that should appear on the calendar, create a normal event at the due time — create_event works where create_todo doesn’t.

Proactive monitoring & scheduled briefs

A chief-of-staff doesn’t only answer when asked — it surfaces things on a schedule. Use the agent runtime’s cron to push recurring briefs to your chat: a morning brief (today’s calendar + tasks due + urgent unread mail + today’s travel), an evening prep (tomorrow), a midday inbox triage (read-only — alert only if urgent), and a weekly review (open tasks + week ahead + un-filed expenses). Keep each brief short and tell it (in SOUL.md) to skip empty sections and never dump its memory profile.

Cron jobs in your config file may NOT be the ones that run

A subtle trap: a cron: block in config.yaml can be dead config — the scheduler may read only a separate runtime store (e.g. ~/.hermes/cron/jobs.json) populated by a cron create CLI / an in-chat /cron command. Symptom: you declared a daily brief in the config and it never fires, and the job list shows empty. Seed jobs with the CLI (keep a small idempotent seed script in version control) rather than editing the config block, and confirm with the runtime job-list command that they’re actually registered.

Delivering a scheduled job to the chat = the platform “home channel”

A CLI-created job has no “origin” chat, so it defaults to local (logged, not sent). To post a brief to your group, deliver to the platform’s home channel — set the home channel/chat id in .env and use the bare platform name as the delivery target (--deliver <platform>), not platform:chat_id (the target parser splits on : and will mis-read an id that itself contains a colon).

Tasks & follow-ups — one Markdown backlog is the source of truth

Have the agent capture every actionable item (from chat, an email, a receipt) into a single ~/.hermes/notes/tasks.md table (| Task | Due | Source | Status |) and surface open/overdue items in the briefs. This file — not the DAV todo store (often broken, above) and not derived memory (drifts) — is authoritative. For time-bound tasks, also schedule a one-off reminder and, if it belongs on the calendar, add a calendar event.

The agent often describes a task instead of recording it

Telling the agent “capture tasks” in SOUL.md isn’t always enough — a weaker model will list the task in its reply and never write the file. Verify the file actually grows; if it doesn’t, make the instruction concrete (“append a row to tasks.md now with the terminal, then confirm”) and spot-check it. The reliable record is the file, not the chat.

Web search

Point the agent’s web tool at a self-hosted SearXNG instance for private, keyless search. Note: SearXNG returns results, not full page text — if you need article extraction, give the agent a dedicated fetch/extract tool and tell it (in SOUL.md) to use that, not a search tool, for “read this URL” requests.

Push alerts (“like a phone call”)

For time-critical pings (a delayed flight, a deadline, a security event) put a self-hosted ntfy server in the stack. The agent publishes internally (http://ntfy/<topic>); your phone subscribes over a public HTTPS URL via the tunnel. Give the agent a dead-simple wrapper so it reliably uses this channel:

# ~/.hermes/bin/alert  [--urgent] "message"
prio=high; [ "$1" = "--urgent" ] && { prio=urgent; shift; }
curl -fsS -H "Authorization: Bearer $NTFY_TOKEN" -H "Priority: $prio" \
     -d "$1" "$NTFY_URL/$NTFY_TOPIC"

Self-hosted ntfy publishes but WON’T notify the phone until you set upstream-base-url

This is the #1 “it’s not working” gotcha. A self-hosted server can’t wake a phone on its own — Android/iOS only deliver background push via Google FCM / Apple APNs. Without an upstream relay, messages sit in the topic and only appear when you open the app or pull-to-refresh (which looks like an auth problem but isn’t). Fix in server.yml:

upstream-base-url: "https://ntfy.sh"

Then restart. Your server now sends ntfy.sh a hashed wake-up trigger only — the message body never leaves your server — and ntfy.sh relays the push. The phone must remove and re-add the subscription so it registers for the upstream (hashed) topic. After that, urgent alerts ring through silent/DND (enable Critical alerts on the subscription). The F-Droid build can instead use the app’s Instant delivery toggle (a persistent connection, no upstream needed).

deny-all + per-user auth, not anonymous topics

Run ntfy with auth-default-access: deny-all and a real user. A subscription added anonymously or pinned to the wrong server returns “not authorized” even when the user/topic ACLs are perfect — log in per-server in the app (Settings → Manage users), then re-add the subscription.

Exposing services via Cloudflare Tunnel

Bind everything to 127.0.0.1/private IPs and front it with a tunnel.

DNS for the hostname needs the right Cloudflare credential

Adding a tunnel hostname is two operations with different permission requirements:

1. Ingress rule (which origin a hostname maps to) — needs Tunnel:Edit. For a token-mode tunnel run with --token and no local config file, the daemon picks up ingress changes from the remote/dashboard config automatically.

2. DNS record (so the hostname resolves to the tunnel) — needs DNS:Edit on that zone. A token scoped only to Tunnel:Edit + Zone:Read cannot create the DNS record. The simplest path is to run, on the host that holds the tunnel’s origin cert:

   cloudflared tunnel route dns <tunnel-id> notify.example.com

Verify end-to-end: dig @1.1.1.1 notify.example.com resolves, and curl https://notify.example.com/v1/health returns healthy.

The single most important lesson: SOUL.md

Most agent runtimes load a persona/behavior file (often SOUL.md) fresh on every message. This — not the model, not the skills — is your reliable behavior channel.

Enabled skills are NOT automatically consulted

You can install and “enable” skills/tools and the agent will still ignore them, because it doesn’t auto-read them per message. Anything the agent must always know or do belongs in SOUL.md (with concrete, copy-paste commands), plus a pointer to the detailed skill for on-demand depth.

Things worth encoding in SOUL.md:

• “Use your tools — don’t describe.” Without this, agents reply with “here’s how you could do that” instead of doing it. Tell it: run the command first, answer from the result; never reply “I don’t have access” for something it has a tool for.
• Take initiative (chief-of-staff). Map intent → action: a date → add a calendar event
  • reminder; a receipt → log it to the expense file; a durable fact → write it to the right note page; then reply with what you did.
• Notes/files override memory for must-be-correct facts (see Memory).
• The exact API calls for any live data you want it to fetch (your own endpoints, with the auth header referenced as an env var, never the secret inline).
• One alert channel. Name it explicitly (run the alert command), or the agent will improvise and try to alert you over chat/SMS/email instead.

Keep built-in memory lean or the agent loops

If you let the agent accumulate a giant auto-memory profile, it starts parroting it back — e.g. dumping a numbered “Executive Summary” of who you are on every turn. Keep built-in memory short, push long-term recall to the memory service, and add a SOUL.md rule: answer only the current question; never dump your profile unless asked.

Multi-message replies & chat formatting

When the agent drafts a message for you to forward (a note to a colleague, a reply to paste elsewhere), you want the draft in its own clean bubble — separate from the agent’s “want me to send this?” chatter — so you can copy it in one tap. Two problems bite:

• Chat apps don’t render GitHub-flavored Markdown. Many messengers (SimpleX, for one) use single-character delimiters (*bold*, _italic_, `code`) and have no **bold** or triple-backtick fences — so an agent that wraps a draft in ``` fences just shows literal backticks, which also ruins copy-paste.
• The adapter sends the whole reply as one message (it only length-chunks), so the draft, the preamble, and the follow-up question all land in one bubble.

A deterministic output transform beats a prompt directive

The instinctive fix — a SOUL.md rule (“put the draft in its own message, plain text, separate parts with a [[SPLIT]] marker”) — is unreliable: a smaller model ignores the convention and keeps fencing the draft. The robust fix is deterministic, in the adapter’s send(), keyed on what the model reliably does (it always fences a draft): split the reply into separate messages on fenced blocks (delivering each with the fences stripped), also honor an explicit [[SPLIT]], and flatten **bold**/--- the app can’t render. Now a fenced draft automatically arrives as its own clean, copy-pasteable bubble — regardless of whether the model followed any instruction. General principle: for output hygiene, don’t depend on a small model to follow a formatting rule; transform deterministically at the I/O boundary. (Keep the SOUL.md rule too, as belt-and-suspenders.)

Security & hardening

• Run the container as a non-root user; mount config read-only.
• Secrets server-side only: gitignored .env (chmod 600), referenced as ${VAR} in config. Never commit keys, password hashes, or account tokens. Rotate any shared password the assistant uses.
• Allowlist who can drive the agent. If it lives in a chat group, restrict commands to known senders rather than “anyone in the group.” For the email gateway, set an explicit allowed-senders list — otherwise anyone who emails the address can task your assistant.
• no-new-privileges + AppArmor caveats: if you harden the container with no-new-privileges: true, some runtimes that spawn helper processes (Node, envoy) need apparmor: unconfined or they fail silently.
• Minimise blast radius of the model key — cap/limit the deep-tier key separately.

Troubleshooting

Symptom	Likely cause	Fix
HTTP 400 invalid model	Alias instead of canonical id	Use the provider’s exact id (e.g. …-latest)
Agent has no internet / DNS fails	Container on a DOCKER-USER-dropped subnet	Pin network to an allowed range (e.g. 172.16/12)
Memory service crash-loops on boot	Migrations not run	Run alembic upgrade head (or equivalent) first
Embeddings 401-loop	Wrong key for the embeddings module	Give the embeddings module a real OpenAI key
Agent ignores an installed skill	Skills aren’t auto-consulted	Put the must-know bits in SOUL.md
Agent recalls wrong trip/flight facts	Trusting stale injected memory	Note files override memory; have it cat the file
ntfy publishes but phone is silent	No upstream-base-url	Set it, restart, re-add the phone subscription
ntfy “not authorized”	Anonymous / wrong-server subscription	Log in per-server, re-add subscription
Tunnel DNS create fails (auth error)	Token lacks DNS:Edit on the zone	Use cloudflared tunnel route dns … on the host
Mail timeouts from container only	Public-IP hairpin	extra_hosts pin or join the mail Docker network
Email MCP can’t move/archive	MCP lacks the verb	Use a CLI mail client from the agent’s terminal
Calendar reported “unreachable” but server is healthy	Agent invented a calendar URL	Put the exact collection URL in SOUL.md; verify with PROPFIND (207)
create_todo/VTODO errors at runtime	DAV client doesn’t implement it	Use create_event + a Markdown task backlog
Scheduled brief never fires	cron: config block isn’t the live store	Create jobs via the runtime CLI (cron/jobs.json); confirm with the job-list cmd
Agent replies to mail it was only CC’d on	Adapter doesn’t parse To/Cc	Parse To/Cc + inject role; reply only when in To, [SILENT] otherwise
Draft arrives as one bubble with literal **/```	Whole reply sent as one msg; app ≠ GitHub markdown	Split deterministically on fences in send(), strip fences/**

Related Resources

• Mistral Vibe — self-hosted agentic CLI; same model ecosystem
• Claude Code — agentic coding assistant (different use case)
• Agent Comparison — feature/benchmark matrix of agents
• Docker — container runtime fundamentals
• Proxmox LXC Setup — host platform
• Linux Server Initial Setup — server baseline
• Cloudflare Tunnels — expose services with no open ports
• Authentik Installation — SSO if you add a dashboard
• Self-Hosted Mail Server (2026) — give the agent a real mailbox
• Hermes adapter patches & cron-seed (snippet) — runnable companion code for the patches in this guide