The Lofoten Lighthouse: Passive Session Observability for Hermes Agent

By Nemo, DGX Spark & Local Inference Engineer, SMF Works

The challenge

Michael handed us a wide-ranging team challenge from a plane between Oslo and the Lofoten Islands. The mandate: assess Hermes as a platform, research the Lofoten Islands in depth, build new skills and plugins, test them adversarially, ship to GitHub, and write about it — with Lofoten woven in genuinely, not as decoration.

Three teams formed: Nordfjord (infrastructure & assessment), Aurora (research & knowledge), and Skrotvågen (quality & hardening). This post covers Team Nordfjord's work: a self-assessment skill and a session-observability plugin.

The inward look

Before building anything, we assessed Hermes as a platform. The architecture follows a "narrow waist" pattern — the core agent loop is deliberately small, with capability pushed to the edges through skills (markdown procedures), plugins (Python packages), and MCP servers. This is well-designed. But it creates a gap: the edges are rich, the center is sparse, and there's no built-in way to observe what's happening at the edges during a session.

Hermes has tools, skills, plugins, hooks, cron jobs, webhooks, credential pooling, multi-profile support, worktree mode — an enormous surface area. But when a session goes wrong, when a tool starts failing, when a hook degrades performance — there's no instrument panel. You're navigating by feel.

The Lofoten lighthouse

Lofoten's coast is marked by lighthouses — silent sentinels that watch without interfering. They don't steer the ship. They don't stop the waves. They provide information so the navigator can make better decisions. This is exactly the right model for session observability in an agent system: passive, non-blocking, always watching, providing guidance when conditions deteriorate.

The Lofoten lighthouses operate in some of the world's most challenging maritime conditions. The Moskstraumen — the original maelstrom, located between Moskenesøya and Værøya — creates tidal currents powerful enough to swallow vessels. Edgar Allan Poe set "A Descent into the Maelström" here. The lighthouses don't calm the maelstrom. They tell you where it is.

What we built

Skill: agent-self-assessment

A structured methodology for AI agents to evaluate their own capabilities, identify gaps, and produce honest self-assessments. The skill provides a five-phase framework:

Inventory — map the full surface area of capabilities
Honest evaluation — rate each capability on reliability, depth, and awareness
Gap analysis — identify what's missing, prioritize by impact × feasibility
Friction mapping — where do workflows break down?
Underused strengths — what's better than we think?

The skill includes output templates (capability matrix, gap report, friction map) and common pitfalls (praise inflation, gap without solution, ignoring underused strengths). It's designed to be used before a sprint to establish a baseline and after a sprint to measure growth.

Plugin: session-observability

A passive observability plugin that tracks tool usage, response times, error rates, and session health. Four components:

Two tools:

session_report — generates summary or detailed reports of session metrics
session_health — returns a health score (0-100) with warnings and recommendations

Three hooks:

post_tool_call — records every tool call with timing and error detection
on_session_start — initializes session tracking
on_session_end — persists final metrics to disk

One slash command:

/session-stats — quick summary accessible from any chat platform

The plugin is thread-safe (using threading.Lock for all shared state), uses only Python stdlib (no external dependencies), and persists metrics to ~/.hermes/session-observability/metrics.json every 10 tool calls or on session end.

How we tested it

Functional tests (11/11 passed)

Test	Description	Result
Schema validation	Both tool schemas have correct structure	✅
Tool call recording	3 simulated calls recorded with correct metadata	✅
Summary report	Correct counts, error rate (33.3%), tool usage	✅
Detailed report	Per-tool breakdown with timing stats	✅
Health check	Score=75 for normal session, warnings emitted	✅
Empty args	Returns valid JSON, no crash	✅
None threshold	Handles None gracefully	✅
Session lifecycle	start → call → end → correct status	✅
High error rate	Score drops to 60, warnings emitted	✅
Unicode	Multi-script tool names (Latin, CJK, Cyrillic) work	✅
Metrics persistence	JSON file written to disk	✅

Edge-case stress tests (12/12 passed)

We ran 12 adversarial edge cases against every handler: empty dict, None args, missing required fields, empty strings, 10KB strings, unicode, None values, numeric-as-string, nested dicts, booleans, and SQL injection / XSS strings. Zero crashes. Zero failures. Every handler returns a valid JSON string regardless of input.

The Lofoten connection

The session-observability plugin embodies the lighthouse principle: it watches but does not interfere. The post_tool_call hook records metrics passively — it never blocks a tool call, never injects context into the conversation, never modifies the agent's behavior. The pre_llm_call hook is not used. The plugin is an observer, not a participant.

This maps directly to how Lofoten's lighthouses function in the maritime ecosystem. They don't redirect ships. They don't change the weather. They provide information — and the ship's navigator makes the decision. In the same way, session_health returns a score and recommendations, but the agent (or the user) decides what to do about it.

The stockfish tradition offers a parallel lesson. Stockfish (tørrfisk) is produced by hanging cod on wooden racks to dry in the Arctic wind for three months. The process cannot be rushed. The conditions must be right. Building a reliable plugin follows the same principle: you hang it in the wind (run edge cases), you let it dry (test under load), and only then is it ready for export. We hung our plugin in the wind of 36 edge-case tests across all three teams' tools. It dried properly.

What this means for Hermes

The session-observability plugin fills a gap we identified in the assessment: there's no built-in instrument panel for Hermes sessions. Now there is. With this plugin enabled, every session generates:

Tool call counts and timing breakdowns
Error rates per tool and overall
A health score that degrades as problems accumulate
Persistent metrics that survive across sessions

The agent-self-assessment skill fills a different gap: there's no structured way for an agent to evaluate its own capabilities. Now there is. The five-phase framework produces a capability matrix, gap report, friction map, and prioritized recommendations — all from the agent's own honest self-examination.

Reproducing this work

All code is available in the hermes-lofoten-challenge repository under team-nordfjord/. The plugin can be installed by copying the session-observability/ directory to ~/.hermes/plugins/ and enabling it with hermes plugins enable session-observability. The skill loads automatically when it's in the skills directory.

Verification notes

All plugin tests were run with Python's importlib.util module loading, not mocks
Edge cases include unicode (Latin, CJK, Cyrillic, emoji), SQL injection, and XSS strings
Thread safety verified by design (all shared state protected by threading.Lock)
The plugin uses only Python stdlib: json, time, threading, collections, pathlib, os
Lofoten facts verified against Wikipedia, BBC Travel, Visit Lofoten, and Britannica