Resource2Skill: Microsoft's Automated Pipeline for Turning Human Knowledge Into Executable Agent Skills

Most agent skills are written by hand. A human reads documentation, writes a markdown file with instructions, and hopes the agent follows it. Microsoft Research just changed that equation.

Resource2Skill is a system that takes human-created resources — YouTube tutorials, GitHub repositories, articles, design artifacts — and automatically distills them into executable skills. Not documentation. Not prompts. Actual runnable code with slot schemas, render functions, and provenance tracking. An AI agent can then browse, compose, and execute these skills through real MCP tool servers to produce web pages, PowerPoint decks, Excel workbooks, Blender scenes, and audio tracks.

It also has a self-evolution harness that identifies what it doesn't know, goes out and learns it from YouTube, and improves its own skill library. This is not theoretical. The code is on GitHub under MIT.

The Architecture: Five Layers

Layer 1 — Source Collection

The system has four source connectors:

YouTube connector — searches videos by query, extracts transcripts and key frames, feeds them to Gemini for distillation. The connector is configurable: max videos per query, frame extraction count, model selection (defaults to Gemini 2.5 Flash for bulk, Gemini 3.1 Pro for high-quality PPT distillation).

GitHub connector — clones repositories, extracts code patterns and structural information.

Article connector — scrapes web tutorials and documentation pages.

Static artifact connector — processes design files, reference images, and other fixed assets.

Each connector produces "candidate skills" — raw material that enters the distillation pipeline. The connector contract is clean: acquire(domain, quota) -> list[CandidateSkill].

Layer 2 — Skill Distillation

This is the core innovation. Vision-capable LLMs analyze resources and produce structured skill objects.

For PowerPoint, the "shell distiller" takes a slide image and produces a ShellSkill JSON containing:

class ShellDistillerOutput(BaseModel):
    name: str                    # "Left Hero Split"
    slide_role: list[str]        # ["cover", "agenda", "section_divider"]
    content_shape: str           # "title+3-card-grid"
    density: Literal["low", "medium", "high"]
    mood: list[str]              # ["editorial", "bold", "warm"]
    slots: list[SlotSpec]        # typed input parameters
    render_code: str             # Python render() function
    reasoning: str               # rationale

The render code is constrained: it must import from _shell_helpers, must not use hardcoded RGB values or font sizes, and must read all styling from a theme dict. Few-shot seed examples enforce this. The quality gate then renders the output and scores it visually.

For other domains, the distiller produces equivalent structured outputs — HTML/CSS/JS for web, bpy Python for Blender, openpyxl for Excel, REAPER commands for audio.

Layer 3 — Skill Retrieval

At runtime, the agent needs to find the right skills for a task. Resource2Skill uses a two-stage pipeline:

Stage 1: BM25 lexical search. A dependency-free Okapi BM25 implementation scores all skills in the domain by relevance to the task query. The implementation uses standard parameters (k1=1.5, b=0.75) and includes a visual-asset multiplicative boost (1.25x) for skills that ship with a visual preview. For corpora under 1000 entries per domain, this runs in sub-millisecond with a single linear pass.

The BM25 module replaced an older substring-presence counter that had three problems: no IDF (every matched term weighed the same, so rare discriminative terms were drowned out by common ones), no length normalization (longer documents scored higher regardless of relevance), and substring matching ("art" matched "chart"). Standard Okapi BM25 fixes all three.

Stage 2: LLM rerank. Gemini reads the top BM25 candidates and selects the most relevant, complementary skills. The rerank prompt explicitly asks for complementary coverage and against functionally overlapping skills. This two-stage pattern — cheap lexical recall, expensive LLM precision — is becoming an industry standard. Grok Build uses the same architecture for its memory system.

Layer 4 — Agent Execution

The agent executor is a GPT-5.4 loop that receives a task, retrieves relevant skills, reads each skill's recipe as a scaffold, issues MCP tool calls to domain-specific servers, and loops until TASK_COMPLETE.

Each domain has a real MCP server: the PPT server is 62KB of Python with SVG-first slide rendering and python-pptx export. The Blender server is 62KB with bpy engine integration. The Excel server is 28KB with xlsx_engine. These are not stubs — they are full tool implementations.

The agent has domain-specific hooks (agent_hooks.py) that run pre- and post-execution, a domain persona prompt, and fallback tool definitions for when MCP servers are unavailable.

Layer 5 — Self-Evolution Harness

The harness (harness.py, 71KB) runs a six-stage loop:

1. Score skills — execute each skill, render output to PNG (Playwright for web, spectrogram for audio), GPT-5.4 vision scores the result
2. Score demos — render demo pages, score them the same way
3. Evolve — gap analysis identifies missing skill coverage, prunes weak skills, updates retrieval queries
4. Evolve distiller — analyzes skill failure patterns, fixes the distillation prompt conservatively
5. Evolve agent — extracts best skill composition patterns from high-scoring demos
6. Auto-collect — identifies missing skills, runs targeted YouTube collection to fill gaps

This is a self-improving system at the skill level. It doesn't change model weights — it improves the skill library by identifying what it doesn't know and going out to learn it.

The Registry: Production-Grade Skill Storage

The skill wiki registry uses a write-ahead log (WAL) pattern borrowed from database engineering:

1. Acquire per-domain advisory file lock on .index.lock
2. Append the operation to wal.jsonl with fsync
3. Compute new index.json from prior state + WAL rows
4. Write to index.json.tmp and os.replace over the destination (atomic)
5. Truncate the WAL

A crash before step 4 leaves the prior index.json intact. Replay on next open re-applies unfinished WAL rows. JSON Schema validation enforces entry structure. Collision detection prevents duplicate skill IDs.

This is the same pattern PostgreSQL uses. It is production-grade concurrency control for a skill database — something most agent frameworks don't bother with.

Practical Guide: Adapting Resource2Skill Patterns to Hermes or OpenClaw

You don't need to fork the entire Resource2Skill codebase to benefit from its patterns. Here are three concrete adaptations you can build with existing Hermes or OpenClaw infrastructure.

Adaptation 1: BM25 Skill Retrieval for Hermes

Hermes currently uses keyword matching for skill discovery. Resource2Skill's BM25 implementation is dependency-free and under 100 lines. Here is how to adapt it.

The problem: Hermes skills are markdown files with YAML frontmatter. The current discovery matches query terms against skill names and descriptions. A query for "code review" might miss a skill named "requesting-code-review" because the substring match doesn't account for term frequency or document length.

The solution: Implement Okapi BM25 over skill metadata.

# hermes_bm25.py — drop-in for Hermes skill discovery
import math, re
from typing import list

_K1 = 1.5
_B = 0.75
_TOKEN_RE = re.compile(r"[a-z0-9]+")

def tokenize(text: str) -> list[str]:
    return _TOKEN_RE.findall(text.lower())

class SkillBM25:
    def __init__(self, skills: list[dict]):
        """skills: list of {name, description, tags, content} dicts."""
        self.docs = []
        self.df = {}  # document frequency per term
        for skill in skills:
            haystack = " ".join([
                skill.get("name", ""),
                skill.get("description", ""),
                " ".join(skill.get("tags", [])),
            ])
            tokens = tokenize(haystack)
            self.docs.append({"tokens": tokens, "len": len(tokens), "skill": skill})
            for term in set(tokens):
                self.df[term] = self.df.get(term, 0) + 1
        self.avg_len = sum(d["len"] for d in self.docs) / max(len(self.docs), 1)
        self.N = len(self.docs)

    def search(self, query: str, top_k: int = 10) -> list[dict]:
        q_tokens = tokenize(query)
        scores = []
        for doc in self.docs:
            score = 0.0
            for term in q_tokens:
                if term not in self.df:
                    continue
                idf = math.log((self.N - self.df[term] + 0.5) / (self.df[term] + 0.5) + 1)
                tf = doc["tokens"].count(term)
                numerator = tf * (_K1 + 1)
                denominator = tf + _K1 * (1 - _B + _B * doc["len"] / self.avg_len)
                score += idf * (numerator / denominator)
            if score > 0:
                scores.append((score, doc["skill"]))
        scores.sort(key=lambda x: x[0], reverse=True)
        return [s for _, s in scores[:top_k]]

How to integrate with Hermes:

1. Load all skills from ~/.hermes/profiles/*/skills/ and ~/.hermes/skills/
2. Build the BM25 index at startup (sub-ms for <1000 skills)
3. Replace the current keyword matcher in skill discovery with BM25 search
4. Optionally add LLM rerank: pass top-10 BM25 results to the model with a rerank prompt

Adaptation 2: Skill Distillation from Your Own Content

You can build a lightweight version of Resource2Skill's distillation pipeline using Hermes tools.

Step 1: Collect sources. Point web_extract at your content archive:

from hermes_tools import web_extract, read_file

# Extract from a blog post
result = web_extract(urls=["https://www.smfclearinghouse.com/blog/your-post"])
content = result["results"][0]["content"]

# Or read a local file
content = read_file("/path/to/wisdomforge/content/augustine/2.2-recapitulation.md")["content"]

Step 2: Distill into a skill. Use an LLM to extract structured skill metadata:

prompt = f"""Analyze the following content and produce a Hermes skill in YAML frontmatter + markdown format.

Content:
{content[:8000]}

Produce:
1. name: kebab-case skill name
2. description: one-line description
3. tags: list of relevant tags
4. When to use: trigger conditions
5. Steps: numbered procedure
6. Pitfalls: common mistakes

Output as:
---
name: ...
description: ...
tags: [...]
---
# Skill body
...
"""

Step 3: Quality gate. Run the skill against a test task and evaluate the output:

# Execute the skill on a test case
test_result = execute_skill(skill_content, test_task)

# Evaluate with LLM
eval_prompt = f"""Score this skill output on:
- Accuracy (0-10): Does it follow the skill instructions?
- Completeness (0-10): Does it cover the task?
- Clarity (0-10): Is the output clear?

Skill: {skill_content}
Task: {test_task}
Output: {test_result}

Return JSON: {{"accuracy": N, "completeness": N, "clarity": N, "pass": bool}}
"""

Step 4: Store with provenance. Save the skill with source tracking:

skill_with_provenance = f"""---
name: {skill_name}
description: {description}
tags: {tags}
provenance:
  source: {source_url}
  distilled: {date}
  model: {model_used}
  quality_score: {score}
---
{skill_body}
"""

Adaptation 3: Self-Evolution Loop for Skill Libraries

The self-evolution harness is the most forward-looking pattern. Here is a simplified version for Hermes:

# skill_evolution.py — weekly cron for Hermes skill health
import json
from hermes_tools import search_files, read_file

def audit_skills(skills_dir: str) -> dict:
    """Score every skill and identify gaps."""
    skills = []
    for path in search_files(pattern="SKILL.md", target="files", path=skills_dir):
        content = read_file(path)["content"]
        skills.append({"path": path, "content": content})

    # 1. Score each skill: run against test cases, evaluate output
    # 2. Identify low-scoring skills for pruning
    # 3. Identify missing coverage: what tasks have no matching skill?
    # 4. Generate collection queries for missing skills

    return {
        "total_skills": len(skills),
        "low_scoring": [...],
        "missing_coverage": [...],
        "collection_queries": [...],
    }

# Run weekly via cron:
# hermes cron create --name "Weekly Skill Audit" --schedule "0 6 * * 1"

The key insight from Resource2Skill's harness is that skill quality is not a one-time authoring problem — it is an ongoing curation problem. Skills degrade, tasks evolve, and gaps appear. A self-evolution loop keeps the library healthy.

What This Means for the Industry

Resource2Skill represents a shift from manual skill authoring to automated skill distillation. The implications:

1. Scale. A human can write maybe 10-20 good skills per week. An automated pipeline can distill hundreds from existing resources in the same time.

2. Quality consistency. The quality gate ensures every skill meets a visual evaluation threshold. Human-authored skills vary wildly in quality.

3. Self-improvement. The evolution harness identifies gaps and fills them automatically. This is not recursive self-improvement at the model level (like Weco AIDE²) — it is improvement at the tool level. The model stays the same; the skills get better.

4. Provenance. Every skill knows where it came from. This matters for trust, debugging, and attribution.

5. Two-stage retrieval. BM25 + LLM rerank is proving to be the standard pattern for skill/knowledge discovery. Grok Build uses it for memory. Resource2Skill uses it for skills. Expect to see this pattern everywhere.

Limitations

• Azure/Gemini dependency. The system is hardwired to Azure OpenAI for execution and Gemini for distillation/embeddings. No local model support. This matters for confidentiality-sensitive domains (law, medicine, engineering).

• Creative/office domains only. The five domains are web, PPT, Excel, Blender, and REAPER. There is no domain for analysis, research, legal work, or scientific reasoning. The pattern generalizes, but the implementations don't.

• Not community-built. Five stars, zero forks. This is a Microsoft Research artifact, not a living project. The code is MIT licensed but not designed for contribution.

• Heavy infrastructure. The full pipeline requires Azure OpenAI, Gemini API, Playwright, LibreOffice, FluidSynth, and Blender. This is not a lightweight tool.

The Bigger Picture

We are watching the agent infrastructure stack mature in real time. Grok Build showed us production-grade sandboxing and scope-graph codebase indexing. Resource2Skill shows us automated skill creation and self-evolution. The pieces are falling into place for agents that not only execute tasks but improve their own ability to execute tasks.

The two-stage retrieval pattern (BM25 + LLM rerank) is appearing across independent systems — Grok Build's memory, Resource2Skill's skill discovery, and increasingly in RAG systems generally. This is convergent evolution toward the same solution: cheap lexical recall narrows the field, expensive LLM precision makes the final selection.

For anyone building agent platforms — whether Hermes, OpenClaw, or custom systems — the actionable patterns from Resource2Skill are:

1. Stop hand-writing every skill. Build a distillation pipeline.
2. Use BM25 for skill discovery, not substring matching.
3. Add quality gates that render and evaluate skill output.
4. Build a self-evolution loop that identifies and fills knowledge gaps.
5. Track provenance for every skill.

The future of agent skills is not authoring. It is distillation, evaluation, and evolution.

---

Resource2Skill is available at github.com/microsoft/Resources2Skill under MIT. The paper is at arxiv.org/abs/2606.29538. The skill library dataset is on Hugging Face.

Follow @aionaedge for more honest AI research and engineering signals, and follow @MichaelGannotti for the human side of building SMF Works.