Distilling Skills From a Knowledge Base You Build While You Work

This is the third post in a small run. The first was about Google’s Open Knowledge Format, a tiny markdown spec for handing curated context to AI agents. The second was about Swamp, a framework for running deterministic, searchable automation that agents operate. This one is the payoff: what you get when you point the two at the same problem. The pattern I want to describe is a loop that grows a knowledge base across every project you own while the agents work, and then distills that accumulated knowledge into reusable skills you can share.

Two flywheels that feed each other

There are two separate good ideas here, and the trick is wiring them so one feeds the other.

The first is knowledge capture. Every time an agent works in one of your repos, it learns things: the stack, the weird gotcha that cost it an hour, the preference you expressed in a code review. Normally that evaporates when the session ends. If instead you make the agent write those facts down in a structured, portable format, you accumulate a real memory of your whole portfolio.

The second is skill distillation. Once you have that memory, patterns start repeating across projects. The same mistake shows up in three repos. The same convention is worth enforcing everywhere. Those recurring patterns are exactly what a reusable skill should encode. So the second flywheel reads the accumulated knowledge and turns the repeated bits into skills.

OKF is the substrate for the first flywheel. Swamp is the engine that turns both of them, on a schedule, deterministically, with a searchable record. Let me walk through how they fit.

Growing the knowledge base with OKF

The knowledge base is an OKF bundle: a directory of markdown files, each one a concept, each with a little YAML frontmatter and a required type field. I gave every project its own sub-bundle, a subdirectory with an auto-generated index.md whose resource field points at the repo and whose links fan out to a few concept documents. In my setup those are a codebase doc, a gotchas doc, and a preferences doc, all of type Concept.

The important design choice is who writes them. The agents do, as a side effect of doing their real work. When an agent finishes a task in a repo, part of its instructions is to append any durable facts it learned to that repo’s concept docs, under a fixed heading. Codebase facts go in one place, gotchas in another, expressed operator preferences in a third. It’s the LLM-wiki pattern the OKF authors talk about: the bookkeeping that makes humans abandon personal wikis is exactly what an LLM doesn’t mind doing.

The other choice that pays off later is structured frontmatter. Each doc carries fields like language, frameworks, key dependencies, and a small set of normalized topical tags. That structure is what lets you do the next part mostly without an LLM.

Turning the crank with Swamp

None of that maintains itself, and this is where Swamp earns its place. The capture and the distillation are jobs in a scheduled workflow, not scripts I remember to run.

Each work cycle, the ideation and implementation jobs do their normal thing and append to the knowledge base as they go. Because a Swamp workflow is a DAG with dependency ordering, I can add a rollup job that depends on those and therefore runs after them. The rollup reads the whole bundle and synthesizes cross-project views: a stack matrix, a digest of recurring gotchas, a set of security notes, a map of which projects neighbor which by shared tags.

The thing I want to stress is that the rollup is almost entirely mechanical. Because OKF keeps the structure in plain YAML frontmatter, you can derive all of those views by parsing files and clustering by tag, with no model in the loop at all. That matters for cost and for determinism. You reserve the LLM for the one step that genuinely needs judgment, which is the writing.

And because every one of these runs produces a versioned, tagged artifact in Swamp’s data layer, the whole history is queryable. I can ask what the rollup looked like last week, or which cycle first noticed a given gotcha, without spelunking through logs.

The distillation pass

Now the second flywheel. After the rollup runs, a final job goes looking for skills worth extracting.

It starts from the synthesized digests, not from raw repos, so it’s reasoning over patterns that already recur across projects rather than one-off details. It feeds that digest to an agent with a narrow brief: propose exactly one focused skill. Either sharpen an existing skill that the recurring evidence shows is incomplete, or add a new one for a pattern that keeps coming up and isn’t covered yet. One per cycle, deliberately, because a firehose of skill PRs is worse than none.

The output is a pull request against a shared skills repository. Mine follows the Claude Code plugin marketplace layout, where each skill is a skills/<name>/SKILL.md and a marketplace manifest bundles them. A good public example of that shape is my python-skills repo. The agent writes the SKILL.md, registers it in the manifest, and opens the PR on a dedicated branch.

The PR is the whole review gate. Nothing gets into the shared skills repo without me merging it, which is the point. The agent proposes; a human decides. To keep it from spamming, the job counts how many of its own skill PRs are already open and stops proposing new ones past a cap. Anti-spam by construction, not by hoping the model behaves.

There’s a nice closing of the loop here. The shared skills repo is also a source the agents read from when they do their normal work. So a skill distilled out of what the agents learned last month becomes house-style guidance the agents follow next month. Knowledge captured in OKF becomes a skill, and the skill flows back into how the work gets done.

If you want to build something like this

You don’t need my exact setup, and honestly you shouldn’t copy it wholesale. The reusable shape is this:

1. Pick a knowledge format that is just files. OKF is a good default because it’s a one-page spec, it’s plain markdown, and the required surface is a single type field. You want something you can grep, render, and hand to any model with no vendor in the middle.
2. Make writing to it a side effect of the work, not a separate chore. Agents append durable facts to a per-project doc as they finish tasks. Give the docs structured frontmatter so you can query them later without a model.
3. Run the whole thing on a scheduler that produces a searchable record. This is what Swamp gives you: typed jobs, dependency ordering so distillation runs after capture, and a data layer so every run is queryable instead of ephemeral.
4. Keep the synthesis mechanical wherever you can. Derive your rollups by parsing frontmatter and clustering by tag. Spend the LLM only on the final authoring step.
5. Make distillation propose, and make a human dispose. One focused skill PR per cycle, into a shared repo, behind a merge gate, with a cap on open PRs so it can’t run away.

The reason this combination works is that each tool is doing the thing it’s actually good at. OKF makes the accumulated knowledge portable and structured enough that most of the synthesis is a parsing problem. Swamp makes the capture and distillation repeatable, scheduled, and recorded instead of a one-off script that rots. And the skills repo turns the whole thing into a compounding asset: the more the agents work, the more they know, and the better the house style they work under gets. That is the part that has actually held up in practice.