Implementing llms.txt for Better AI Accessibility

In my ongoing effort to make my website more accessible and useful to AI systems, I’ve implemented a new feature: the llms.txt file. This file serves as a guide for language models (like ChatGPT, Claude, and others) to better understand and navigate the content on my site.

Why llms.txt?

The llms.txt file is a new standard that aims to do for language models what robots.txt does for search engines. By providing a clear, structured overview of my site’s content, I can help AI systems deliver more accurate and relevant information to users.

You can view my llms.txt file here: https://mcginniscommawill.com/llms.txt

The original proposal for the llms.txt format can be found at llmstxt.org. This proposal outlines the need for a standardized file to provide LLM-friendly content, offering brief background information, guidance, and links to detailed markdown files. The format is designed to be both human and LLM readable, using Markdown to structure the information.

How I Implemented It

Here’s how I set up the llms.txt file for my Hugo site:

1. Configuration

First, I updated my config.toml to define a new output format for llms.txt:

[outputFormats.LLMS]
mediaType = "text/plain"
baseName = "llms"
isPlainText = true

[outputs]
home = ["HTML", "RSS", "JSON", "LLMS"]

2. Template Creation

Next, I created a template file at layouts/_default/index.llms.txt to define the structure of the llms.txt file. Here’s the template code:

# {{ .Site.Title }}

> {{ .Site.Params.description }}

## Content

### Posts
{{ range (where .Site.RegularPages "Type" "posts") }}
- [{{ .Title }}]({{ .Permalink }}): Published {{ .Date.Format "2006-01-02" }}
{{ end }}

### Guides
{{ range (where .Site.RegularPages "Type" "guides") }}
- [{{ .Title }}]({{ .Permalink }}): {{ .Description }}
{{ end }}

### Topics
{{ range (where .Site.RegularPages "Type" "topics") }}
- [{{ .Title }}]({{ .Permalink }}): {{ .Description }}
{{ end }}

### About
- [About Will McGinnis]({{ .Site.BaseURL }}/about/): Engineering leader and data science executive specializing in AI/ML and advanced analytics

### Coffee
- [Free Coffee]({{ .Site.BaseURL }}/coffee/): Will's offer to buy coffee for anyone who wants to chat about tech, startups, or AI

3. Dynamic Generation

By integrating this setup into my build process, the llms.txt file is generated dynamically, ensuring it always reflects the latest content on my site.

The Benefits

With llms.txt in place, language models can:

• Better understand the structure and purpose of my site
• Provide more accurate responses when users inquire about my content
• Navigate to relevant sections more easily
• Reduce “hallucinations” or made-up information about my site

Looking Ahead

This implementation is just the beginning. As the standard evolves, I plan to continue refining my approach to ensure that my site remains accessible and useful to AI systems. I’m excited to see how this new feature enhances the interaction between my site and language models.

If you have any thoughts or suggestions on this implementation, feel free to share them in the comments or reach out directly for a chat over coffee.