Implementing llms.txt for Better AI Accessibility
In my ongoing effort to make my website more accessible and useful to AI systems, I’ve implemented a new feature: the llms.txt
file. This file serves as a guide for language models (like ChatGPT, Claude, and others) to better understand and navigate the content on my site.
Why llms.txt?
The llms.txt
file is a new standard that aims to do for language models what robots.txt
does for search engines. By providing a clear, structured overview of my site’s content, I can help AI systems deliver more accurate and relevant information to users.
You can view my llms.txt
file here: https://mcginniscommawill.com/llms.txt
The original proposal for the llms.txt
format can be found at llmstxt.org. This proposal outlines the need for a standardized file to provide LLM-friendly content, offering brief background information, guidance, and links to detailed markdown files. The format is designed to be both human and LLM readable, using Markdown to structure the information.
How I Implemented It
Here’s how I set up the llms.txt
file for my Hugo site:
1. Configuration
First, I updated my config.toml
to define a new output format for llms.txt
:
[outputFormats.LLMS]
mediaType = "text/plain"
baseName = "llms"
isPlainText = true
[outputs]
home = ["HTML", "RSS", "JSON", "LLMS"]
2. Template Creation
Next, I created a template file at layouts/_default/index.llms.txt
to define the structure of the llms.txt
file. Here’s the template code:
# {{ .Site.Title }}
> {{ .Site.Params.description }}
## Content
### Posts
{{ range (where .Site.RegularPages "Type" "posts") }}
- [{{ .Title }}]({{ .Permalink }}): Published {{ .Date.Format "2006-01-02" }}
{{ end }}
### Guides
{{ range (where .Site.RegularPages "Type" "guides") }}
- [{{ .Title }}]({{ .Permalink }}): {{ .Description }}
{{ end }}
### Topics
{{ range (where .Site.RegularPages "Type" "topics") }}
- [{{ .Title }}]({{ .Permalink }}): {{ .Description }}
{{ end }}
### About
- [About Will McGinnis]({{ .Site.BaseURL }}/about/): Engineering leader and data science executive specializing in AI/ML and advanced analytics
### Coffee
- [Free Coffee]({{ .Site.BaseURL }}/coffee/): Will's offer to buy coffee for anyone who wants to chat about tech, startups, or AI
3. Dynamic Generation
By integrating this setup into my build process, the llms.txt
file is generated dynamically, ensuring it always reflects the latest content on my site.
The Benefits
With llms.txt
in place, language models can:
- Better understand the structure and purpose of my site
- Provide more accurate responses when users inquire about my content
- Navigate to relevant sections more easily
- Reduce “hallucinations” or made-up information about my site
Looking Ahead
This implementation is just the beginning. As the standard evolves, I plan to continue refining my approach to ensure that my site remains accessible and useful to AI systems. I’m excited to see how this new feature enhances the interaction between my site and language models.
If you have any thoughts or suggestions on this implementation, feel free to share them in the comments or reach out directly for a chat over coffee.
Subscribe to the Newsletter
Get the latest posts and insights delivered straight to your inbox.