How to Set Up an llms.txt File: Examples and Best Practices

Key Takeaways

• When setting up an llms.txt file, place the llms.txt file in your website’s root directory so AI agents can discover and prioritize your authoritative content list.
• Use Markdown to provide high-signal content, avoid noisy HTML, and directly improve your website’s citation “stickiness” in answer engines.
• Define a ## Core Context section that acts as a system prompt for the LLM, ensuring it understands your technical architecture and brand constraints.

AI answer engines decide which sources to read, summarize, and cite before users ever see organic search results. To influence this process, you need to set up an llms.txt file: place it in your website’s root directory and clearly define your site’s content context and priority content. 

In this guide, we’ll explain why setting up llms.txt matters, how to configure it step by step, and share practical examples, monitoring methods, and best practices.

Why Setting Up llms.txt Matters for AI Search

The llms.txt file specifies which content AI answer systems can access and use. Here’s why that matters:

1. AI Answers Are Appearing Before Organic Search Results

Today, roughly 50% of queries end with an AI-generated answer. If AI models cannot access your content, it may be skipped entirely, even if your SEO is strong. llms.txt helps ensure your pages have a chance to appear in AI answers, not just traditional rankings.

2. You Can Control What AI Can Read and Learn From

Not every page should be visible to AI. With llms.txt, you can allow helpful content such as blogs and guides while blocking sensitive areas, such as dashboards, staging URLs, and permission-gated pages. This gives you clear control over access without disrupting traditional SEO crawlers.

3. Clear Access Permissions Increase Your Chances of Being Cited

AI systems are more likely to cite sources they can access clearly. A properly configured llms.txt file reduces ambiguity and helps your content be cited by name, rather than summarized without attribution.

💡 Practical Tip:Think of llms.txt as robots.txt for AI. It will not directly improve rankings, but it determines whether you get a chance to participate in the conversation.

Once llms.txt is live, the real question is: is it actually working? 

WorkDuo helps youtrack brand mentions in AI search, citations, and visibility changes, so you can see what works and adjust in real time.

Nowsign up, and start tracking your brand and competitors across ChatGPT, Perplexity, Gemini, and other platforms.

How to Set Up an llms.txt File: Step-by-Step Guide

The key to setting up llms.txt is to clearly communicate your content access preferences to AI crawlers. Follow the steps below to make sure the file is easy to find, easy to read, and clear in intent.

Step 1: Create the File

Create a file named llms.txt in your project’s root directory.

File location：https://yourdomain.com/llms.txt Or the top-level folder of your GitHub repo.

Step 2: Write an H1 Title and Summary

Start with a single # heading, followed by a blockquote >. 

This helps AI understand what it is looking at within seconds.

Step 3: Define the Core Context

Use ## as the H2 heading to give AI its “operating constraints” and prevent it from generating code in the wrong language or style.

Step 4: Add Categorized Resource Links

This is the “map” for AI. Use Markdown links to point to specific files or URLs. 

Group them by importance.

Step 5: Add “Full” Context (Optional)

For AI systems with large context windows, you can provide a link to a second, larger file that contains the full content of your documentation.

You can add it to the bottom of llms.txt.

Step 6: Deploy and Verify

Upload the file and confirm it is publicly accessible.

Checklist:

• Open yourdomain.com/llms.txt。
• in your browser and confirm the file is plain text, without being wrapped in HTML.
• Make sure every link in the file is an absolute URL starting with https:// so AI can easily follow it.

llms.txt Configuration Examples

The examples below show common, effective ways teams structure llms.txt.

1: SaaS and Product Documentation

This setup emphasizes clarity around technical specifications and APIs. 

Organizing links under “Core Docs” and “API Reference” helps AI find specific features accurately without getting lost in marketing language. 

As a result, when a user asks, “How do I integrate X?” AI can pull the correct endpoint from your source list.

2: E-commerce and Product Catalogs

For e-commerce, the priority is presenting product attributes and inventory status. 

This template uses a “Product Specs” section to list specification sheets for each category. 

This helps AI assistants provide accurate comparisons and pricing information, increasing the chance your store is recommended for specific purchase intent instead of being replaced by generic competitors.

3. Content-Heavy Blogs and Media

Publishers can use llms.txt to highlight their most authoritative evergreen content. 

This example separates “Breaking News” from “Reference Guides.” 

By providing clear paths to your long-form pillar content, you can prevent AI models from citing outdated social posts or short snippets, ensuring your complete, in-depth research becomes the primary source.

How to Monitor AI visibility After Setup

After llms.txt goes live, keep tracking how your content appears in AI-generated answers over time. 

Use the checklist below to track meaningful signals.

Track Brand Mentions in AI Answers

In AI search tools, search for questions related to your product, category, or content topics, and record:

• Whether your brand name appears
• How often and how consistently it is mentioned
• What descriptions are used

Focus on exposure and consistency, not ranking position.

Watch for Direct and Indirect Citations

Some AI answers link directly to sources, while others mention a brand or page without including a link.

Monitor:

• Which pages are mentioned by name
• Whether core pages, such as guides, documentation, and resources, appear more often
• Whether outdated or excluded pages gradually stop appearing over time

This helps you verify whether your initial scope was set correctly.

Compare visibility Before and After Setup

After updating llms.txt, briefly record the following:

• The questions you test regularly
• The answer content before the change
• The answer content a few weeks later

You are not testing whether changes are forcibly applied. You are looking for directional movement.

Identify Which Content AI Pulls Most Often

Over time, patterns will emerge:

• Certain guides or documentation are cited repeatedly
• Certain sections never appear

These insights help you decide:

• Which content to optimize
• Which sections should be presented more prominently
• Which pages should remain excluded

📌 Note:There are no immediately visible signals and no guaranteed feedback loop. Monitoring AI visibility is an ongoing process of observation and gradual accumulation. Tools like WorkDuo can automatically track your brand’s footprint across all major answer engines, showing where you are cited and where competitors are gaining an advantage.

Start using WorkDuo today and see how it works.

llms.txt Best Practices 

The following best practices help keep llms.txt useful, accurate, and easy to maintain, with a focus on clarity and explicit intent.

1. Use Absolute URLs for Every Link

When adding links to llms.txt, always use absolute URLs.

Relative paths (for example, /docs/setup) depend on context, while absolute URLs remove ambiguity and make the destination clear.

Recommended practice:

• https://yourdomain.com/docs/setup

Avoid

• /docs/setup

This principle applies whether the format is plain text or Markdown.

2. Use simple, readable text pages

When linking to resources, prioritize pages with text content that is easy to read.

Avoid:

• Pages that require login or extensive interaction to view
• Content blocked by scripts, pop-ups, or authentication

This is about accessibility, not training preference.

Markdown is widely used precisely because it is easy to read.

3. Use llms.txt as a guide, not a data dump

Treat llms.txt as a map, not a content repository.

Use llms.txt to:

• Describe website content in clear, plain language
• Link to the most important public resources
• Indicate which content is in scope

Avoid:

• Copying and pasting full documentation into the file
• Turning it into a knowledge base
• Assuming a larger file performs better

If you maintain additional reference files, treat them as tool-specific optional items, not required components of best practice.

Mistakes to avoid when setting up llms.txt 

Some mistakes can make this file effectively meaningless.

Common mistakes include:

1. Using marketing language instead of factual statements

Avoid overly sales-driven or promotional language. It adds noise and does not help clarify what the website actually contains.

Avoid:

• “The world’s #1 platform”
• “The most impressive solution”
• Vague, exaggerated claims without context

Use accurate, descriptive language to explain what the website or product actually does.

Examples:

• Instead of “amazing performance,” say “low-latency API”
• Instead of “cutting-edge user interface,” say “interface built with React”

The goal is clarity, not persuasion.

2. Placing the file outside the root directory

If llms.txt is not accessible through the following path, it may not be discovered at all:

Common incorrect locations include:

• /assets/
• /static/
• /uploads/

Place the file in the folder that maps directly to the website root, following the same placement rule as robots.txt.

3. Letting content context become outdated

If llms.txt references content, versions, or sections that no longer exist, it creates confusion.

This can happen when:

• Documentation is reorganized
• Major versions are updated
• Legacy resources remain listed after being deprecated

Whenever public content changes, llms.txt should be updated.

If you already have a release or deployment checklist, add a simple check to confirm:

• Whether the links still work
• Whether the description still matches the current website

Make AI content governance and visibility measurable with WorkDuo

You can set up an llms.txt file in minutes. The harder part is knowing whether it actually affects how your content appears in AI-generated answers.

Without visibility data, you can only make assumptions:

• Which pages AI systems are using as sources
• Whether recent updates caused any changes
• Whether important content is being overlooked

WorkDuo removes that uncertainty and helps you clearly see:

• How your content appears in AI-generated answers
• Which pages are being cited
• How visibility changes after updating your llms.txt configuration

WorkDuo helps you treat llms.txt not as a one-time task, but as an ongoing process you can measure, validate, and optimize.

Start using WorkDuo today and see how it works in practice.

FAQ: Setting up an llms.txt file

What is an llms.txt file?

llms.txt is a public text file placed in a website’s root directory. It summarizes the website’s content and points to the most important public pages.

This file is a proposed open standard introduced by Answer.AI co-founder Jeremy Howard in September 2024 to give AI systems clearer, publisher-defined content context.

In practice, it can help you:

• Explain the purpose of your website in plain text
• Highlight important public resources
• Clarify which sections are intentionally included in scope

What is the main difference between llms.txt and robots.txt?

The difference lies in who they serve and what they do.

• robots.txt is for traditional web crawlers, such as search engines
• llms.txt is for AI systems and answer engines

They serve different purposes and can be used together.

Robots.txt:

• Controls crawler access permissions (allow/disallow)
• Used by search engines
• Enforces crawling rules

llms.txt:

• Describes the main focus of a website’s content
• Highlights important public content
• Does not control or block access permissions

Where should an llms.txt file be placed?

To make your configuration automatically discoverable by AI, the file must be placed in the project’s root directory.

• Correct setup：https://yourdomain.com/llms.txt
• Incorrect setup:https://yourdomain.com/assets/llms.txt

If you use a static site generator or a framework such as Next.js, place the file in the public folder so it appears at the top-level URL after deployment.

How llms.txt impacts Answer Engine Optimization (AEO)

llms.txt does not directly improve rankings. Its impact on AEO is indirect and context-dependent.

In practice, it can reduce ambiguity when AI systems generate responses based on your website content.

It can influence AEO by:

• Clarifying the scope of your website’s content and reducing misinterpretation
• Guiding AI systems to authoritative public pages, helping you rank in AI Overviews。
• Helping generated answers better align with the content you want to present

However, it cannot:

• Guarantee citations
• Force inclusion in answers
• Replace the quality and authority of the content itself

In summary:

llms.txt supports AEO by improving clarity and content context—not as a ranking or control mechanism.