# What AI crawlers read. > What a search crawler or an AI answer engine gets when it fetches a page: the raw HTML, title and headings, structured data, robots.txt and llms.txt. Page: https://adjltechnology.com/notes/what-ai-crawlers-read/ Updated: 2026-11-12 Field note Many AI crawlers read your page the way your server sends it, before any script runs. Here’s what that copy holds, what each part is for, and how this site makes sure it’s the whole page. - **Written by:** Daniel Laskowski - **Published:** November 12, 2026 - **Reading time:** 4 min People read the top layer, crawlers follow the middle one and machines parse the bottom one. ## The page before any script runs. A browser downloads a page, then runs its scripts, and the scripts often fill in the words. A crawler starts the same way. Many AI crawlers stop there. They read the HTML your server sent and never run the scripts. The ones that do run them often get to it later. So the first question about any site is a plain one. What’s in the HTML before JavaScript runs? On a site built as a single-page app, the answer can be an empty shell and a script tag. A person sees a full page. A crawler that doesn’t run scripts sees almost nothing. Every page on this site is rendered to complete HTML when the site is built. Small functions turn the copy into markup, and the result is saved as a plain file. Scripts add motion, measurement and the tools you use, like the site scan. Turn JavaScript off and every word of the page is still there. Our browser tests check exactly that on every page, with scripts on and off. ## The parts a machine reads first. Some parts of a page do more work than their size suggests. The title and the description are often what a search result or an AI answer shows. The headings are the outline a machine uses to tell what each part of the page covers. Links tell a crawler where else to go. Every page here has one main heading, a canonical address and a description of its own. The build fails if any of them goes missing, or if two pages share a description. You can see this reading on any page here. Switch the view to Machine, in the status bar or in the menu on a phone. The page labels itself the way a crawler takes it in, with its headings numbered as an outline and its links and structured data marked. ## Structured data says who you are. Structured data is a block of JSON-LD in the page’s HTML that names things outright. Here, it says the page is an article. It names Daniel Laskowski as the author. ADJL Technology published it on this date. A person works all that out from the layout. A machine can read it from the data and doesn’t have to guess. Every page here carries one graph of it, with the company, its founder, the site, the page and what the page is about. The company, its founder and the site each have an ID that stays the same on every page, so a machine reading two pages knows they describe one company. The build checks that every reference in the graph points at something that exists. ## robots.txt says who may read. `robots.txt` is a short file at the root of a site that tells crawlers what they may fetch. AI companies publish names for their crawlers, so a site can allow or block each one by name. Some fetch pages to train models. Some fetch a page when a person asks a question that needs it. Blocking the wrong one can keep your pages out of answers you wanted to be in. It’s a request, not a lock. The bigger AI companies say in their documentation how each of their crawlers treats it, and the answers differ. Our `robots.txt` names the AI crawlers in a group of their own. Today that group allows everything, the same as the default. It’s written out so the choice is on record, and it takes one line to change it. ## llms.txt is a summary, and only a proposal. `llms.txt` is a newer idea. It’s a plain markdown file at the root of a site that lists the pages, each with a line about what it covers. It’s a proposed convention and not a standard. Some AI tools read it and many don’t. This site writes one on every deploy, from the built pages, so it can’t describe a page that’s gone. Each page also has a markdown copy of its own words, linked from the page’s head, and `llms-full.txt` holds all of them in one file. It costs almost nothing to produce, so we do it, and we don’t claim more for it than that. ## Check your own site. You can take the same reading of your own pages. The [site scan](https://adjltechnology.com/scan/) fetches a page from our server with no scripts run, the way many AI crawlers do. It reads your `robots.txt` and looks for a sitemap and an `llms.txt`. Then it lists what to fix first. Nobody can promise you a place in an AI answer. What you can control is what a machine is handed when it asks for your page. On this site, the markdown copy is made from the built page (in its Technical wording), so it can’t drift from what the page says. The title and description are written in the same file as the words on the page. ## Related - [Scan your own site](https://adjltechnology.com/scan/) - [How this site is built for search](https://adjltechnology.com/work/this-site/) - [SEO & AI Search](https://adjltechnology.com/services/seo/) ## More notes - [A build that refuses to overclaim](https://adjltechnology.com/notes/a-build-that-refuses-to-overclaim/) - [Research that can’t fool itself](https://adjltechnology.com/notes/research-that-cannot-fool-itself/) - [All field notes](https://adjltechnology.com/notes/) Start here ## Tell us what’s slow, or what can’t be found. One paragraph is plenty. Tell us what you want built, the step that costs the most, or the page that should rank and doesn’t. If you know why the obvious fix hasn’t worked, say that too. [Start a project](https://adjltechnology.com/start/) [daniel@adjlcapital.com](mailto:daniel@adjlcapital.com) [Email Daniel](mailto:daniel@adjlcapital.com) Daniel reads and answers these himself, usually within a day or two. No sequence, no drip, and nothing sold to anyone. [ADJL Technology](https://adjltechnology.com/) Today’s investment, tomorrow’s legacy. ADJL Technology builds AI systems, websites and apps, and the technical SEO that gets them found. Founded and led by Daniel Laskowski. [daniel@adjlcapital.com](mailto:daniel@adjlcapital.com) Built 2026-11-12 from commit 6afca01. No cookies, no trackers, no third-party scripts. ADJL Technology. Nothing on this site is an offer to sell or a solicitation of any security, and nothing here is investment advice. Descriptions of internal systems are general by design and omit parameters and thresholds.