The page before any script runs.
A browser downloads a page, then runs its scripts, and the scripts often fill in the words. A crawler starts the same way. Many AI crawlers stop there. They read the HTML your server sent and never run the scripts. The ones that do run them often get to it later.
So the first question about any site is a plain one. What’s in the HTML before JavaScript runs? On a site built as a single-page app (a site whose pages are put together in the browser by scripts), the answer can be an empty shell and a script tag. A person sees a full page. A crawler that doesn’t run scripts sees almost nothing.
Every page on this site is rendered to complete HTML when the site is built. Small functions turn the copy into markup, and the result is saved as a plain file. Scripts add motion, measurement and the tools you use, like the site scan. Turn JavaScript off and every word of the page is still there. Our browser tests check exactly that on every page, with scripts on and off.
The parts a machine reads first.
Some parts of a page do more work than their size suggests. The title and the description are often what a search result or an AI answer shows. The headings are the outline a machine uses to tell what each part of the page covers. Links tell a crawler where else to go.
Every page here has one main heading, a canonical address (the one address a page names as its main copy) and a description of its own. The build fails if any of them goes missing, or if two pages share a description.
You can see this reading on any page here. Switch the view to Machine, in the status bar or in the menu on a phone. The page labels itself the way a crawler takes it in, with its headings numbered as an outline and its links and structured data marked.
Structured data says who you are.
Structured data is a block of JSON-LD in the page’s HTML that names things outright. Here, it says the page is an article. It names Daniel Laskowski as the author. ADJL Technology published it on this date. A person works all that out from the layout. A machine can read it from the data and doesn’t have to guess.
Every page here carries one graph of it, with the company, its founder, the site, the page and what the page is about. The company, its founder and the site each have an ID that stays the same on every page, so a machine reading two pages knows they describe one company. The build checks that every reference in the graph points at something that exists.
robots.txt says who may read.
robots.txt is a short file at the root of a site that tells crawlers what they may fetch. AI companies publish names for their crawlers, so a site can allow or block each one by name. Some fetch pages to train models. Some fetch a page when a person asks a question that needs it. Blocking the wrong one can keep your pages out of answers you wanted to be in.
It’s a request, not a lock. The bigger AI companies say in their documentation how each of their crawlers treats it, and the answers differ.
Our robots.txt names the AI crawlers in a group of their own. Today that group allows everything, the same as the default. It’s written out so the choice is on record, and it takes one line to change it.
llms.txt is a summary, and only a proposal.
llms.txt is a newer idea. It’s a plain markdown file at the root of a site that lists the pages, each with a line about what it covers. It’s a proposed convention and not a standard. Some AI tools read it and many don’t.
This site writes one on every deploy, from the built pages, so it can’t describe a page that’s gone. Each page also has a markdown copy of its own words, linked from the page’s head, and llms-full.txt holds all of them in one file. It costs almost nothing to produce, so we do it, and we don’t claim more for it than that.
Check your own site.
You can take the same reading of your own pages. The site scan fetches a page from our server with no scripts run, the way many AI crawlers do. It reads your robots.txt and looks for a sitemap (a file listing a site’s pages for search engines) and an llms.txt. Then it lists what to fix first.
Nobody can promise you a place in an AI answer. What you can control is what a machine is handed when it asks for your page. On this site, the markdown copy is made from the built page (in its Technical wording), so it can’t drift from what the page says. The title and description are written in the same file as the words on the page.