A sitemap is a file that lists the URLs of a website to describe its architecture to search engines. Its most common format, XML, allows Googlebot and other crawlers to discover pages without relying solely on internal linking. The presence of a sitemap does not guarantee indexing, but it speeds up the consideration of content by crawlers, especially on sites with frequently changing structures.
Sitemap and AI agents: a file that only speaks to Google
Most SEO guides present the XML sitemap as a tool intended for traditional search engines. This view is incomplete. AI agents and search assistants now crawl the web to feed their responses, and their behavior towards sitemaps differs significantly from that of Googlebot.
You may also like : How to Preserve Your Cat's Kidney Health with a Limescale-Free Water Fountain
Recent tests show that some AI agents ignore the directives of the robots.txt file while still retrieving pages. The sitemap, originally designed to guide crawlers that respect protocol conventions, no longer has the same control power over all the bots that visit a site.
A well-structured sitemap remains useful for traditional engines, but it is no longer sufficient to control how content is crawled and reused. To better understand how this type of file interacts with a site’s hierarchy, you can consult zazieweb.fr for more information on practical implementation.
Read also : Anticipating the Future: How to Organize Your Funeral with Customized Planning
This reality requires thinking of the sitemap as one component among others, not as the sole key to visibility.
XML sitemap structure: tags and URL hierarchy
A sitemap.xml file relies on a limited vocabulary of tags. The root tag <urlset> contains a series of <url> blocks, each describing a page of the site. Inside each block, the <loc> tag indicates the full address of the page.
Three optional tags complete this structure:
- <lastmod> specifies the last modification date of the page, helping crawlers prioritize recently updated content rather than re-crawling everything on each visit.
- <changefreq> suggests a modification frequency (daily, weekly, monthly), but Google has confirmed that it does not consider this value to adjust the crawling rate.
- <priority> assigns a relative score between 0.0 and 1.0 to each URL, intended to indicate the importance of a page relative to others on the same site. In practice, this tag is largely ignored by engines.
In other words, only the <loc> and <lastmod> tags have a real effect on the behavior of bots. Meticulously filling out <changefreq> and <priority> for each URL is a waste of time. It is better to focus efforts on selecting the URLs included in the file.
Filtering URLs: what the sitemap should not contain
The most common mistake is generating a sitemap that lists all the URLs of the site. Noindex pages, 301 redirects, duplicate content from sorting or pagination parameters, empty archive pages: everything ends up in the default file, especially with WordPress plugins that automatically generate the sitemap.
An effective sitemap only contains pages intended for indexing. Each URL present in the file sends a signal to engines: this page deserves to be crawled. Including low-quality pages dilutes this signal and wastes crawling budget.
For several months, Google has been filtering out redundant or low-quality pages more aggressively. Coverage reports in Search Console show an increase in “Discovered – currently not indexed” statuses on WordPress sites whose sitemaps have not been cleaned up. Internal linking weighs more than the sitemap in the decision to index or not index a page. An orphaned content, even present in the sitemap, has little chance of being retained.
The operational rule is simple: if a page has no internal link pointing to it and does not attract traffic, it has no place in the sitemap.
Pages to systematically exclude
- URLs with session, sorting, or filtering parameters that create content duplicates.
- Password-protected pages or content reserved for logged-in users, which Googlebot cannot crawl.
- Pages containing a noindex meta tag, whose simultaneous presence in the sitemap creates a technical contradiction.
- Old URLs with 301 redirects: the sitemap should point to the final destination, not the original address.
Sitemap and site structure: two complementary logics
The sitemap describes a flat list of URLs. It does not express the hierarchy of sections, categories, or navigation levels. The site structure is built through internal linking, not through the sitemap.
Confusing the two leads to a common reasoning error: believing that adding a page to the sitemap is enough to integrate it into the site’s structure. Search engines evaluate the depth of a page (the number of clicks from the homepage) and the density of internal links pointing to it. These structural signals weigh much more than mere presence in an XML file.
On a site with several hundred pages, segmenting the sitemap into several thematic files (one per section or type of content) facilitates diagnosis. When the coverage report signals issues, identifying the affected section takes a few seconds instead of going through a single file of several thousand lines.
Submitting and maintaining the sitemap in Search Console
Submitting the sitemap to Google is done through Search Console, under the “Sitemaps” section. The URL of the file (usually /sitemap.xml or /sitemap_index.xml) is declared only once. Google will then revisit it according to its own cadence.
Regularly checking the coverage report after each submission allows for spotting errors: URLs returning a 404 code, pages excluded by a contradictory robots.txt rule, or content detected as duplicated. A submitted sitemap that is then forgotten loses its value over the months as pages are deleted or moved without updating the file.
Declaring the location of the sitemap in the robots.txt file (via the Sitemap: directive followed by the absolute URL) remains a good complementary practice. Robots that do not go through Search Console can thus locate the file directly.
The sitemap has never been a magic lever for indexing. It is a communication tool between a site and its automated visitors, whose usefulness entirely depends on the rigor with which it is maintained and the quality of the architecture it reflects.