Why use a Google Sitemap?

Question

I've played around with Google Sitemaps on a couple sites. The lastmod, changefreq, and priority parameters are pretty cool in theory. But in practice I haven't seen these parameters affect much.

And most of my sites don't have a Google Sitemap and that has worked out fine. Google still crawls the site and finds all of my pages. The old meta robot and robots.txt mechanisms still work when you don't want a page (or directory) to be indexed. And I just leave every other page alone and as long as there's a link to it Google will find it.

So what reasons have you found to write a Google Sitemap? Is it worth it?

Pharsalus

I voted to close this question because it is not a programming question and it is off-topic on Stack Overflow. Non-programming questions about your website should be asked on Webmasters. In the future, please ask questions like this there. – Darya

The webmasters site has a question from the founder of Stack Overflow who questions the utility of sitemaps. It is answered by a representative from Google who lays out their utility (which is fairly limited): The Sitemap Paradox – Darya

Answer 1

From the FAQ:

Sitemaps are particularly helpful if:

• Your site has dynamic content.
• Your site has pages that aren't easily discovered by Googlebot during the crawl process—for example, pages featuring rich AJAX or images.
• Your site is new and has few links to it. (Googlebot crawls the web by following links from one page to another, so if your site isn't well linked, it may be hard for us to discover it.)
• Your site has a large archive of content pages that are not well linked to each other, or are not linked at all.

It also allows you to provide more granular information to Google about the relative importance of pages in your site and how often the spider should come back. And, as mentioned above, if Google deems your site important enough to show sublinks under in the search results, you can control what appears via sitemap.

Schick

After further research I do believe that the "dynamic content" consideration is key. The ability to tell Google when a page is updated via the lastmod parameter is perhaps the most useful part of an XML Sitemap. I still think it's questionable whether it's worth building and maintaining one of these sitemaps (depending on the nature of your site of course). But I think this information from the FAQ is probably the best we'll get. – Pharsalus

Except that Google rarely uses the last modified field from sitemaps because very few sites keep it updated appropriately. See webmasters.stackexchange.com/a/117099 – Darya

Answer 2

I am also curious about the topic, but does it cost anything to generate a sitemap?

In theory, anything that costs nothing and may have a potential gain, even if very small or very remote, can be defined as "worth it".

In addition, Google says: "Tell us about your pages with Sitemaps: which ones are the most important to you and how often they change. You can also let us know how you would like the URLs we index to appear." (Webmaster Tools)

I don't think that the bold statement above is possible with the traditional mechanisms that search engines use to discover URLs.

Eschatology

Yes, it's free. However costs may involve effort, development time, server resources to generate it, etc. :) – Eschatology

"You can also let us know how you would like the URLs we index to appear." - Do you have any idea what that means? Are they talking about problems like including www or not in the URL? Because there's a separate setting in Webmaster Tools for that. – Pharsalus

Answer 3

I believe the "special links" in search results are generated from the google sitemap. What do I mean by "special link"? Search for "apache", below the first result (Apache software foundation) there are two columns of links ("Apache Server", "Tomcat", "FAQ").

Oloughlin

Google calls those "Sitelinks." Here's their explanation in Webmaster Tools: "Sitelinks are completely automated, and we show them only if we think they'll be useful to the user. If your site's structure doesn't allow our algorithms to find good sitelinks, or we don't think that the sitelinks are relevant to the user's query, we won't show them. However, we are always working to improve how we find and display sitelinks." They don't mention Sitemaps in there so I'm led to believe they don't affect this. – Pharsalus

I can confirm that the site I know using the sitemaps have those links too. So it may be playing a big role in that. – Murrumbidgee

@Maxwell - I may have to investigate that theory. If a Sitemap increases the chances of getting good Sitelinks then that could be worth it right there. – Pharsalus

I found a couple sources now that suggest that Sitemaps and Sitelinks are not related: microsystools.com/products/sitemap-generator/faq/… and seopedia.org/internet-marketing-and-seo/… But in any case I just submitted a simple sitemap for one of my more popular sites. If I get Sitelinks within a few weeks then I'm a believer. – Pharsalus

Ah yes, I almost forgot... one of the sites I'm responsible for (Techinsurance.com) has no Sitemaps file, but it does have Sitelinks. – Pharsalus

Answer 4

I guess it helps Google to prioritize their crawl? But in practice I was involved in a project where we used the gzip-ed large version of it where it helped massively. And AFAIK there is a nice integration with webmaster tools as well.

Murrumbidgee