Sitemap for dynamic content?

Question

I am developing a webapp using codeigniter (mvc) and php. I was wondering how do I create a sitemap for search engines when my contact is dynamic? Are there best practices for this, or is there a way to automate the process? I ask because content will be constantly and rapidly added so I'd like the sitemap be recent. Thanks

Update: to help answer my question, the type of site that I am creating is a public blog type site where users post articles.

Update2: Just to elaborate on my confusion: I may not understand exactly how a sitemap is structured. On a site like stackoverflow for example, as users add new questions (and thus, new pages get created) does the sitemap change and need to be resubmitted? How important is it for a site like this one to even have a sitemap?

Brachycephalic

Answer 1

Even though your content is dynamic, you want your sitemap to be static. Sure, update the sitemap once a day if you want, OR update it whenever you post a new blog entry... but don't try to create it at the time that it's requested. That'll be a performance nightmare.

I think Stackoverflow's sitemap is updated daily. It contains the 50,000 most recently changed pages. The sitemap helps with SEO (search engine optimization).

Recipe for sitemap generation over here

Generalship

Answer 2

I have a similar situation to yours, except I develop in Zend Framework and Doctrine instead of Code Igniter.

The way we solved this issue, was to build a Sitemap model that acts as a Doctrine Nested Set. It's basically just a few fields, like an id, parent_id, title, uri, created, updated. You can add more fields if you need, but that's the basic premise.

We then using Zend_Navigation to render this tree into a menu for the site.

Zend_Navigation also has a helper method for automatically generating an XML sitemap, so we have a controller and view that is setup to render that XML document.

Then we have written a Zend Action Controller Plugin, that monitors the sitemap controller, and when updates are made, it checks against the last update, and pings Google that the sitemap was updated.

All of these concepts can easily be ported into CI, I would imagine.

If there isn't a sitemap view helper in CI, you can probably use the Zend Navigation classes, since Zend is loosely decoupled.

Or you can just generate the xml document with CI, using the standard for sitemaps here.

http://www.sitemaps.org/protocol.php

Dru

Do you write sitemaps to a file or just cache them? To u List all urls or only most important once? How do u ping google? – Ubangishari

Hi Manuel. You can build your sitemap progressively. Start with the most important and continue to add over time. You will probably want to cache to a file. You can ping google by sending an HTTP request to www.google.com/webmasters/tools/ping?sitemap=sitemap_url and replacing sitemap_url with a url encoded version of your sitemap's URL. – Dru

Answer 3

Assuming you're trying to create an xml sitemap, not at html one?

I haven't done this yet, and want to try it out soon, but what I would do is: set up a cron job that calls a path like example.com/index.php/sitemap. the index function in your sitemap controller would then run a query to select all titles of all your posts + everything you need to construct a full url to those entries and, since its a required element in the sitemaps protocol, when the entry was last updated. if you don't have a field in your db for this you may have to add it.

then its just a matter of looping through the results and writing out an xml file with the proper structure. as you're looping you'd also have to decide the priority, changefreq and all other elements on a per entry basis.

ive also been looking at the paid version of http://www.xml-sitemaps.com/ which should do everything but setting up the cron for you, at quite a nice price and running on your own server. i dont think you could develop this yourself for 19,99USD worth of your time. And, this app also notifies the search engines of new content, a method for which i havent outlined yet above.

Ritual

Answer 4

Depending on what type of site you creating you can go about it in several ways.

For example if you were making a news site. You could have a single page that links to each news article's permalink. If you could detail what kind of site/ its content is we could help better.

Chondrite

As long as you provide a way to reach every page through site navigation in a consistent way you should be fine. Or are you want to support google sitemaps? – Chondrite

Answer 5

An update-frequency parameter in sitemap structure tells the search engine how frequent a webpage gets updated.

Don't get confused with the URL of a page because it can be changed via many methods.

For stackoverflow, it may be questions.php?id=1317539 for this page and for example, update frequency is set to one minute for questions.php

So every minute, a new sitemap is generated for stackoverflow website.

Update: This answer by Robert explains what happens to Stack Overflow's sitemap when a new question is added to the system.

Nenitanenney

Answer 6

The best solution is to add to your apache .htaccess file the following line after RewriteEngine On

RewriteRule ^sitemap\.xml$ sitemap.php [L]

and then simply having a file sitemap.php in your root folder that would be normally accessible via http://yoursite.com/sitemap.xml, the default URL where all search engines will firstly search.

The file sitemap.php shall be something like this

<?php header('Content-type: application/xml; charset=utf-8') ?>
<?php echo '<?xml version="1.0" encoding="UTF-8"?>' ?>
<?include_once('./myVariables.php'); ?>

<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc><?php echo $URL1; ?></loc>
    <lastmod>2005-01-01</lastmod>
    <changefreq>monthly</changefreq>
  </url>
  <url>
    <loc><?php echo $URL2; ?></loc>
    <lastmod>2005-01-01</lastmod>
    <changefreq>monthly</changefreq>
  </url>
</urlset> 

it works :)

Numeral