How fast does Google take to crawl new page, and can we influence Google's crawler?

Asked 26/1, 2012 at 3:40 Answered 30/6, 2021 at 6:51

I want to submit my site to Google. How much time does it take to crawl a new post on the website?

Also, is there a way to feed this post to Google crawler as soon as a post is created?

Penninite answered 26/1, 2012 at 3:40 Comment(3)

-1: I can't figure out what you are doing. Are you talking about google's crawler on SO, or are you talking about google's crawler on your site, or are you talking about telling google's crawler how often to re-index your site? – Robbinrobbins 26/1, 2012 at 3:47

its a general question about google crawler. be it stackoverflow or any website that the bot crawls – Penninite 26/1, 2012 at 4:4

This needs to be on the webmasters site. – Flutter 27/1, 2012 at 0:8

Google has three modes of entering a website into its results - discover, crawl, index.

In order to 'discover' your site, it must be made aware of it's existence - normally through back-links. If you're site is brand new you can use the submit URL form - but this isn't really a trusted method. You're better off signing up for a Google Webmaster Tools account and submitting your site. An additional step is to submit an XML sitemap of your site. If you are publishing to your site in a blogging/posting way - you can always consider PubSubHubbub.

From there on, crawl frequency is normally based on site popularity (as measured by ye olde PageRank). Depth of crawl (crawl-budget) is also determined by PR.

Vanlandingham answered 26/1, 2012 at 22:32 Comment(3)

marked as answer as google itself suggest the XML site map submission – Penninite 10/2, 2012 at 3:2

how much is limit to upload site map? is that based on the upload file size or the number of entries in a file? – Penninite 19/3, 2012 at 8:11

+1 For the point re: crawl times and PR - You'll note that SO Questions appear almost instantly, whereas less popular sites can take days/weeks for new content to appear. – Lecherous 6/8, 2012 at 12:18

There are a couple ways to help "feed" the Google Crawler a URL.

The first way is to go here and submit a URL ---> www.google.com/webmasters/tools/submit-url/

The second way is to go to your Google Webmasters Tools and clicking "Fetch as GoogleBot"

And then inputting the URL you want to add:

https://i.sstatic.net/Q3Iva.png

The URL will then appear similar to this:

http:\\example.site Web Success URL submitted to index 1/22/12 2:51 AM

As for how long it takes for a question on here to appear on google, there are many factors that are put in to this.

If the owners of the site use Google Webmasters Tools, the following setting is available:

https://i.sstatic.net/RqvOi.png

Dimorphism answered 26/1, 2012 at 3:58 Comment(2)

Can't agree with Revisit-After - see Google's own comments here - code.google.com/webstats/2005-12/metadata.html, and more here: googlewebmastercentral.blogspot.com/2007/12/… – Vanlandingham 26/1, 2012 at 22:22

Mike, thank for point out resources for Revisit-after. Wow, I didn't know this was not being used. I also liked your second link so giving you a thumbs up. :-) – Stinko 28/1, 2012 at 16:50

For fast crawl you should submit your xml sitemap in google web master and manually crawled and index your web pages url through google webmaster fetch.
I also used google crawled and index method and after that this practices give me best result.

Gerontocracy answered 10/4, 2014 at 7:56 Comment(0)

This is a great resource that really breaks down all the factors that affect a crawl budget and how to optimize your website to increase it. Cleaning up your broken links and removing outdated content, for example, can work wonders. https://prerender.io/crawl-budget-seo/

Sustainer answered 30/6, 2021 at 6:51 Comment(0)

-1

I acknowledged error in my response by adding a comment to original question a long time ago. Now, I am updating this post in interest of keeping future readers from being misguided as I was. Please see notes from other users below - they are correct. Google does not make use of the revisit-after meta tag. I am still keeping the original response text here to make sure that anyone else looking for similar answer will find it here along with this note confirming that this meta tag IS NOT VALID! Hope this helps someone.

You may use HTML meta tag as follows:

<meta name="revisit-after" content="1 day">

Adjust time period as necessary. There is no guarantee that robots will return in given time frame but this is how you are telling robots about how often a given page is likely to change.

The Revisit Meta Tag is used to tell search engines when to come back next.

Stinko answered 26/1, 2012 at 3:54 Comment(3)

Ok, I think you are looking for Google Custom Search integration into your site and controlling frequency at Google will index your site. I think better description (use at least the product/service name as provider uses) would have helped. You will find answers you are looking for here .... Updating pages for on-demand indexing - link – Stinko 26/1, 2012 at 4:4

Disagree with this, see other comment. – Vanlandingham 26/1, 2012 at 22:25

Google does not utilize this tag, see googlewebmastercentral.blogspot.com/2007/12/… – Purposeful 3/8, 2012 at 13:38

Recommended topics

Hot tags