google-crawlers Questions
6
Solved
From the HTTP server's perspective.
Tartu asked 22/7, 2010 at 12:6
6
My problem is Crawl in Google Search Console can't found sub-routes in React.
The URL is https://huynhsamha.github.io/crypto, and crawler can fetch and render homepage (route /) and static files s...
Melentha asked 28/12, 2018 at 8:9
4
Solved
I would like to get the same result as this command line :
scrapy crawl linkedin_anonymous -a first=James -a last=Bond -o output.json
My script is as follows :
import scrapy
from linkedin_anonymo...
Massage asked 20/12, 2015 at 15:6
3
I am trying to tell Google and other search engines not to crawl some parts of my web page.
What I do is:
<!--googleoff: all-->
<select name="ddlCountry" id="ddlCountry">
<option ...
Elayneelazaro asked 14/1, 2013 at 6:21
1
Solved
I have made my Github repo public a week ago but it is still not visible in google search even if I search it like site:https://github.com/user/reponame. Answers for similar questions on Stackoverf...
Inexplicable asked 24/4, 2019 at 9:59
2
I'm having an issue with getting Googlebot to correctly render my webpage(s).
It's rendering the header and one "row" of my page (just the page's top background picture), and then failing to rende...
Littleton asked 29/6, 2016 at 15:13
1
Solved
I am new to python and using unofficial pytrends API to crawl Google Trend. I have 2000+ keywords as DNA list and try to crawl data. When I run this code, it appears with "Google returned a respons...
Jackhammer asked 27/11, 2017 at 2:50
1
Solved
I have a deployed angular 2 app working nicely in Production. The issue is the web crawlers are not actually able to crawl and index the whole site, I only see the main index page/route being...
Moreau asked 24/8, 2017 at 11:2
1
Solved
Can someone explain me how should i write a robots.txt file if i want that all crawlers index root and some specific subdomains
User-agent: *
Allow: /
Allow: /subdomain1/
Allow: /subdomain2/
Is ...
Whale asked 10/4, 2017 at 10:1
4
Solved
We can tell bots to crawl or not to crawl our website in robot.txt. On the other hand, we can control the crawling speed in Google Webmasters (how much Google bot crawls the website). I wonder if i...
Unfair asked 16/10, 2011 at 20:56
2
I've built some widget for websites which is asynchronously loaded after the page is loaded:
<html>
<head>...</head>
<body>
<div>...</div>
...
Outage asked 3/10, 2011 at 19:58
2
Solved
I created a php page that is only accessible by means of token/pass received through $_GET
Therefore if you go to the following url you'll get a generic or blank page
http://fakepage11.com/secret_...
Busboy asked 19/2, 2016 at 11:30
2
Solved
I'm reading this specification which is an agreement between web servers and search engine crawlers that allows for dynamically created content to be visible to crawlers.
It's stated there that in ...
Loathsome asked 15/7, 2015 at 18:3
2
Solved
I have recently created a webpage using Angularjs and I'm currently trying to get it indexed by Google using pushstate.
I've done quite abit of research and found out that I can use Googlebot-sim...
Extravasation asked 18/9, 2015 at 11:31
2
Solved
I have been trying to get my AngularJS and Parse web app crawlable for Google and Facebook share and even with prerender-parse I have not been able to get it working. I have tried using tips from t...
Vaporize asked 15/3, 2015 at 0:31
3
Should I add PDFs to my XML sitemap?
I want to know if Google will crawl the PDFs.
Inclined asked 2/7, 2009 at 6:56
1
Google webmaster guide explains that web server should handle requests for url that contains _escaped_fragment_ (The crawler modifies www.example.com/ajax.html#!mystate to www.example.com/ajax.html...
Refuse asked 9/10, 2012 at 12:38
2
Solved
Google says about this meta tag:
The following important restrictions apply:
The meta tag may only appear in pages without hash fragments.
Only "!" may appear in the content field.
The me...
Singband asked 18/6, 2013 at 20:38
3
Solved
I have been working with some advanced javascript applications using a lot of ajax requests to render my page. To make the applications crawlable (by google), I have to follow https://develop...
Valli asked 10/10, 2013 at 5:5
2
Solved
I am creating a new web crawler using C# to crawl some specific websites. every thing goes fine. but the problem is that some websites are blocking my crawler IP address after some requests. I trie...
Unsound asked 4/10, 2011 at 6:28
4
Solved
When googlebot crawls pages does it have session? For example I am storing some variables on the session and using them in my site's pages. When googlebot crawls these pages will I still have the s...
Annadiane asked 31/10, 2011 at 19:54
4
Solved
The contents of my robots.txt file are actually itself indexed and show up in Google search results. It's only Google and not Yahoo for example.
I really think Google should understand not to...
Debbradebby asked 8/11, 2011 at 10:13
1
Solved
Im writing a review site where the community rates posts. I have noticed that Google can pick up on this ratings and display them in its search results. Does anyone know how this is achieved?
An e...
Enhanced asked 21/8, 2011 at 14:33
1
© 2022 - 2024 — McMap. All rights reserved.