Allow google and other robots to index this forum? Topic is solved
Allow google and other robots to index this forum?
As far as I know, robots are not allowed to index the wxforum. Why is this? I would see it as a good feature, as I would much rather use Google to search the forum.
Hi,
This is because googlebot slows down this forum a lot. Whenever googlebot passes it and indexes the pages, it starts to get sluggish and unresponsive. We have no real influence over the hardware of this forum, so we disabled it ..
With regards,
- Jorgen
This is because googlebot slows down this forum a lot. Whenever googlebot passes it and indexes the pages, it starts to get sluggish and unresponsive. We have no real influence over the hardware of this forum, so we disabled it ..
With regards,
- Jorgen
Forensic Software Engineer
Netherlands Forensic Insitute
http://english.forensischinstituut.nl/
-------------------------------------
Jorg's WasteBucket
http://www.xs4all.nl/~jorgb/wb
Netherlands Forensic Insitute
http://english.forensischinstituut.nl/
-------------------------------------
Jorg's WasteBucket
http://www.xs4all.nl/~jorgb/wb
- evstevemd
- Part Of The Furniture
- Posts: 2409
- Joined: Wed Jan 28, 2009 11:57 am
- Location: United Republic of Tanzania
Have you tested other search engines to see if they slow down?
This forum is full of nuggets but it is diffu=icult to get them via normal search
What about yahoo or metacrawler?
This forum is full of nuggets but it is diffu=icult to get them via normal search
What about yahoo or metacrawler?
Chief Justice: We have trouble dear citizens!
Citizens: What it is his honor?
Chief Justice:Our president is an atheist, who will he swear to?
Citizens: What it is his honor?
Chief Justice:Our president is an atheist, who will he swear to?
Hi,
Or does google 'forget' about pages that it's no longer allowed to trawl?
Regards,
David
What would happen if you allowed indexing for a day (or until it happened once) and then switched it off again? That would get the archive into google; and you could repeat every month or two to add new posts.This is because googlebot slows down this forum a lot. Whenever googlebot passes it and indexes the pages, it starts to get sluggish and unresponsive.
Or does google 'forget' about pages that it's no longer allowed to trawl?
Regards,
David
I have not a clue. I would think that google will keep some kind of retention in it's cache, but it is also plausible to assume that when a site does not want to get indexed, that google will flush it from cache. As example, if a book site does not want google to find the books and add a meta tag not to index and google already did it, people would still find the old(er) results in the cache..
So honestly I do not know if that would help..
With regards,
- Jorgen
So honestly I do not know if that would help..
With regards,
- Jorgen
Forensic Software Engineer
Netherlands Forensic Insitute
http://english.forensischinstituut.nl/
-------------------------------------
Jorg's WasteBucket
http://www.xs4all.nl/~jorgb/wb
Netherlands Forensic Insitute
http://english.forensischinstituut.nl/
-------------------------------------
Jorg's WasteBucket
http://www.xs4all.nl/~jorgb/wb