Page 1 of 1

Allow google and other robots to index this forum?

Posted: Tue Jan 12, 2010 12:25 pm
by thile
As far as I know, robots are not allowed to index the wxforum. Why is this? I would see it as a good feature, as I would much rather use Google to search the forum.

Posted: Tue Jan 12, 2010 1:09 pm
by Jorg
Hi,

This is because googlebot slows down this forum a lot. Whenever googlebot passes it and indexes the pages, it starts to get sluggish and unresponsive. We have no real influence over the hardware of this forum, so we disabled it ..

With regards,
- Jorgen

Posted: Thu Feb 11, 2010 10:20 am
by evstevemd
Have you tested other search engines to see if they slow down?
This forum is full of nuggets but it is diffu=icult to get them via normal search

What about yahoo or metacrawler?

Posted: Thu Feb 11, 2010 12:04 pm
by DavidHart
Hi,
This is because googlebot slows down this forum a lot. Whenever googlebot passes it and indexes the pages, it starts to get sluggish and unresponsive.
What would happen if you allowed indexing for a day (or until it happened once) and then switched it off again? That would get the archive into google; and you could repeat every month or two to add new posts.

Or does google 'forget' about pages that it's no longer allowed to trawl?

Regards,

David

Posted: Thu Feb 11, 2010 2:59 pm
by Jorg
I have not a clue. I would think that google will keep some kind of retention in it's cache, but it is also plausible to assume that when a site does not want to get indexed, that google will flush it from cache. As example, if a book site does not want google to find the books and add a meta tag not to index and google already did it, people would still find the old(er) results in the cache..

So honestly I do not know if that would help..

With regards,
- Jorgen