Allow google and other robots to index this forum? Topic is solved

The Forum Rules, general FAQ and questions or comments about the forum are posted here.
Post Reply
thile
Knows some wx things
Knows some wx things
Posts: 26
Joined: Sat Oct 25, 2008 3:46 pm

Allow google and other robots to index this forum?

Post by thile » Tue Jan 12, 2010 12:25 pm

As far as I know, robots are not allowed to index the wxforum. Why is this? I would see it as a good feature, as I would much rather use Google to search the forum.

Jorg
Moderator
Moderator
Posts: 3971
Joined: Fri Aug 27, 2004 9:38 pm
Location: Delft, Netherlands
Contact:

Post by Jorg » Tue Jan 12, 2010 1:09 pm

Hi,

This is because googlebot slows down this forum a lot. Whenever googlebot passes it and indexes the pages, it starts to get sluggish and unresponsive. We have no real influence over the hardware of this forum, so we disabled it ..

With regards,
- Jorgen
Forensic Software Engineer
Netherlands Forensic Insitute
http://english.forensischinstituut.nl/
-------------------------------------
Jorg's WasteBucket
http://www.xs4all.nl/~jorgb/wb

User avatar
evstevemd
Part Of The Furniture
Part Of The Furniture
Posts: 2248
Joined: Wed Jan 28, 2009 11:57 am
Location: United Republic of Tanzania
Contact:

Post by evstevemd » Thu Feb 11, 2010 10:20 am

Have you tested other search engines to see if they slow down?
This forum is full of nuggets but it is diffu=icult to get them via normal search

What about yahoo or metacrawler?
Chief Justice: We have trouble dear citizens!
Citizens: What it is his honor?
Chief Justice:Our president is an atheist, who will he swear to?
[Ubuntu 15.04/Windows 10 Pro - GCC/MinGW, CodeLite IDE et al]

DavidHart
Site Admin
Site Admin
Posts: 3881
Joined: Thu Jan 12, 2006 6:23 pm
Location: IoW, UK

Post by DavidHart » Thu Feb 11, 2010 12:04 pm

Hi,
This is because googlebot slows down this forum a lot. Whenever googlebot passes it and indexes the pages, it starts to get sluggish and unresponsive.
What would happen if you allowed indexing for a day (or until it happened once) and then switched it off again? That would get the archive into google; and you could repeat every month or two to add new posts.

Or does google 'forget' about pages that it's no longer allowed to trawl?

Regards,

David

Jorg
Moderator
Moderator
Posts: 3971
Joined: Fri Aug 27, 2004 9:38 pm
Location: Delft, Netherlands
Contact:

Post by Jorg » Thu Feb 11, 2010 2:59 pm

I have not a clue. I would think that google will keep some kind of retention in it's cache, but it is also plausible to assume that when a site does not want to get indexed, that google will flush it from cache. As example, if a book site does not want google to find the books and add a meta tag not to index and google already did it, people would still find the old(er) results in the cache..

So honestly I do not know if that would help..

With regards,
- Jorgen
Forensic Software Engineer
Netherlands Forensic Insitute
http://english.forensischinstituut.nl/
-------------------------------------
Jorg's WasteBucket
http://www.xs4all.nl/~jorgb/wb

Post Reply