Forum Downtime Topic is solved

The Forum Rules, general FAQ and questions or comments about the forum are posted here.
Post Reply
Jorg
Moderator
Moderator
Posts: 3971
Joined: Fri Aug 27, 2004 9:38 pm
Location: Delft, Netherlands
Contact:

Forum Downtime

Post by Jorg »

Hi All,

My apologies for the downtime of the forum. It appeared there was a big coincidence that the downtime of the forum happened on the same period as an explosion at a server data store. I am not 100% sure if this is where Jason keeps his servers, but it is very likely the downtime for days was related to this event. More info about it:

http://www.allheadlinenews.com/articles/7011131199

With regards,
- Jorgen
Forensic Software Engineer
Netherlands Forensic Insitute
http://english.forensischinstituut.nl/
-------------------------------------
Jorg's WasteBucket
http://www.xs4all.nl/~jorgb/wb
Sof_T
Can't get richer than this
Can't get richer than this
Posts: 864
Joined: Thu Jul 28, 2005 9:48 pm
Location: New Forest, United Kingdom
Contact:

Post by Sof_T »

We are just glad to see you back :D
The home of Sof.T http://www.sof-t.site88.net/
Author of Programming with wxDevC++
http://sourceforge.net/projects/wxdevcpp-book/
Jorg
Moderator
Moderator
Posts: 3971
Joined: Fri Aug 27, 2004 9:38 pm
Location: Delft, Netherlands
Contact:

Post by Jorg »

As I was able to browse some of the forum yesterday morning, in the afternoon it was gone again. It seems there is some kind of ripple effect. Let's hope this was the last downtime. I'm glad to be back it is kind of your daily habit to check posts and reply to questions ;-)

- Jorgen
Forensic Software Engineer
Netherlands Forensic Insitute
http://english.forensischinstituut.nl/
-------------------------------------
Jorg's WasteBucket
http://www.xs4all.nl/~jorgb/wb
User avatar
tierra
Site Admin
Site Admin
Posts: 1355
Joined: Sun Aug 29, 2004 7:14 pm
Location: Salt Lake City, Utah, USA
Contact:

Post by tierra »

Quick Update:

Based on the behavior of the downtime on the forums (and the internet route to them), I think I can safely say that the forums are hosted at The Planet's Houston data center (H1) in Phase 1 (the first floor). The first level, containing 3,000 servers, is where the electrical equipment room that caught fire is located. They were able to bring the second floor back online with their own onsite generator within about 32 hours of the explosion powered, but the temporary generator delivered to power the first floor failed after a few hours of service (I think this is the outage that Jorg saw) and a second generator was needed to bring power back up again.

If the forums are in fact on a server in Phase 1 and not in Phase 2, we will see approximately 4-6 hours of downtime either on the 7th or the 14th at midnight (CDT I think) when they switch power back to the replacement battery backup system. We won't know until Jason can get back to us (he may not know himself if Idaho Web Hoster hasn't said). It's still possible that the second outage was caused by other DNS or network issues which H1 has been resolving while bringing up power to each floor, in which case, it's still possible that the forums are in Phase 2 which won't experience that 4-6 hours of downtime.
Jorg
Moderator
Moderator
Posts: 3971
Joined: Fri Aug 27, 2004 9:38 pm
Location: Delft, Netherlands
Contact:

Post by Jorg »

Hi Tierra,

Thanks a lot for the explanation. I kind of thought there was some problem again, but you always hope it is over soon. I don't think the forum itself had any damage by the downtime as it seems to run fine again, and people are happily posting away like nothing happened.

Jason did not respond I think he is really busy helping his paid customers to get their sites in the air, or he's not working at the moment. In any case, I'm glad it seenms to be resolved. Let's hope the generators hold until everything is a-ok there again.

- Jorgen
Forensic Software Engineer
Netherlands Forensic Insitute
http://english.forensischinstituut.nl/
-------------------------------------
Jorg's WasteBucket
http://www.xs4all.nl/~jorgb/wb
jsheets
Earned a small fee
Earned a small fee
Posts: 22
Joined: Wed Oct 19, 2005 1:21 pm

Post by jsheets »

Good morning, yes the server is hosted at The Planet and unfortunately was affected by their outage.

They were able to bring the servers back up on generator power but unfortunately the generator failed, leading to the additional outage.

At this time they are still running on generator power so at some point down the road there will be another, hopefully brief, period of unavailability when they switch back to main power.
Jorg
Moderator
Moderator
Posts: 3971
Joined: Fri Aug 27, 2004 9:38 pm
Location: Delft, Netherlands
Contact:

Post by Jorg »

Hi Jason,

Thanks for the explanation and ofcourse we're ever thankful for your hosting efforts for the wxForum. If we have outage now and then (even when a freak accident like this occurs) we'll patiently wait until it is over ;-)

Thanks,
- Jorgen
Forensic Software Engineer
Netherlands Forensic Insitute
http://english.forensischinstituut.nl/
-------------------------------------
Jorg's WasteBucket
http://www.xs4all.nl/~jorgb/wb
jsheets
Earned a small fee
Earned a small fee
Posts: 22
Joined: Wed Oct 19, 2005 1:21 pm

Post by jsheets »

Hello all, The Planet is continuing to have problems and honestly is killing us. We are in the process of moving the remaining server away from them.

Over the next few days I will be coordinating with Jorg and other site admin to migrate wxforum to another server.

The other server should be somewhat faster as it has more modern hardware including dual core processor and more memory.
User avatar
doublemax
Moderator
Moderator
Posts: 19114
Joined: Fri Apr 21, 2006 8:03 pm
Location: $FCE2

Post by doublemax »

i'm experiencing downtimes pretty often lately. Are you currently in the progress of moving it to another server? If not, are you still planing to do so?
Use the source, Luke!
upCASE
Moderator
Moderator
Posts: 3176
Joined: Mon Aug 30, 2004 6:55 am
Location: Germany, Cologne

Post by upCASE »

doublemax wrote:i'm experiencing downtimes pretty often lately. Are you currently in the progress of moving it to another server? If not, are you still planing to do so?
Yep, same for me.
No, we're not moving, unless Jorg decided to do so :)

Jorg: I didn't have time to mail you, but did you contact Jason about the downtimes?
OS: OpenSuSE, Ubuntu, Win XP Pro
wx: svn
Compiler: gcc 4.5.1, VC 2008, eVC 4

"If it was hard to write it should be hard to read..." - the unknown coder
"Try not! Do. Or do not. There is no try." - Yoda
jsheets
Earned a small fee
Earned a small fee
Posts: 22
Joined: Wed Oct 19, 2005 1:21 pm

Post by jsheets »

Hello all, I will be moving wxForum to a new server this evening. I have decreased DNS TTL, and will also be setting up an additional alias for those that wind up with cached DNS.
User avatar
doublemax
Moderator
Moderator
Posts: 19114
Joined: Fri Apr 21, 2006 8:03 pm
Location: $FCE2

Post by doublemax »

that's good news - much appreciated =D>
Use the source, Luke!
Jorg
Moderator
Moderator
Posts: 3971
Joined: Fri Aug 27, 2004 9:38 pm
Location: Delft, Netherlands
Contact:

Post by Jorg »

Hi Jason,

Thank you very much. I am sorry I saw your message too late I have been sick and a bit tired from sleeping bad (from geing sick) so I wanted to contact you about steps to take, but it seems you beat me to the punch.

Thanks again for your great efforts to host this forum on your domain!

With regards,
- Jorgen
Forensic Software Engineer
Netherlands Forensic Insitute
http://english.forensischinstituut.nl/
-------------------------------------
Jorg's WasteBucket
http://www.xs4all.nl/~jorgb/wb
jsheets
Earned a small fee
Earned a small fee
Posts: 22
Joined: Wed Oct 19, 2005 1:21 pm

wxForum Move Complete

Post by jsheets »

I've moved the site to the new server and everything appears to be well.

Some users may still have cached DNS entries but they should timeout soon.

The site should may even be somewhat faster as this server is considerably better.
Post Reply