Forum Downtime Topic is solved

The Forum Rules, general FAQ and questions or comments about the forum are posted here.
Post Reply
Jorg
Moderator
Moderator
Posts: 3971
Joined: Fri Aug 27, 2004 9:38 pm
Location: Delft, Netherlands
Contact:

Forum Downtime

Post by Jorg » Tue Jun 03, 2008 6:10 am

Hi All,

My apologies for the downtime of the forum. It appeared there was a big coincidence that the downtime of the forum happened on the same period as an explosion at a server data store. I am not 100% sure if this is where Jason keeps his servers, but it is very likely the downtime for days was related to this event. More info about it:

http://www.allheadlinenews.com/articles/7011131199

With regards,
- Jorgen
Forensic Software Engineer
Netherlands Forensic Insitute
http://english.forensischinstituut.nl/
-------------------------------------
Jorg's WasteBucket
http://www.xs4all.nl/~jorgb/wb

Sof_T
Can't get richer than this
Can't get richer than this
Posts: 864
Joined: Thu Jul 28, 2005 9:48 pm
Location: New Forest, United Kingdom
Contact:

Post by Sof_T » Wed Jun 04, 2008 6:37 am

We are just glad to see you back :D
The home of Sof.T http://www.sof-t.site88.net/
Author of Programming with wxDevC++
http://sourceforge.net/projects/wxdevcpp-book/

Jorg
Moderator
Moderator
Posts: 3971
Joined: Fri Aug 27, 2004 9:38 pm
Location: Delft, Netherlands
Contact:

Post by Jorg » Wed Jun 04, 2008 7:57 am

As I was able to browse some of the forum yesterday morning, in the afternoon it was gone again. It seems there is some kind of ripple effect. Let's hope this was the last downtime. I'm glad to be back it is kind of your daily habit to check posts and reply to questions ;-)

- Jorgen
Forensic Software Engineer
Netherlands Forensic Insitute
http://english.forensischinstituut.nl/
-------------------------------------
Jorg's WasteBucket
http://www.xs4all.nl/~jorgb/wb

User avatar
tierra
Site Admin
Site Admin
Posts: 1340
Joined: Sun Aug 29, 2004 7:14 pm
Location: Salt Lake City, Utah, USA
Contact:

Post by tierra » Wed Jun 04, 2008 9:38 am

Quick Update:

Based on the behavior of the downtime on the forums (and the internet route to them), I think I can safely say that the forums are hosted at The Planet's Houston data center (H1) in Phase 1 (the first floor). The first level, containing 3,000 servers, is where the electrical equipment room that caught fire is located. They were able to bring the second floor back online with their own onsite generator within about 32 hours of the explosion powered, but the temporary generator delivered to power the first floor failed after a few hours of service (I think this is the outage that Jorg saw) and a second generator was needed to bring power back up again.

If the forums are in fact on a server in Phase 1 and not in Phase 2, we will see approximately 4-6 hours of downtime either on the 7th or the 14th at midnight (CDT I think) when they switch power back to the replacement battery backup system. We won't know until Jason can get back to us (he may not know himself if Idaho Web Hoster hasn't said). It's still possible that the second outage was caused by other DNS or network issues which H1 has been resolving while bringing up power to each floor, in which case, it's still possible that the forums are in Phase 2 which won't experience that 4-6 hours of downtime.

Jorg
Moderator
Moderator
Posts: 3971
Joined: Fri Aug 27, 2004 9:38 pm
Location: Delft, Netherlands
Contact:

Post by Jorg » Wed Jun 04, 2008 10:16 am

Hi Tierra,

Thanks a lot for the explanation. I kind of thought there was some problem again, but you always hope it is over soon. I don't think the forum itself had any damage by the downtime as it seems to run fine again, and people are happily posting away like nothing happened.

Jason did not respond I think he is really busy helping his paid customers to get their sites in the air, or he's not working at the moment. In any case, I'm glad it seenms to be resolved. Let's hope the generators hold until everything is a-ok there again.

- Jorgen
Forensic Software Engineer
Netherlands Forensic Insitute
http://english.forensischinstituut.nl/
-------------------------------------
Jorg's WasteBucket
http://www.xs4all.nl/~jorgb/wb

jsheets
Earned a small fee
Earned a small fee
Posts: 22
Joined: Wed Oct 19, 2005 1:21 pm

Post by jsheets » Wed Jun 04, 2008 6:18 pm

Good morning, yes the server is hosted at The Planet and unfortunately was affected by their outage.

They were able to bring the servers back up on generator power but unfortunately the generator failed, leading to the additional outage.

At this time they are still running on generator power so at some point down the road there will be another, hopefully brief, period of unavailability when they switch back to main power.

Jorg
Moderator
Moderator
Posts: 3971
Joined: Fri Aug 27, 2004 9:38 pm
Location: Delft, Netherlands
Contact:

Post by Jorg » Wed Jun 04, 2008 8:51 pm

Hi Jason,

Thanks for the explanation and ofcourse we're ever thankful for your hosting efforts for the wxForum. If we have outage now and then (even when a freak accident like this occurs) we'll patiently wait until it is over ;-)

Thanks,
- Jorgen
Forensic Software Engineer
Netherlands Forensic Insitute
http://english.forensischinstituut.nl/
-------------------------------------
Jorg's WasteBucket
http://www.xs4all.nl/~jorgb/wb

jsheets
Earned a small fee
Earned a small fee
Posts: 22
Joined: Wed Oct 19, 2005 1:21 pm

Post by jsheets » Fri Jun 06, 2008 6:38 pm

Hello all, The Planet is continuing to have problems and honestly is killing us. We are in the process of moving the remaining server away from them.

Over the next few days I will be coordinating with Jorg and other site admin to migrate wxforum to another server.

The other server should be somewhat faster as it has more modern hardware including dual core processor and more memory.

User avatar
doublemax
Moderator
Moderator
Posts: 14176
Joined: Fri Apr 21, 2006 8:03 pm
Location: $FCE2

Post by doublemax » Thu Jul 17, 2008 1:23 pm

i'm experiencing downtimes pretty often lately. Are you currently in the progress of moving it to another server? If not, are you still planing to do so?
Use the source, Luke!

upCASE
Site Admin
Site Admin
Posts: 3176
Joined: Mon Aug 30, 2004 6:55 am
Location: Germany, Cologne

Post by upCASE » Thu Jul 17, 2008 1:27 pm

doublemax wrote:i'm experiencing downtimes pretty often lately. Are you currently in the progress of moving it to another server? If not, are you still planing to do so?
Yep, same for me.
No, we're not moving, unless Jorg decided to do so :)

Jorg: I didn't have time to mail you, but did you contact Jason about the downtimes?
OS: OpenSuSE, Ubuntu, Win XP Pro
wx: svn
Compiler: gcc 4.5.1, VC 2008, eVC 4

"If it was hard to write it should be hard to read..." - the unknown coder
"Try not! Do. Or do not. There is no try." - Yoda

jsheets
Earned a small fee
Earned a small fee
Posts: 22
Joined: Wed Oct 19, 2005 1:21 pm

Post by jsheets » Thu Jul 17, 2008 6:54 pm

Hello all, I will be moving wxForum to a new server this evening. I have decreased DNS TTL, and will also be setting up an additional alias for those that wind up with cached DNS.

User avatar
doublemax
Moderator
Moderator
Posts: 14176
Joined: Fri Apr 21, 2006 8:03 pm
Location: $FCE2

Post by doublemax » Thu Jul 17, 2008 7:35 pm

that's good news - much appreciated =D>
Use the source, Luke!

Jorg
Moderator
Moderator
Posts: 3971
Joined: Fri Aug 27, 2004 9:38 pm
Location: Delft, Netherlands
Contact:

Post by Jorg » Thu Jul 17, 2008 9:38 pm

Hi Jason,

Thank you very much. I am sorry I saw your message too late I have been sick and a bit tired from sleeping bad (from geing sick) so I wanted to contact you about steps to take, but it seems you beat me to the punch.

Thanks again for your great efforts to host this forum on your domain!

With regards,
- Jorgen
Forensic Software Engineer
Netherlands Forensic Insitute
http://english.forensischinstituut.nl/
-------------------------------------
Jorg's WasteBucket
http://www.xs4all.nl/~jorgb/wb

jsheets
Earned a small fee
Earned a small fee
Posts: 22
Joined: Wed Oct 19, 2005 1:21 pm

wxForum Move Complete

Post by jsheets » Fri Jul 18, 2008 2:50 am

I've moved the site to the new server and everything appears to be well.

Some users may still have cached DNS entries but they should timeout soon.

The site should may even be somewhat faster as this server is considerably better.

Post Reply