The Day rhettandlink.com Stood Still

In the night of 2014-03-15, as in from midnight until this writing, rhetandlink.com went down.

I've done some checking with nmap. and this is what I've got:

As you can see, the server shows very minimal signs of life, but other than that, it's dead.
Did someone execute the wrong command? Sabotage!? We'll probably never know...
Of course I've confirmed other users experience the same error, from other parts of the world as well, so it's not just me or my internet connection, etc.

So the site's down, I've tweeted @rhettandlink and @linklamont several times, but no reaction and/or responses. Hmm...
And they've been tweeting themselves, though...

Anyway, the rather unpleasant side effect is this:

As you can see, most of the komm is unusable, because although most of it is hosted on NING, some of it (scripts, images, etc.) is hosted on the main site, which is x_x. This actually stalls some of the scripts, hindering functionality. The problem goes way beyond a few pictures not being displayed.

PROTIP: Host everything Komm related on NING, or at least implement a contingency mode.

Most of this editor doesn't work either, but It's kinda manageable.

Using two domains is OK, as long as the person taking care of it makes the right decisions. Having a production system not having contingency modes is what happens this night...
I'd like to see someone taking care of it though.

On the other hand tough, things may be quite severe. When data loss occurs, it takes it's time to roll backups, etc. I've worked in that industry for years. In that case, it can't be helped. However, please re-think design elements and contingency plans.

Cheers,
-- int 80h

3 members like this

▶ Reply to This

Replies to This Discussion

K-ModPermalink Reply by Gumbo123 on March 15, 2014 at 1:02pm

Yes, I noticed the same thing last night . . . went offline from the kommchat to post some welkomm notes but was unable to because the COMMENT box on member profile pages was either not showing at all or, if you could see the box, had none of the icons for format and attaching pics, video, etc.

FYI - - - Brian Layman the eHermit is the k-mod who does most of the site programming and maintains when fixes are needed. you might enjoy some techie talk with him.

▶ Reply

Permalink Reply by int 80h on March 15, 2014 at 1:34pm

I managed to get most of the editing parts working, by using Web developer tools (DOM inspector, etc) and stopping all scripts running from the main site. The edit boxes could be brought to almost normal working order that way. (Tip for future issues).

I know Brian, had a little message exchange with him in the past.

From what I can tell, the downtime lasted at least 18 hours, from before 23:00 UTC to after 17:00 UTC (2014-03-14 to 2014-03-15).

▶ Reply

Permalink Reply by Spencer (Smofgrog OF DOOM) on March 17, 2014 at 1:39pm

I was freaking out when it was not working. Lol i am such an addict

▶ Reply

K-ModPermalink Reply by Victoria (Centauromadoose) on March 17, 2014 at 6:31pm

Did they ever officially acknowledge the fact the site went down? Even a simple "The Rhett and Link website is down, but don't worry, we're working on it!" would have been nice to see.