Mail server maintenance tonight! May 30, 2012 No Comments
Quick update: We’re upgrading the servers that store your email tonight starting at 11pm ET. We anticipate that the outage will last about 60 minutes.
Last night, just before turning off the lights and harassing my wife, I received a text message from our server monitoring software saying that the mail queue on one of our shared web servers had suddenly spiked. Lots of emails being pumped out of a shared web server is almost always the sign of something bad.
Logged into machine and examined one of the emails in the mail queue. Because we roll our own PHP its compiled with a patch that inserts the full path to the script that sent the email. Years ago, when we didn’t have this patch installed, determining which site and/or script sent an email could have taken hours – or be nearly impossible to figure out. Here’s what the mail header looked like (note: the actual web site address has been modified to protect the client):
Another fantastic year of uptime February 14, 2012 No Comments
Speaking of uptime, in case you aren’t aware, our network is “fiber cut proof.” What does that mean? Two of our upstream connections are via large capacity fiber optic cables, while our third is via high capacity, high speed microwave radios (the same technology that telephone lines often use). So if our two fiber cables get cut we can push all of our traffic through our backup microwave connection, and your site and email don’t miss a beat.
Fiber cut PROOF!!!
Just a quick note: Nearly all of our on-site and off-site back up servers have undergone big upgrades in the last few weeks. We’re spending big bucks and large amounts of time improving our backup scheme.
I’d wager that many hosting companies skimp on this since this isn’t something the customer would ever know about – unless data was lost. So worry not, we’ve got data covered. In many places and in many versions.
Did I mention that incremental backup is free with all Brownrice email, shared hosting, and virtual server plans?
One ridge, two nerds, a couple of cameras, and lots of snow… December 8, 2011 No Comments
The Taos Ski Valley ridge opened for the first time this year – nice and early with lots of snow. Of course Dave, our lead systems administrator, suddenly needed the day off (pashaw!!). And my brother Sam, who works for TaosNet and who installed the nearly-famous Ridge-Cam, had some urgent things to “fix” on the ridge today. Sooo… this is what the Ridge-Cam normally looks like:
And this is what it looked like for a few minutes today…
4.8 years! October 5, 2011 No Comments
Mass hacks – Not in our House! October 3, 2011 No Comments
From a recent Slashdot article:
More than 70,000 websites were compromised in a recent breach of InMotion. Thousands of websites were defaced and others had alterations made to give users a hard time accessing their accounts and fixing the damage. A similar attack hit JustHost back in June, and in a breach of Australian Web host DistributeIT just prior to that, hackers completely deleted more than 4,800 websites that the company was unable to recover. The incidents raise concern that hacker groups are bypassing single targets and hitting Web hosts directly, giving them access to tens of thousands of websites, rather than single targets. While the attacks have caused damage, they weren’t as malicious as they could have been. Rather than defacing and deleting, hackers could have quietly planted malware in the sites or stolen customer data. Web hosting companies could be one of the largest holes in non-government cybersecurity, since malicious hackers can gain access through openings left by the Web host, regardless of the security of a given site.
We’ve already closed these holes. Are you really still hosting your sites with the volume-based hosters!
Brownrice on the bus! September 26, 2011 No Comments
Oban Talks Hackers 2011 New Mexico Tour May 31, 2011 2 Comments
The “Oban Talks Hackers 2011 New Mexico Tour” continues this Friday, June 3rd at the prestigious Hotel don Fernando de Taos at 11:30am.
Its $15 for Taos Chamber of Commerce members to attend and $20 for non-members. Note that this loot doesn’t line my pockets, it goes toward the fantastic hotel lunch that you’ll be served while I dissect how hackers will get into your site and make your life miserable. The details about the talk are here: http://www.chambermaster.com/directory/jsp/events/EventPage.jsp?ccid=80&eventid=8511
Please RSVP to Steve Fuhlendorf at firstname.lastname@example.org.
If enough people RSVP we’ll do a live webcast of this event as well. i.e. It’ll be blacked out just like a big sporting event unless it sells out locally, so RSVP!
PowerDNS migration creates bug ripples March 30, 2011 5 Comments
There is nothing more tricky and fraught with potential problems than DNS upgrades.
This week we migrated from BIND to PowerDNS. Prior to the migration we dutifully tested PowerDNS on different servers, in different configurations, consulted other sysadmins who were running PowerDNS, and found all tests to be working flawlessly.
So we went ahead and upgraded all three of our DNS servers from BIND to PowerDNS, and watched…