Category Archives: Systems Administration

The new Brownrice Data Center!

Systems Administration

Our existing server space is close to capacity so we’re building a bigger and more awesome-r one!

The new Brownrice data center will be larger, greener, and more secure than our existing server space and will utilize a smart, fresh air cooling system with air conditioning and generator backup, multiple layers of physical security, and will improve on our already robust physical network redundancy.

It will also look *super* cool.

So if you’ve been looking for a place to co-locate your servers or server racks, look no further than Brownrice. And feel free to come by anytime and we’ll give you a tour.

New Summit Cam!

Systems Administration

I LOVE when ski areas put our cameras in far away and cool places.  Red River Ski Area installed this one and its called “The Summit Cam.”  Check it out:

This will only display if no Flash or Javascript


WordPress hack post-mortem

Systems Administration, Web Hosting

Yesterday morning we had a client who’s got a site on a virtual server email to say:

Hi Oban –

I just had a business colleague say that he went to my site, got a malware warning, and his entire hard drive was wiped out instantly.
Hard drive instantly wiped out instantly?!?  Pa-leeease!
Nonetheless, this is a WordPress site so Dave looked through the code and didn’t see anything immediately out of line.  We both visited the customer’s site and neither of our hard drives were instantly wiped out (we are craaazy risk takers!)   I also looked at what Google’s Safe Browsing site currently thought of our network – which was that everyone was clean a whistle.
Dave emailed the client to say that this sounded like a false alarm but to keep us posted.   I decided to run the site through Sucuri.net’s free site scan and bingo! a javascript exploit was found.

Hacker extraction – New personal best: 10 minutes!

Programming, Systems Administration, Web Hosting

Last night, just before turning off the lights and harassing my wife, I received a text message from our server monitoring software saying that the mail queue on one of our shared web servers had suddenly spiked.  Lots of emails being pumped out of a shared web server is almost always the sign of something bad.

10:25pm

Logged into machine and examined one of the emails in the mail queue.  Because we roll our own PHP its compiled with a patch that inserts the full path to the script that sent the email. Years ago, when we didn’t have this patch installed, determining which site and/or script sent an email could have taken hours – or be nearly impossible to figure out.   Here’s what the mail header looked like (note: the actual web site address has been modified to protect the client):

Continue reading Hacker extraction – New personal best: 10 minutes! →

Another fantastic year of uptime

Systems Administration, Web Hosting

Just another year of practically perfect network uptime.  How many 9’s was it exactly?  I dunno.  However, this is starting to sound redundant.  Our 2010 and 2011 uptime was also somewhere around 99.9999%.

Speaking of uptime, in case you aren’t aware, our network is “fiber cut proof.”  What does that mean?  Two of our upstream connections are via large capacity fiber optic cables, while our third is via high capacity, high speed microwave radios (the exact same technology that high speed financial traders use).  So if our two fiber cables get cut we can push all of our traffic through our backup microwave connection, and your site and email don’t miss a beat.

Revamping our backup servers

Systems Administration

Just a quick note:  Nearly all of our on-site and off-site back up servers have undergone big upgrades in the last few weeks.  We’re spending big bucks and large amounts of time improving our backup scheme.

I’d wager that many hosting companies skimp on this since this isn’t something the customer would ever know about – unless data was lost.  So worry not, we’ve got data covered.  In many places and in many versions.

Did I mention that incremental backup is free with all Brownrice email, shared hosting, and virtual server plans?

One ridge, two nerds, a couple of cameras, and lots of snow…

Systems Administration

The Taos Ski Valley ridge opened for the first time this year – nice and early with lots of snow.  Of course Dave, our lead systems administrator, suddenly needed the day off (pashaw!!). And my brother Sam, who works for TaosNet and who installed the nearly-famous Ridge-Cam, had some urgent things to “fix” on the ridge today. Sooo… this is what the Ridge-Cam normally looks like:

 

 

 

 

 

 

 

 

 

 

 

 

 

 

And this is what it looked like for a few minutes today…

Continue reading One ridge, two nerds, a couple of cameras, and lots of snow… →