WordPress hack post-mortem

Yesterday morning we had a client who’s got a site on a virtual server email to say:

Hi Oban –

I just had a business colleague say that he went to my site, got a malware warning, and his entire hard drive was wiped out instantly.
Hard drive instantly wiped out instantly?!?  Pa-leeease!
Nonetheless, this is a WordPress site so Dave looked through the code and didn’t see anything immediately out of line.  We both visited the customer’s site and neither of our hard drives were instantly wiped out (we are craaazy risk takers!)   I also looked at what Google’s Safe Browsing site currently thought of our network – which was that everyone was clean a whistle.
Dave emailed the client to say that this sounded like a false alarm but to keep us posted.   I decided to run the site through Sucuri.net’s free site scan and bingo! a javascript exploit was found.

Hacker extraction – New personal best: 10 minutes!

Last night, just before turning off the lights and harassing my wife, I received a text message from our server monitoring software saying that the mail queue on one of our shared web servers had suddenly spiked.  Lots of emails being pumped out of a shared web server is almost always the sign of something bad.

10:25pm

Logged into machine and examined one of the emails in the mail queue.  Because we roll our own PHP its compiled with a patch that inserts the full path to the script that sent the email. Years ago, when we didn’t have this patch installed, determining which site and/or script sent an email could have taken hours – or be nearly impossible to figure out.   Here’s what the mail header looked like (note: the actual web site address has been modified to protect the client):

Continue reading

Another fantastic year of uptime

Just another year of practically perfect network uptime.  How many 9’s was it exactly?  I dunno.  However, this is starting to sound redundant.  Our 2010 and 2011 uptime was also somewhere around 99.9999%.

Speaking of uptime, in case you aren’t aware, our network is “fiber cut proof.”  What does that mean?  Two of our upstream connections are via large capacity fiber optic cables, while our third is via high capacity, high speed microwave radios (the exact same technology that high speed financial traders use).  So if our two fiber cables get cut we can push all of our traffic through our backup microwave connection, and your site and email don’t miss a beat.

Mass hacks – Not in our House!

From a recent Slashdot article:

More than 70,000 websites were compromised in a recent breach of InMotion. Thousands of websites were defaced and others had alterations made to give users a hard time accessing their accounts and fixing the damage. A similar attack hit JustHost back in June, and in a breach of Australian Web host DistributeIT just prior to that, hackers completely deleted more than 4,800 websites that the company was unable to recover. The incidents raise concern that hacker groups are bypassing single targets and hitting Web hosts directly, giving them access to tens of thousands of websites, rather than single targets. While the attacks have caused damage, they weren’t as malicious as they could have been. Rather than defacing and deleting, hackers could have quietly planted malware in the sites or stolen customer data. Web hosting companies could be one of the largest holes in non-government cybersecurity, since malicious hackers can gain access through openings left by the Web host, regardless of the security of a given site.

We’ve already closed these holes.   Are you really still hosting your sites with the volume-based hosters!

~ Oban

Oban Talks Hackers 2011 New Mexico Tour

A web site owner

The “Oban Talks Hackers 2011 New Mexico Tour” continues this Friday, June 3rd at the prestigious Hotel don Fernando de Taos at 11:30am.

Its $15 for Taos Chamber of Commerce members to attend and $20 for non-members.  Note that this loot doesn’t line my pockets, it goes toward the fantastic hotel lunch that you’ll be served while I dissect how hackers will get into your site and make your life miserable.  The details about the talk are here: http://www.chambermaster.com/directory/jsp/events/EventPage.jsp?ccid=80&eventid=8511

Please RSVP to Steve Fuhlendorf at steve@taoschamber.com.

If enough people RSVP we’ll do a live webcast of this event as well.  i.e.  It’ll be blacked out just like a big sporting event unless it sells out locally, so RSVP!

Thanks,

~ Oban

PowerDNS migration creates bug ripples

Ripples

There is nothing more tricky and fraught with potential problems than DNS upgrades.

This week we migrated from BIND to PowerDNS.  Prior to the migration we dutifully tested PowerDNS on different servers, in different configurations, consulted other sysadmins who were running PowerDNS, and found all tests to be working flawlessly.

So we went ahead and upgraded all three of our DNS servers from BIND to PowerDNS, and watched…

Continue reading

The most secure shared hosting – Brownrice!

Hacked!

Until recently our shared hosting servers suffered from some of the same vulnerabilities that many of the volume-based hosting providers do.  Namely, if one site on a shared server was hacked it was possible for the hacker to deface other sites on the same server that had files or directories with loose permissions.  I.e. 777 permissions.

Yep, even if you diligently keep your site and code up-to-date your site could still be hacked because someone else’s site on the same server was hacked.

Ugly, eh?

Read about this nastiness in action at Network Solutions, Bluehost, Dreamhost, and GoDaddy here: Continue reading

Can’t login to Squirrelmail!

Squirrelmail, aka “The old webmail,” has gone wonkey on us as we migrate it to more robust servers and upgrade it.  i.e.  Its online but its broke.  So don’t use it.

For the four of you who actually use “The old webmail” to read your email, I apologize.  In the meantime you can use the real webmail, here.

We’ll update this blog, and our twitter account, when its fully functioning again.