What tools does Brownrice use to alert us to a compromised hosted web site or server? Let me show you:
OSSEC: A great open source tool that constantly monitors server log files and file systems in real-time. OSSEC’s log monitoring helps with an important part of PCI Compliance, it can be configured to automatically block bad guys from doing bad things, and its a fantastic tool for post-mortem hack analysis. We have OSSEC installed on all of our hosting servers, virtual servers, and managed customer servers. It reports back to a mother-ship server so we can keep an eye on things from a central location.
Five years ago we were constantly fighting off hackers who would hack an insecure web site then try and install a rootkit so that they could own the server. Now? Nothing. They don’t even try and attack the server. We have all sorts of rootkit detection software on our servers (rkhunter, OSSEC, etc.) and I’m starting to wonder why we bother when a hacker has everything they need when they’ve compromise a web site.
We’re rolling out our new web cam video players (less flash, more iOS and HTML5 support!) to our clients. The first official release was for the Taos Ski Valley web site, which you can see and use here: http://skitaos.org/webcams
So what’s the geek back-story on these video players? Read on…
Yesterday morning we had a client who’s got a site on a virtual server email to say:
Hi Oban –
I just had a business colleague say that he went to my site, got a malware warning, and his entire hard drive was wiped out instantly.
Hard drive instantly wiped out instantly?!? Pa-leeease!
Nonetheless, this is a WordPress site so Dave looked through the code and didn’t see anything immediately out of line. We both visited the customer’s site and neither of our hard drives were instantly wiped out (we are craaazy risk takers!) I also looked at what Google’s Safe Browsing site currently thought of our network – which was that everyone was clean a whistle.
Last night, just before turning off the lights and harassing my wife, I received a text message from our server monitoring software saying that the mail queue on one of our shared web servers had suddenly spiked. Lots of emails being pumped out of a shared web server is almost always the sign of something bad.
Logged into machine and examined one of the emails in the mail queue. Because we roll our own PHP its compiled with a patch that inserts the full path to the script that sent the email. Years ago, when we didn’t have this patch installed, determining which site and/or script sent an email could have taken hours – or be nearly impossible to figure out. Here’s what the mail header looked like (note: the actual web site address has been modified to protect the client):
Just another year of practically perfect network uptime. How many 9’s was it exactly? I dunno. However, this is starting to sound redundant. Our 2010 and 2011 uptime was also somewhere around 99.9999%.
Speaking of uptime, in case you aren’t aware, our network is “fiber cut proof.” What does that mean? Two of our upstream connections are via large capacity fiber optic cables, while our third is via high capacity, high speed microwave radios (the exact same technology that high speed financial traders use). So if our two fiber cables get cut we can push all of our traffic through our backup microwave connection, and your site and email don’t miss a beat.
More than 70,000 websites were compromised in a recent breach of InMotion. Thousands of websites were defaced and others had alterations made to give users a hard time accessing their accounts and fixing the damage. A similar attack hit JustHost back in June, and in a breach of Australian Web host DistributeIT just prior to that, hackers completely deleted more than 4,800 websites that the company was unable to recover. The incidents raise concern that hacker groups are bypassing single targets and hitting Web hosts directly, giving them access to tens of thousands of websites, rather than single targets. While the attacks have caused damage, they weren’t as malicious as they could have been. Rather than defacing and deleting, hackers could have quietly planted malware in the sites or stolen customer data. Web hosting companies could be one of the largest holes in non-government cybersecurity, since malicious hackers can gain access through openings left by the Web host, regardless of the security of a given site.
We’ve already closed these holes. Are you really still hosting your sites with the volume-based hosters!
There is nothing more tricky and fraught with potential problems than DNS upgrades.
This week we migrated from BIND to PowerDNS. Prior to the migration we dutifully tested PowerDNS on different servers, in different configurations, consulted other sysadmins who were running PowerDNS, and found all tests to be working flawlessly.
So we went ahead and upgraded all three of our DNS servers from BIND to PowerDNS, and watched…