Battling WordPress Malware with Git & Logstash

For years, this site and several others hosted on its server fell victim to a series of malware attacks. This is the post-mortem of how I discovered and recovered from the infections.

Background

KyleAnderson.us started in 2009 as my first WordPress site. It was originally on a shared host, so naturally my only interactions with it were through cPanel and WordPress itself. This was all fine and dandy, so I took on several other WordPress installations for my good friend Troy and we agreed to split the hosting cost.

Then, in 2015, the hosting provider started sending automated emails regarding high resource usage. (Everything except CPU/memory usage is unlimited for these kinds of hosts, and going overboard can have your site temporarily or permanently shut down.) I thought they were giving me the business, trying to upsell me on a more expensive hosting package just because their server was overcrowded.

Misplaced though my blame was, I made the decision to migrate to an AWS t2.micro virtual server instance. This was a blessing and a curse as I was free from arbitrary and unknown resource limits, but at the same time more responsible for administration of the server.

The Hacks

In brief, here is what the hacks were able to accomplish:

  • Send tens of thousands of spam emails by installing a cron job with the www-data user
  • Create dummy HTML pages (other infections) in a plugin folder with spam content
  • Inject malicious javascript, masquerading as a minified jQuery file, into all sites
  • Upload arbitrary code via a backdoor script

Yeah, not good.

File permissions

My most basic mistake was with setting file permissions too liberally. There’s a trade-off with WordPress: One-click installs and updates require the Apache user to have read/write access to all of its own files. I stuck with this scheme (chmod -R 775) for each site’s entire file base. It served me well on trusted intranet installations of WordPress, so why not?

What I didn’t realize is that this left core WordPress files open to exploit. New policy going forward, based on the excellent Hardening WordPress documentation, is 755/644 for files and directories with exception to the wp-content folder. This is to allow content uploads and plugin updates. The only hindrance is needing to change permissions to upgrade WordPress itself.

Git/Gitlab

These are key tools in discovering when new malicious code was added to a site’s code base. Every hour, a full commit is made on each individual site then pushed to Gitlab, which is installed on a VM running on a fancy new (read: refurbished) Dell R610 virtualization server running at home. This has the added benefit of off-site backup!

48GB of RAM, 2x Xeon E5540 @ 2.53GHz, and two RAID 10 arrays totaling 1.2 TB. Thank you, ebay!

48GB of RAM, 2x Xeon E5540 @ 2.53GHz, and two RAID 10 arrays totaling 1.2 TB. Thank you, ebay!

This is the bash script that runs every hour on the web server. Some files are excluded from commits because they are changed frequently by Wordfence and don’t contain any important data.

ubuntu@ip-172-31-11-12:/var/www/html$ cat ~/scripts/wwwgit.sh
#!/bin/bash


cd /var/www/html/;
# Set permissions to be correct for wordpress
sudo find ./ -type d -exec chmod 755 {} \;
sudo find ./ -type f -exec chmod 644 {} \;
sudo find ./ -type d -name 'wp-content' -exec chmod -R 775 {} \;
sudo find ./ -name '.git' -exec chmod 700 {} \;
sudo chown -R ubuntu:www-data ./*;


# In each installation root, add and commit all files
commitAndPush () {
git add -u .;
git add $(git ls-files -o --exclude-standard);
git reset HEAD ./logs/*
git reset HEAD ./wp-content/plugins/wordfence/tmp/configCache.php;
git reset HEAD ./wp-content/wflogs/config.php;
git reset HEAD ./wp-content/wflogs/ips.php;
git reset HEAD ./wp-content/wflogs/attack-data.php;
git reset HEAD ./wp-content/debug.log;
git commit -m "Regular commit - $(date)";
git push origin master;
}
cd /var/www/html/kyleanderson.us/
commitAndPush
cd /var/www/html/troyfarah.com/
commitAndPush
cd /var/www/html/delunula.com/
commitAndPush
cd /var/www/html/phxsux.com/
commitAndPush

The commits are easy to navigate with Gitlab.

gitlab_activity

And, with Slack integration, keeping tabs on file changes is a breeze.

slack

Finally, with proper source control, removing malware couldn’t be easier. All it takes is a simple git revert.

Logstash

While git was able to track new infections, existing malware was more difficult to track down.

Some patterns were simple to discover, such as PHP files in a JS directory. Other hacks inserted obfuscated code at the beginning or end of legitimate PHP files. This still required manually going through all 8,000 or so PHP files, and that wouldn’t do.

Pro-tip: Anything with an eval call is probably malicious.

Pro-tip: Anything with an eval call is probably malicious.

Enter Logstash, the L in Elastic’s fantastic ELK stack. Logstash watches log files, parses them based on patterns that you specify, stores them in Elasticsearch and provides the Kibana tool for visualizing that data.

In this case, the log files from the web server are copied to kyleanderson.us/logs (HTTP basic auth enabled). Then, the Logstash server fetches the file so Logstash can start importing. Just like the git commits, this is scheduled once per hour.

The strategy here is to look for suspicious POST requests, the kind that would allow uploading of new malicious code. Here is a prime example:

Many of those POST requests are to pages that are not legitimate.

Many of those POST requests are to pages that are not legitimate.

Once identified, the files were deleted or repaired then manually committed through git.

It took a week or so of on-and-off monitoring and scrubbing, but it’s safe to say at this point that all malware has been identified and removed.

As a bonus, the Kibana dashboard, which lets you drill down with any criteria you wish, has revealed some notable hacking attempts, such as requests to supposedly exploitable software that isn’t installed (this actually seemed to crash MySQL) and over 6 million POSTs to wp-login.php in the course of an hour – an attempt to brute-force passwords.

wp-login

Sorry, FCKEditor is not installed!

Sorry, FCKEditor is not installed!

A Word on Wordfence

During all of this, all sites except kyleanderson.us had the Wordfence plugin installed. The plugin is great for restricting login attempts and notifying you of changed files. Nothing, however, is perfect. Wordfence failed to detect some of the malicious files, and when it was set to scan all files as if they were executable, it would consume large amounts of CPU daily. It’s a good plugin to have, but know that simply having it installed won’t solve all malware woes.

Final Takeaways

If you operate your own WordPress installations and they are internet-facing, it is clear from my experience that neglect will only lead to compromised websites. Some general best practices and takeaways:

  • Keep WordPress Core, Plugins and Themes up to date as often as possible
  • Ensure correct file permissions are in place
  • Monitor file changes, whether manually, with a plugin like Wordfence or with source control like git
  • Restrict cron for the Apache user
  • Only allow SSH authentication with keys. I shudder to think how bad this would have been if password auth was allowed.
  • Because hacking attempts can crash the MySQL database, run a simple script to restart the service if it’s down.

About The Author

Kyle Anderson
I'm a media and IT professional and JavaScript developer who worked most recently as an Associate Broadcast IT Engineer (Tier II) for CNN in Atlanta. One of my life-long goals is to help bridge data divides - missing connections between software systems and data stores - promoting inter-system communication and automation. Many of the projects described here reflect this goal in some way or another.

Web Development