Checking a potentially hacked machine and reinstalling packages when you break things


Sometimes you may see something odd go past in the logs, on the screen, or something might just seem a bit ‘off’ somehow. Its a good idea when this happens to just double check that things are as they should be, verify packages, binaries, check logs etc

Here are a few things i find helpful when doing this

Install something like rkhunter or chkrootkit, run it regularly and check the output. These tools are excellent at helping detect oddities that should not be there. I find rkhunter better in that it emails me regularly (every night) and lets me know if there are changes or things happening.

Sometimes the server is not mine (ok, a lot of the time since i work on peoples servers), so these things may not be installed already, and i want to check packages installed are the correct ones, and have not been replaced by a hacker, or some other odd process. I use something like this

For Redhat based systems (Fedora, Centos etc)

apt-get install yum-verify
yum verify-all

For Debian based systems its a little harder, but scriptable

dpkg -l \*|while read s n rest; do if [ "$s" == "ii" ]; then echo $n;
fi; done > ~/tmp.txt
for f in `cat ~/tmp.txt`; do debsums -s -a $f; done

These commands will take some time, and use a fair amount of CPU and disk IO when you run them, so be aware they may slow the VPS/Server down a little.
Also, be aware you may have a fair few false positives, usually on configurations which have changed from the default packages

IF a package appears to be tampered with you can reinstall it like this
Centos/RedHat/Fedora

yum reinstall packagename

Or reinstall all packages

yum reinstall $(yum list installed | awk '{print $1}')

Debian/Ubuntu

apt-get install --reinstall packagename

or reinstall all packages

apt-get install --reinstall $(dpkg --get-selections |grep -v deinstall)

Possibly not such a great idea to reinstall all packages, if its that bad you really should reinstall the server from scratch. Ideally if you have been root exploited in any way you should reinstall from scratch as nothing will be safe (especially your package manager or SSH daemon). The main reason i put these commands here is because they are handy if you break things also :)

Lastly, it pays to regularly run netstat to check for strange open ports or connections, ps for odd running things, and of course all your logs for things (especially apache error log).


One response to “Checking a potentially hacked machine and reinstalling packages when you break things”