We get asked these questions regularly, and 9 out of 10 times it may be because your server may have been hacked, either on a user level, web application, or more.

Here are some ways to quickly check for the most common things we see , and some of the processes we go through to find the culprits

Probably the most common thing we see, is insecure web applications. Either sending spam emails, or sending the load up high by running applications from the apache user (bots or similar).

First of all, check what is listening to external ports using something like

netstat -pant |grep LISTEN

You probably want to look into stopping the processes that seem suspicious, this may involve restarting apache or using the kill command. Next run ps auxf and see if there is anything odd, you should regularly check a ps even if its just to familiarize yourself with what is normal or not.

If you find files that are running, use find to locate where on the filesystem they are

find / -name "filename"

Since most sites use Apache Web Server to serve web pages, check the error.log, you can find it in /var/log/apache2 or /var/log/httpd (depending what distro you use).

Things to look for are outputs from shell commands (ie wget or curl) or anything that just does not look “right”. If you can get the time/day from these it helps, even if its an approximate (between log above it and below it).

Once you have that, check the access.log and see what happened on those days. Check for repeated POST entries in a row, or odd file names in image directories.

Checking for larger logs in /var/log will give you hints since 1000 emails is often created with 1000 hits ( ls -sh /var/log/ ). If you have 3 or 4 very large apache and mail logs, then you know they rotate every day, that’s 3-4 days ago it happened.

You can check apache logs for what IP address is accessing your website most, this is more often the spammer , this is how

cat access.log | awk '{print $1}' | sort | uniq -c | sort -n | tail

The first number result is the amount of hits, the second is the IP. You can then check the other logs for any instance of that IP, view what pages they were on.

Once you narrow down the time it happened its a lot easier to find out what happened. If you can Narrow that down to an IP address, this is even better, you can check the logs to find exactly what that ip address did, and block it.

If you are dealing with spamming from your server, check the mail queue (run ‘mailq’) it should give you an idea what has been sent, check /var/spool/postfix/deferred to read the emails in the queue (use less or vi ) to see if they have headers and who they are from.

For postfix users, its easy enough to clear any email queued up in your server, just run something like

postsuper -d ALL

Beware this will also purge any legitimate email at the same time.

Once you have tracked it down to and IP you can then track it down to a file or two. It pays to grep for that file in case the user hacking changed IP several times, and try and track back to the original entry point.

Once found, check what permissions that file/script had access too (read the code, or just check what it could possibly have written to) and replace those things.
If you have a CMS, upgrade that, backup all images and check themes/plugins for back doors. Remove everything else and replace with new ones from the fresh download you do.

Check all tmp dirs for files that should not be there, /tmp and /var/tmp are often used. Make sure you use ls -al and check for directories named as ” ” (as in a space) or any oddities and extra line feeds for hidden directories. Another one often used is naming a directory ” .” so it looks like a . rather than a space dot etc.

If your server has been root exploited then re-install and wipe as much data as you possibly can, only migrate what you must. Often if its root exploited you will find problems with package management being unable to write to some files, or similar.
Usually if a server is root exploited they can replace binaries (ie your mailserver) and you will not notice, they intercept everything, and its safe to assume they have all your passwords logged as clear text somewhere.