It sucks getting hacked
Every now and then servers get hacked. Often because of an exploitable webapp, or because (most commonly) a weak, easily guessable password was used on a well known user account (like ‘root’ or ‘info’ or ‘test’).
Once hackers gain access they often install some kind of malware. e.g. something that goes about trying to brute force other servers. Or a proxy, through which they can send out email spam.
Hackers installing nginx, an running open proxies
Recently Glenn here pointed out that a few of us had seen the VPS’s around with port 8080 open to public. It appears the application nginx was opening these ports, and this is a popular web caching tool.
However the difference in these cases was that our customers had not installed this application. Instead, hackers had gotten access to the server (e.g. via a weak password) and installed nginx and they were using that setup so our customers’ servers were being used as an open proxy.
This allows people to run IRC bots and other interesting things via our customers’ VPSs.
Spotting the malware
Its fairly easy to spot, if you are not running it just check with
sudo netstat -pant |grep nginx
If you see this, and didn’t install nginx yourself, then you can shut it down and safely remove the binary.
Also check cron jobs that reinstall and restart it, as they usually come with it.
Cleaning up after the exploit
You will need to figure out, also, how the exploit was originally installed. Else a hacker will re-exploit that hole and reinstall malware gain.
RimuHosting can help with trying to find how your server was exploited, and advice on how to ‘clean up’ your server. e.g. often a clean reinstall with a new password is a good idea. See also http://rimuhosting.com/knowledgebase/rimuhosting/argh-my-server-was-exploited
Update: As an update to this story, we see that we found it before everyone else it seems. Its now on
Slashdot http://linux.slashdot.org/story/09/09/12/1413246/First-Botnet-of-Linux-Web-Servers-Discovered
and the register http://www.theregister.co.uk/2009/09/12/linux_zombies_push_malware/
We noticed this one a we while back, pity we didn’t get around to blogging it earlier.
Update 2: I see now there is an advisory for nginx advising a buffer overflow http://www.kb.cert.org/vuls/id/180065