Currently we are being inundated with scans and exploits looking for new machines to take over, most of these are using insecure phpmyadmin installs.

If you are unsure if your box has been hacked then the things to look for are similar to this…

Perl has a high CPU load in ‘top’

top - 20:43:09 up 174 days, 20:47,  1 user,  load average: 1.00, 1.00, 1.00
Tasks: 279 total,   2 running, 277 sleeping,   0 stopped,   0 zombie
Cpu(s):  7.1%us,  0.3%sy,  0.0%ni, 92.4%id,  0.1%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:    921948k total,   897940k used,    24008k free,   130788k buffers
Swap:   131064k total,    79704k used,    51360k free,   210824k cached

PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
2239 www-data  20   0  4896 2872 1224 R 99.9  0.3   3853:33 perl   <<---
21482 root      20   0  2344 1112  744 R  3.8  0.1   0:00.04 top
28751 root      20   0 43108  13m 1604 S  1.9  1.4   6:05.22 apache2
1 root      20   0  1824  404  364 S  0.0  0.0   1:43.69 init

Odd things running as the apache user

~# ps aux |grep www-data
www-data  9442  0.0  2.1  23060  8596 ?        S    Aug18   0:00 /usr/sbin/apache2 -k start -DSSL
www-data  9448  0.0  0.0      0     0 ?        Z    Aug18   0:00 [sh] <defunct> <- sure sign here
www-data  9458 99.2  0.7   4916  2888 ?        R    Aug18 1297:10 /usr/sbin/apache/loggs
www-data  9701  0.0  0.4   4916  1892 ?        S    Aug18   0:00 /usr/sbin/apache/loggs  <- this doesnt exist
www-data  9704  0.0  0.7   5860  3020 ?        R    Aug18   0:17 inetd   <- this should never run as apache user
www-data 31545  0.0  1.1  21164  4628 ?        S    08:20   0:00 /usr/sbin/apache2 -k start -DSSL
www-data 31546  0.0  1.1  21164  4620 ?        S    08:20   0:00 /usr/sbin/apache2 -k start -DSSL
root     31672  0.0  0.2   2732   800 pts/1    D+   08:32   0:00 grep www-data

The best recovery is a reinstall or revert to a non-hacked backup. You can do this in your control panel easily yourself. Once you have restored to a non-hacked backup you need to secure it. Shutdown all daemons like apache, postfix, mysql etc. Change any user passwords (especially root!)

Find insecure instances of web software such as phpmadmin, or other things.

This will find most cases of phpmyadmin

find / -iname "*phpmyadmin*"

Then you can move that somewhere outside a documentroot or use htaccess to deny anyone usage.

Now you probably want to mount the hacked/exploited image . First you attach it as /dev/xvda3 in your rimuhosting control panel, this will reboot your vps. When it comes back up re-shutdown all daemons until you have finished.

Run:

mkdir /oldfs
mount /dev/xvda3 /oldfs

Now you can view all your old VPS files. You can copy over email from /oldfs/var/mail to /var/mail or mysql from /oldfs/var/lib/mysql to /var/lib/mysql etc. Be very careful about any files you copy over, they may be exploited.

Once you have done that, and gone over anything else with a fine tooth comb, and updated any other packages, you can restart your daemons again. Keep an eye on logs regularly for more exploits and hacks.

Image credit flickr:Clive Darr

It has come to our attention that we have had a decent number of  alerts about machines scanning on our networks in the last day. All of them have been running hacks from /tmp named things like /tmp/dd_ssh etc. These files are owned and run by the apache user, and on further investigation all put there by an old PHPMyAdmin install.

If you run something like PHPMyAdmin then this is the time to upgrade it. If you get stuck or are unsure in any way then we are happy to help out.

It always pays to keep your system up to date to prevent excessive traffic or downtime.

This script will search /var/www (changeable in a variable) for any wordpress installs and make sure its upgraded to the latest version.
It will run a backup to /root/wp_upgrade/ of all files and database before doing anything with the site in case of major catastrophe.
It pays to check each site after the upgrade to make sure plugins all worked (and upgrade plugins before the upgrade works also!).

Let me know if you have any bugs at all, or any problems.

Here’s your script to upgrade them all.

wget http://b.ri.mu/files/wordpress-upgrade-3.0.1.sh 
sh wordpress-upgrade-3.0.1.sh

While I was kicking back camping in Yosemite Justin here has been ‘messing about’ with CSS on our Pingability web check and alert service.  Bringing it from my 2003 design to something a little more contemporary.  Thanks, Justin.

Before

After

–

Pingability lets you monitor your websites pages, check that things are running OK, and alert you (via email, txt or phone) when they are not.  It supports SSH actions (so you it can, say, SSH in and restart apache/mysql or reboot your server) if a web page stops responding.  And it can check things like email servers are working, DNS is returning results, and ports are open.  There are free and paid options.

We use gandi.net internally for some of the domains we have.  Gandi have given me a few ‘promo codes’ for 66% off .org domains.

ORG2010-4B82-5C48-3F27
ORG2010-24C9-AB2B-0C59
ORG2010-481D-7BC2-A29D
ORG2010-7631-D3DE-DF74
ORG2010-FF7A-39C1-FB54
ORG2010-48E2-9F2C-F006
ORG2010-87EF-F85F-01AD

These codes will only work once each.  So if you use one (by using that as the promo code when you order at gandi.net), just mention that in a comment here?

If the codes below are all used up and you are a RimuHosting customer, just email us and there are a few extra ones we can give out.

Want to upgrade from a VPS on a shared host to a VPS on a dedicated server?  More memory, more disk space, dedicated CPU and disks?  We have a great offer for a few of you…

We have a few ‘old’ servers (only a couple of years old).  Most are retired host servers or dedicated servers returned from customers who no longer need them.  These server specs are mostly ones we no longer offer on our ordering pages.

Rather than let them gather dust, we thought we would offer them to customers who are currently have a VPS on a shared host.  And would like an upgrade to a dedicated machine.  You would get (most likely) more memory; more disk space; and, most importantly, less contention.  The CPU and disk IO is dedicated solely to you.  See also http://rimuhosting.com/knowledgebase/rimuhosting/vps-setup-on-dedicated-servers

The process is very simple, no work is required.  You simply let us know which server you’d like.  And we move your VPS to that machine.  You get to keep your IP address.  And no configuration changes or migration work is required.  There is simply a few minutes per GB of disk space you use downtime while we move your VPS image to the new machine.

The servers on offer are:

• host157.rimuhosting.com C2D E6400/2GB RAM/2x80GB Disks 110.00 USD/month
• SOLD! host146.rimuhosting.com C2D E6400/2GB RAM/2x250GB Disks 110.00 USD/month
• SOLD! host235.rimuhosting.com C2D E6400/2GB RAM/2x250GB Disks 110.00 USD/month
• SOLD! host214.rimuhosting.com C2D E6400/2GB RAM/2x400GB Disks 110.00 USD/month
• SOLD! host188.rimuhosting.com C2D E6400/4GB RAM/2x400GB Disks 110.00 USD/month
• SOLD! host190.rimuhosting.com C2D E6400/4GB RAM/2x400GB Disks 110.00 USD/month
• SOLD! host162.rimuhosting.com C2D E6400/4GB RAM/2x500GB Disks 115.00 USD/month
  
• SOLD! host296.rimuhosting.com 2xXeon E5310 (8 cores)/4GB RAM/2x250GB Disks 150.00 USD/month
• SOLD! host326.rimuhosting.com 2xXeon E5310 (8 cores)/4GB RAM/2x250GB Disks (4 bays) 170.00 USD/month
• host262.rimuhosting.com 2xXeon E5310 (8 cores)/4GB RAM/2x400GB Disks (4 bays) 175.00 USD/month
• host444.rimuhosting.com 2xXeon E5410 (8 cores)/4GB RAM/2x250GB Disks (4 bays) 175.00 USD/month
• SOLD! host548.rimuhosting.com 2xXeon E5410 (8 cores)/12GB RAM/2x500GB Disks (4 disks) 200.00 USD/month

The fine print:

• Disk upgrade to 1TB 7200RPM SATA drive for +13.00 USD/month per drive. A one off 49.00 USD setup fee per upgrade request applies.
  
• Sorry no buydowns on these servers.
• The memory size is fixed (since we have only a limited supply of the memory required for some of the older machines).
• First come first served.  Each of these servers is a one-off.  We will try to keep this post updated as servers are grabbed by others.
• Private Network connections are available +130 USD setup, +10.00 monthly (unmetered, Gb port) on Xeon servers only.
• These deals are provided on a first-in first-serve basis and subject to availablility are valid till the end of July 2010. To order, follow the link provided, leave the default values set on the order page but write a note including the hostname/quote you are after in the order notes, along with a reference to this page.
• All servers (except where noted otherwise) have two (2) drive bays supporting SATA disks.
• Subtract about 512MB from the server memory size (this will be used by the VPS hypervisor).

If you’d like to upgrade just pop in a support ticket and let us know which server you wanted to have.  We will work to move your VPS to that dedicated server within 24 hours and update your VPS’s billing from next month (no extra cost for this month).

Note: All these servers are at our Dallas datacenter.

We often encourage our users to use software which is better, and does things in a standard way. When it comes to control panels we prefer Virtualmin. It works with the distro you are using and the packages that come with it rather than setting up things in its own odd way and tieing you into using strange configs that lock you into using it.
Also it imports from all other platforms nicely (cpanel, plesk etc) which is handy to those who are locked into a lot of the more overpriced web panels. But the best thing yet is that its a reasonable price, in fact the GPL version is free and fully functioning!

By default it uses FCGI for PHP, which is a great option, however we had a customer who was struggling to keep ram usage down so wanted to convert to mod_php which was slightly faster and used less memory. Since he had more than a few domains this could have been painstaking, in fact it was until i emailed one of the creators of Virtualmin.
Not only did i get a great response, but i got it within minutes!

I had wondered if Virtualmin had a command line interface, and thus asked if there was any command line way i could do it, i got this reply.

Yes – you can use the command :

virtualmin modify-web --domain whatever.com --mode mod_php"

You can combine this with a shell loop using the output from :

virtualmin list-domains --with-feature web --name-only

Something like :

for dom in `virtualmin list-domains --with-feature web --name-only`; do
  virtualmin modify-web --domain $dom --mode mod_php
done

I modified this somewhat so i had an exclude list and then went with it. I think perhaps I should have put a sleep in there somewhere, the load went rather high for some time

[root@hostname ~]# virtualmin
usage: /usr/sbin/virtualmin  [args..]
   or: /usr/sbin/virtualmin help 

Available commands :

Backup and restore
------------------
backup-domain            Backup one or more virtual servers
list-scheduled-backups   Outputs a list of scheduled backups
restore-domain           Restore one or more virtual servers

Virtual servers
---------------
create-domain            Create a virtual server
delete-domain            Delete one or more virtual servers.
disable-domain           Temporarily disable a virtual server
enable-domain            Re-enable one virtual server
migrate-domain           Imports a virtual server from some other product
modify-domain            Change parameters of a virtual server
move-domain              Change the owner of a virtual server
unalias-domain           Convert an alias domain into a sub-server.
unsub-domain             Convert a sub-domain into a sub-server.
list-domains             Lists all virtual servers.
notify-domains           Send email to some or all virtual server owners.
validate-domains         Check the configuration of virtual servers
enable-feature           Turn on some features for a virtual server
disable-feature          Turn off some features for a virtual server
modify-dns               Change DNS settings for virtual servers
modify-spam              Change spam filtering and delivery settings for a
                         virtual server
modify-web               Change a virtual server's web configuration
modify-mail              Change mail-related settings for some domains
resend-email             Re-send the signup email for a domain

Mail and FTP users
------------------
create-user              Create a mail, FTP or database user
delete-user              Delete a mail, FTP or database user
modify-user              Change attributes of a mail, FTP or database user
list-users               List users in a virtual server
list-available-shells    List all shells for use with domain owners and
                         mailboxes
list-mailbox             Dump inbox email for one user

Mail aliases
------------
create-alias             Create a new mail alias
create-simple-alias      Adds a mail alias to some domain, with simple
                         parameters
delete-alias             Delete a mail alias
list-aliases             List aliases for a virtual server
list-simple-aliases      Lists mail aliases in a simple format for some domain
create-simple-alias      Adds a mail alias to some domain, with simple
                         parameters
list-simple-aliases      Lists mail aliases in a simple format for some domain

Server owner limits
-------------------
disable-limit            Removes access to some feature or edit capability for
                         some virtual servers
enable-limit             Grants access to some feature or edit capability for
                         some virtual servers
modify-limits            Changes the owner limits for some virtual server
modify-resources         Changes resource limits for some virtual server(s)

Extra administrators
--------------------
create-admin             Creates an extra administrator for a virtual server
delete-admin             Deletes an extra administrator from a virtual server
modify-admin             Updates an extra administrator for a virtual server
list-admins              Lists administrators belonging to a virtual server

Custom fields
-------------
list-custom              List custom fields for virtual servers
modify-custom            Modify custom fields for a virtual server

Databases
---------
create-database          Creates a database for a virtual server
delete-database          Deletes one database
import-database          Adds an existing database to a virtual server
list-databases           Lists databases for some virtual server
modify-database-hosts    Adds or removes an allowed MySQL host for some or all
                         domains 

Reseller accounts
-----------------
create-reseller          Create one reseller account
delete-reseller          Delete one reseller
modify-reseller          Modify attributes for an existing reseller
list-resellers           List existing resellers

Script installers
-----------------
install-script           Install one third-party script
delete-script            Un-install one script from a virtual server
list-scripts             Display script installed into some virtual server
list-available-scripts   List known scripts

Proxies and balancers
---------------------
create-proxy             Adds a per-directory proxy to some domain
delete-proxy             Removes a proxy balancer from some domain
list-proxies             Lists web proxy balancers in some domain

PHP versions
------------
delete-php-directory     Remove any custom version of PHP for some directory
set-php-directory        Set the version of PHP to run in some directory
list-php-directories     List all directories in which a specific version of PHP
                         has been activated 

SSL certificates
----------------
list-certs               Output the certificates for some or all virtual
                         servers.
generate-cert            Generate a new self-signed cert or CSR for a virtual
                         server.
install-cert             Replace the SSL certificate or private key for a
                         virtual server. 

Virtual server templates
------------------------
create-template          Creates a template for use by new domains.
delete-template          Removes one virtual server template.
get-template             Outputs all settings in a template.
modify-template          Changes one or more settings in a template.
list-templates           List available templates for new domains

Account plans
-------------
create-plan              Creates a new account plan for use with virtual
                         servers.
delete-plan              Removes one existing account plan.
modify-plan              Modifies an existing account plan for use with virtual
                         servers.
list-plans               List available account plans for new domains

Other scripts
-------------
change-licence           Change a system's Virtualmin license key
change-password          Changes the password of some Virtualmin user.
check-config             Run the Virtualmin config check
check-connectivity       Checks the external accessibility of virtual servers.
copy-mailbox             Copy mail from one location to another, perhaps
                         converting formats.
create-redirect          Adds a web redirect or alias to some domain
create-shared-address    Adds an IP address for use by multiple virtual servers
delete-redirect          Removes a web redirect or alias from some domain
delete-shared-address    Removes an IP address that can be used by virtual
                         servers
disable-writelogs        Disable logging via program
enable-writelogs         Enable logging via program
get-dns                  Output all DNS records for a domain.
get-ssl                  Output SSL certificate information for a domain.
info                     Show general information about this Virtualmin system.
licence-info             Show license counts for this Virtualmin system.
list-bandwidth           Shows bandwidth usage for some domain over a date range
list-commands            Lists API scripts available
list-features            Lists features available when creating a domain
list-php-ini             Show PHP variables for some or all domains.
list-redirects           Lists web redirects and aliases in some domain
list-shared-addresses    Lists shared IP addresses for virtual servers
list-styles              Lists all configuration templates
lookup-domain-daemon     A background process for looking up user details.
modify-php-ini           Changes PHP variables for some or all domains.
run-all-webalizer        Run Webalizer reports for all virtual servers
search-maillogs          Searches mail server and procmail logs
set-spam                 Change the spam and virus scanners for all domains
test-imap                Checks if IMAP login to some server works.
test-pop3                Checks if POP3 login to some server works.
test-smtp                Checks if the mail server can RCPT to some address

Sorry for the slower than usual update this time. I was rather busy/sidetracked and left it for a day or two before doing it. Since it wasn’t a exploit fix release i figured it wasn’t a major.

Anyway, here goes the new wordpress mass upgrade script. I no longer have a wordpress MU available to test on, it should technically work, but let me know if there are any problems at all with it.

This script will search /var/www (changeable in a variable) for any wordpress install and make sure its upgraded to the latest version. It pays to check each site after the upgrade to make sure plugins all worked (and upgrade plugins before the upgrade works also!). This version has a database change so i have forced the $sitename/upgrade.php call with a wget to save you some time. This occasionally fails, for no reason i can find, so it pays to check this.

Let me know if you have any bugs at all, or any problems.

Here’s your script to upgrade them all.

wget http://b.ri.mu/files/wordpress-upgrade-3.0.sh 
sh wordpress-upgrade-3.0.sh

Next time I may endeavour to see if i can script putting sites into maintaince mode, disable plugins, upgrade etc

Want some more memory on your RimuHosting VPS? Guess what? It is a RimuHosting free memory upgrade day!

Here is how it works…

Update 2010-06-17: Our first run of the free memory upgrade promotion was a huge success.  We were rushed off our feet for 6+ hours doing upgrades.    Plus who would have thought giving away stuff could be so much fun?

Update #2 2010-06-28: Yep, we’re doing it again.  Quiet Monday at the NZ office.  So we will are re-running the free memory upgrades.  Offer open for the next 6 hours… (And 6+ hours later we are now done.  If you missed don’t worry.  Follow us on twitter and get the upgrade next time).

To get your memory upgrade:

• Pop in a support ticket.
• Say: Would you please add some to all my VPSs?  And do that for free?
• We will grab the ticket.
• We will add some more memory to your server.
• That will require a VPS restart (we will work to do the upgrade within an hour or hopefully a lot less) of getting your ticket.
• And then we will email you and let you know it is done.

The fine print:

• If the host server you are on doesn’t have any available memory, we won’t be able to add anything extra.  Sorry.  (But we can talk about moving your VPS to a host with more memory, no problem).
• The amount of memory increase will depend on a few things.  Your host server memory size, number of VPSs on your host, and the amount of caffeine in the RimuHosting sysadmin handling the request.
• One free memory upgrade request per customer for the time being
• Does not apply to dedicated servers (if only we could add memory there with the push of a button…)
• Does not apply to semi-dedicated / low-contention servers (since those guys have fixed vs. variable memory sizes).
• This offer is valid _only_ for the next 6 hours.  So get in quick!
• We’d discourage ‘double dipping’ (requesting multiple free upgrades).

Today we had a customer who had an interesting problem. They were getting the message ‘disk is full’ despite having plenty of free space. Luckily for him, my first thought was ‘inodes?’
I logged in and checked his inode usage

root@askdev:# df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/xvda1 525312 524844 468 100% /
varrun 65579 27 65552 1% /var/run
varlock 65579 2 65577 1% /var/lock
udev 65579 2696 62883 5% /dev
devshm 65579 1 65578 1% /dev/shm

This shows that all the inodes on the disk itself are full.
I used the following script to determine where the inode usage was most

root@askdev:/# for i in `ls -1A`; do echo “`find $i | sort -u | wc -l` $i”; done | sort -rn | head -5
468388 var
49844 usr
18741 proc
5187 sys
5026 root

I tracked it down to /var/lib/php5/ and all the session files in there.

I used a find to then find any that were older than 10 days
root@askdev:/var/lib/php5# find ./ -type f -mtime +10 | wc -l
111041

High inode usage is usually caused by a massive number of small files. In this case the session files are normally stored somewhere temporary and removed when not in use. Either there could have been a bug in the code not removing them or it was a higher traffic website.

You can delete files older than 10 days if you want with the following command
cd /dir/of/inodes
find ./ -type f -mtime +10 | xargs rm