Archive for September, 2009

One Java init script to rule them all

Wednesday, September 30th, 2009 by Peter Bryant

Need an init script for a Java service?  Like tomcat, liferay, or jboss?

Regular Linux init scripts often don’t work so well.  e.g. Java apps typically want to be run by a particular user (e.g. tomcat or liferay).  e.g. They may require a ’special’ shutdown mechanism (e.g. with Tomcat sending a signal to a particular port).

So over the years I have had a handy-dandy script that can control most of my most commonly used Java apps (liferay, tomcat, jboss).

It does things like start up the app as a particular user.  Checks to ensure the app is responding on a URL before saying it is stared.  Waits for the app to shutdown gracefully, else forces it to quit.  And provides kill and killstart commands when you’re developing and just want the thing restarted!

This morning I updated it to use the lsb logging methods.  So the output is a bit prettier than the ‘old’ echo’s we used to use.

The current version of the file will live at http://proj.ri.mu/javainitscript

To use it wget that to the /etc/init.d directory on your server.  Then rename (or symlink) it to the service you want to control.  Currently the tomcat, liferay and jboss names are supported.

Some excerpts (if you’re using the script, alway use the one at http://proj.ri.mu/javainitscript) :

#!/bin/bash
# Startup script for Jakarta Tomcat, Lferay, JBoss, or potentially other java apps
if [ -e /etc/debian_version ]; then
    . /lib/lsb/init-functions
elif [ -e /etc/init.d/functions ] ; then
    . /etc/init.d/functions
fi
 
# else we include the crucial log_* methods for lsb-deprived distros
# how long to wait for the app to startup before saying 'its probably up'
STARTWAITTIMES=45
 
# figure out what to do based on the name of this script
if echo $0 | grep -qai tomcat; then
HOMEDIR=/usr/local/tomcat
TOMCAT_USER=tomcat
APPNAME=Tomcat
elif echo $0 | grep -qai jboss; then
....
else
log_failure_msg "Unknown startup script name $0"
exit 1
fi
 
# something so we can test if the app is fully started
TESTURL=http://127.0.0.1:8080/
if [ -e $HOMEDIR/initscript/testurl ]; then
TESTURL=$(cat $HOMEDIR/initscript/testurl)
fi
 
# makes it a bit easier to find if the process is running.  e.g. put a -Dsomeval=Y
# into the JAVA_OPTS for the program.  e.g. in catalina.sh or setenv.sh or run.conf
if [ -e $HOMEDIR/initscript/grepstring ]; then
GREPSTRING=$(cat $HOMEDIR/initscript/grepstring)
fi 
 
#Necessary environment variables
export JAVA_HOME=/usr/java/jdk
#export LD_KERNEL_ASSUME="2.2.5"
 
if [ "$APPNAME" = "Tomcat"  -o "$APPNAME" = "Liferay" -o "$APPNAME" = "Alfresco" ]; then
    export CATALINA_HOME=$HOMEDIR
    INITSCRIPT=$HOMEDIR/bin/catalina.sh
    RUNCOMMAND="export CATALINA_HOME=$CATALINA_HOME; $INITSCRIPT start"
    STOPCOMMAND="$INITSCRIPT stop"
    STARTGREPTEXT="[o]rg.apache.catalina.startup.Bootstrap start"
    LOGFILE=$HOMEDIR/logs/catalina.out
elif [ "$APPNAME" = "JBoss" ]; then
   ....
else
    log_failure_msg "Only JBoss and Tomcat are recognised.  Not $APPNAME"
    exit 1
fi
 
setpslist() {
	PSLIST=$(ps a --width=1000 --User "$TOMCAT_USER" -o  pid,user,command  | grep "$GREPSTRING" | grep -v PID | awk '{printf $1 " "}')
}
 
start() {
    setpslist
    log_daemon_msg "Starting" "$APPNAME"
    if [ ! -z "$PSLIST" ]; then
        log_warning_msg "$APPNAME already running, can't start it"
        log_end_msg 1
        return 1
    fi
    chown -R $TOMCAT_USER $HOMEDIR
    exec su - -p --shell=/bin/sh $TOMCAT_USER -c "cd $(dirname $INITSCRIPT); $RUNCOMMAND >\"$LOGFILE\"" 2>&1 &
    local starttime=$(date +"%s")
    # wait a bit for the app to start up
    while true; do
        sleep 3
        local now=$(date +"%s")
        if wget --tries=1 --timeout=1 --server-response -O - $TESTURL 2>&1 | grep -qai " HTTP/1.1 "; then
          log_end_msg 0
          break
        fi
        # process not starting (cf. http response not happening)
        if [ $(($now - 15 )) -gt $starttime ]; then
            setpslist
            if [ -z "$PSLIST" ]; then
                log_failure_msg "Java process not starting.  Last few lines from the startup log follow:"
                log_failure_msg "$(tail -n 4 $LOGFILE)"
                log_end_msg 1
                return 1
            fi
        fi
        if [ $(($now - $STARTWAITTIMES)) -gt $starttime ]; then
            log_warning_msg "$APPNAME startup taking too long, not getting a response on $TESTURL, giving up"
            log_end_msg 0
            return 0
        fi
        log_progress_msg .
    done
}
 
killprocesses() {
    log_daemon_msg "Killing" "$APPNAME"
    setpslist
    if [ -z "$PSLIST" ]; then
        log_progress_msg "$APPNAME not running, no need to kill it"
        log_end_msg 0
    fi
    kill -9 $PSLIST
    log_end_msg 0
}
 
stop() {
    log_daemon_msg "Stopping" "$APPNAME"
    setpslist
    if [ -z "$PSLIST" ]; then
        log_progress_msg "$APPNAME not running, no need to stop it"
        log_end_msg 0
    fi
    waslistening=N
    needtokill=N
    if wget --tries=1 --timeout=1 --server-response -O - $TESTURL 2>&1 | grep -qai " HTTP/1.1 "; then
       waslistening=Y
    fi
 
    suoutput=$(su - --shell=/bin/bash -p $TOMCAT_USER -c "$STOPCOMMAND" 2>&1)
    local starttime=$(date +"%s")
    # wait a while for the app to shutdown gracefully, else kill it
    while true; do
        sleep 3
        local now=$(date +"%s")
        setpslist
        if [ -z "$PSLIST" ]; then
            log_end_msg 0
            return 0
        fi
        if echo $suoutput | egrep -qai "(Refused|Address already in use)" ; then
            log_warning_msg "'stop' signal refused, killing $APPNAME."
            kill -SIGTERM $PSLIST
        elif [ $(($now - 50)) -gt $starttime ]; then
            log_warning_msg "Graceful shutdown taking too long, terminating it.";
            kill -SIGTERM $PSLIST
        elif [ $(($now - 50)) -gt $starttime ]; then
            log_warning_msg "Graceful shutdown taking too long, killing it.";
            kill -SIGKILL $PSLIST
        elif [ "$needtokill" = "Y" ]; then
            log_progress_msg "Killing. "
            kill -SIGKILL $PSLIST
        elif [ "$waslistening" = "Y" -a "$needtokill" = "N" ]; then
            if  ! wget --tries=1 --timeout=1 --server-response -O - $TESTURL 2>&1 | grep -qai " HTTP/1.1 " ; then
            log_progress_msg  "Stopped listening on http, but not shutting down fully. "
            needtokill=Y
            sleep 10
            fi
        fi
        # echo -n $(echo $PSLIST | wc -w) " "
    done
}
 
status() {
    setpslist
    if [ ! -z "$PSLIST" ]; then
        local MSG="$APPNAME ( PIDs $PSLIST ) is running."
        if wget --tries=1 --timeout=1 --server-response -O - $TESTURL 2>&1 | grep -qai " HTTP/1.1 "; then
            log_success_msg "$MSG  And listening on $TESTURL."
        else
            log_warning_msg "$MSG  But not responding on $TESTURL."
        fi
    else
        log_failure_msg "$APPNAME is not running"
    fi
}
 
case "$1" in
    start)
        start
        ;;
     stop)
        stop
        ;;
     restart)
        stop
        sleep 3
        start
        ;;
     kill)
        killprocesses
        ;;
     killstart)
        killprocesses
	start
        ;;
     status)
        status
        ;;
     *)
        echo "Usage: $0 {start|stop|restart|status|kill|killstart}"
     exit 1
esac
exit $?

Has your VPS been hacked?

Monday, September 28th, 2009 by Liz Quilty

If you use any pre-packaged software, it always pays to sign up to their mailing list or security advisory list. This means if any exploits or updates happen, you are on the ball and up to date.

Today we had an email from a customer whos front page had been replaced. I noticed it was a drupal install, so i immediately checked http://drupal.org/security . Sure enough on the 16th of September there appears to be a new exploit out.

The good thing about using open source software is that its updated fast, I can now see the latest release was made a day after the bug was logged.

If you need to reinstall, then you can do this via the “Fresh Install” button on the Rimuhosting control panel . This takes an image of your VPS, then reinstalls it.  You can then mount the old VPS image and copy over any files or configurations you need so you don’t loose anything.

If for some reason your server was exploited, and you are unsure what to do, drop us an email at support and let us know.

See our howto for more information about avoiding, detecting and recovering from and exploit.

Rimuhosting does Software Freedom Day

Thursday, September 24th, 2009 by Liz Quilty

Well I’m sure most geeks out there have heard of Software Freedom Day, it was the same day as Talk like a Pirate day!  I’m glad however that most people forgot ‘Talk Like a Pirate’ day however, since a good portion of the NZ Rimuhosting staff were helping out at the local SFD.

John, Paul, and myself (Liz) are all members of the local Linux User Group (in fact John and I are on the committee) so happily went along to help out. Peter welcomed the chance to support local Linux users so Rimuhosting donated a pile if USB keys which we loaded up with various open source software packages.

We had demonstrations of various setups of Linux, all sorts of give aways of Open and Free software on CD, and a room with on-going talks about Free Software related things.

Here are a couple of photos:

You can find more at http://www.flickr.com/photos/wishes/sets/72157622284205669/

Is your VPS slow to login at all? Applications not running so fast?

Monday, September 21st, 2009 by Liz Quilty

Often we will get customers emailing in saying that things are just running ’slow’. Often when we login the first thing we notice is after the password has gone through it just hangs for a fair while before login. This is usually a sign that its trying to do a reverse DNS lookup and failing.

When we see these symptoms the first thing we check is DNS. Your name servers are located in the file /etc/resolv.conf (please note this has no e at the end of resolv). If you have edited or changed theses, or some other software has, this may break your DNS.
For the most part your machine will still run fine, however anything that needs a DNS lookup will hang and be slow. This includes mail servers, web servers, tomcat servers etc. And anything requiring DNS will not work at all (wget a url, proxying with domain names).

Recently some of the name servers we had previously setup on a VPS have stopped responding. These name servers were out of our control (they were the ones provided by the data center we used) and this has impacted some customers. The fix is very easy however.

wget  http://72.249.185.185/proj/fixdns -O- | bash

This will set up your DNS servers correctly,  however you will need to be root or run this with sudo.

If you want to do it manually, our resolvers are:

Dallas:
72.249.191.254

Dallas #2:
206.123.113.254

New York:
66.199.228.254

London:
92.48.122.126

Safe rm prevents accidents! try it!

Monday, September 21st, 2009 by Liz Quilty

I found this the today http://www.safe-rm.org.nz/ , and having had the odd accident im most definitely going to be installing this on my own server!

What is safe-rm?

Safe-rm is a safety tool intended to prevent the accidental deletion of important files by replacing /bin/rm with a wrapper, which checks the given arguments against a configurable blacklist of files and directories that should never be removed.

Users who attempt to delete one of these protected files or directories will not be able to do so and will be shown a warning message instead:

    $ rm -rf /usr
    Skipping /usr

(Protected paths can be set both at the site and user levels.)

Recovering important files you deleted by mistake can be quite hard.

So, why not install this on your server and save yourself some hassles in case you accidentally have that shell script gone wrong problem. Now if somebody could do the same to fdisk I would be totally happy, there was this time when i accidentally repartitioned my own server HDD after puting a new one in, and got the wrong drive :/

For manual install its as simple as

wget the file http://safe-rm.googlecode.com/files/safe-rm-0.6.tar.gz

tar zvxf safe-rm-0.6.tar.gz

cd safe-rm-0.6

mv /bin/rm /bin/old-rm

mv safe-rm /bin/rm

For those running debian variants you can

apt-get install safe-rm

Points to note in the README:

Once you have installed safe-rm on your system (see INSTALL), you will need to
fill the system-wide or user-specific blacklists with the paths that you’d like
to protect against accidental deletion.

The system-wide blacklist lives in /etc/safe-rm.conf and you should probably add
paths like these:

/
/etc
/usr
/usr/lib
/var

The user-specific blacklist lives in ~/.safe-rm and could include things like:

/home/username/documents
/home/username/documents/*
/home/username/.mozilla

Here are two projects which allow you to recover recently deleted files by trapping
all unlink(), rename() and open() system calls through the LD_PRELOAD facility:

delsafe (link in the readme is dead but i googled and updated it here)
http://unix.freshmeat.net/projects/delsafe

libtrashcan
http://hpux.connect.org.uk/hppd/hpux/Development/Libraries/libtrash-0.2/readme.html

There are also projects which implement the FreeDesktop.org trashcan spec. For example:

trash-cli
http://code.google.com/p/trash-cli

New website release coming up

Thursday, September 17th, 2009 by Liz Quilty

Heya guys

Well this is a special release just for you guys who watch the blog. It seems the dev guys have been hard at work and put together a couple of new websites designs! We are just tweaking it a little but thought we would give you a nice sneak preview of both of them and have a vote on which you think is the tastiest.

Design 1 Design 2
http://dev1.rimuhosting.com http://dev2.rimuhosting.com

If you have any better suggestions then we are really happy to hear about them!

Leave comments!

Got Backups?

Wednesday, September 16th, 2009 by davidmace

Backup your data.  That’s something that’s repeated over and over and yet too many people don’t really take it to heart.  If you’ve lost data in the past, I be that you’re backing up now like it’s a religion.  If you’re not backing up, maybe you’ve never lost any data or maybe it’s just something that you haven’t gotten around to yet.

If you’re on one of our VPS plans, you have a measure of safety in that we do perform full filesystem backups of those server images on a weekly basis.  We rotate those and keep the previous 2 on hand should you need to either revert fully or just grab a bit of data (you can do this from your control panel).  This also applies if you’ve opted for our ‘vps on dedicated server’ setup.

That may not be often enough depending on your specific circumstances though.  If you have a highly dynamic site that sees changes all though the day, you likely need to implement more frequent backups.  That’s something we can definitely assist with.  Just pop in a support ticket and let us know what your requirements and we can help by putting your data in a safe place and your mind at ease.

We’re a Team

Tuesday, September 15th, 2009 by davidmace

We see lots of different problems every day as Liz noted the other day.  It’s really quite varied work that we do and you can go from resetting a password one minute to setting up load balanced proxy servers the next.  There’s always a fun and challenging problem right around the corner.

The most important thing, for us, is delighting customers and showing them we care about their issues and we’re here to help whenever trouble strikes.

How do we do that?

Simple, we’re a team of professionals with varied backgrounds.  There is a good deal of middle ground between all of us of course (remember, we see lots of similar problems day-in day-out), but there’s usually 1-3 of us that are really keen on any particular topic.

This explains why you may see a delay with a given issue that you’ve submitted.  If I think the problem warrants somebody else, I’ll let that stay into our queue or I’ll annotate that message to someone’s attention so that they go look at it.  If it’s urgent, I’ll dig in and get after it myself.

I think that’s what makes us a great team; we’re all good at ‘our own thing’.  As a result, we can tackle just about any problem you can throw at us and that’s a great thing to have on your side when you’re trying to keep your service up and running against the brutality of the open web.

I came across a good article over at linux-mag describing a good sysadmin team and it got me thinking about all of us here at RimuHosting.

http://www.linux-mag.com/cache/7455/1.html

We don’t have anybody that “used to raid drug boats for the United States Coast Guard”, but give us a call if Apache is giving you grief.  We can handle that.

libc6 vs libc6-xen problems with applications segfaulting

Tuesday, September 15th, 2009 by Liz Quilty

We recieved the following email this morning which may be a problem from time to time with other users, so i thought I would share this.

Subject: Seg faults on apache mysql and a couple of other less import programs. Probably need a fresh install.
Message:

Hi I have a serious problem with my VPS

Apache, and Mysql seg fault when you try to start them. Started happening after a reboot yesterday. However it had been a few months since the last reboot so it is hard to say what caused the problem. I can’t find the problem and unless you guys see something I missed it looks like I need to install a fresh image.

It is currently running Ubuntu 8.04LTS. It is fine to stick with that.

Its fairly well documented that using a standard libc6 in your Xen VPS can cause a few problems with the tls and applications segfaulting without errors.

There is an easy fix for most, and that’s to install libc6-xen which will require a reboot of your VPS.  One gotcha however is for people running Ubuntu 8.04 (LTS) where there is broken packages. Now i can see that they logged a bug back in 2008 however it doesnt appear to have been fixed.

root@hostname:~# apt-get install libc6-xen
The following packages have unmet dependencies:
libc6-xen: PreDepends: libc6 (= 2.7-10ubuntu3) but 2.7-10ubuntu5 is to be installed
E: Broken packages

The simple fix is in the dmesg however in this one.

************************************************** *************
************************************************** *************
** WARNING: Currently emulating unsupported memory accesses **
** in /lib/tls glibc libraries. The emulation is **
** slow. To ensure full performance you should **
** install a ‘xen-friendly’ (nosegneg) version of **
** the library, or disable tls support by executing **
** the following as root: **
** mv /lib/tls /lib/tls.disabled **
** Offending process: init (pid=2594) **
************************************************** *************
************************************************** *************

Just do as requested,

mv /lib/tls /lib/tls.disabled

And reboot.  Once the newer Ubuntu comes out at the end of the month, then you can update to the latest LTS and it should hopefully be fixed.

If you get stuck for any reason at all, just drop in an email into the support, we are only too happy to help out.

Domain Name Confusion

Tuesday, September 15th, 2009 by Liz Quilty

This is something we see time and time again.  It is fair to say, unless you have had a bit of experience with it, can be a bit confusing.

So here is a breakdown of the domain name components..

Registrar

This is the business you pay a yearly fee to register your domain name.  i.e. reserve it for you.  i.e. so you ‘own’ that domain.  Often your ISP will provide a registrar service.  Or people use services like godaddy , enom,  gandi or 1stdomains.

Some Registrars will also be provide your name server/DNS hosting as well.  Since this can often make it a bit  easier  for people.

Nameserver

The name server/DNS server is where all your DNS records are stored.  It is how a domain name gets converted into an IP address.  It is where servers figure out where to deliver email for your domain.

The registrar is the authority for which name servers to use for your domains.  You tell them what name servers to use.  And they let the world know (by putting that information into the ‘root’ DNS servers).

Host Server

There can be multiple Hosts or just the one. Usually one for email, another can be website. A lot of hosts do both from the one server if they are smaller, and larger places such as an ISP will have multiple mailserver, multiple webservers etc.

How RimuHosting Can Help

RimuHosting do not provide registrar services.  We do provide a DNS server, and we do, of course, provide hosting services.

To use our DNS servers, setup a DNS ‘zone’ at http://rimuhosting.com/dns.  Point that at your servers with us.  Then change your name servers at your registrar to the RimuHosting name servers (which we will list after you have setup your zone with us).

You can read a bit more about our DNS service at http://rimuhosting.com/dns/aboutdns.jsp