Monit is brilliant for monitoring and restarting services when they are down, it can alert you or just restart after 5 failed connects/attempts. It handles everything from disk space, to memory, services, and other things. Its very easy to configure and very customizable.
To install ..
apt-get install monit
Instructions are here http://www.howtoforge.com/server-monitoring-with-munin-and-monit-on-centos-5.2 . Basically use rpmforge. and install from there.
Do your basic config, and anything else you may need/want.
Now for the tomcat part - this is based on tomcat being in /usr/local/tomcat where our typical setup script puts everything.
Tomcat, method a (recomended):
Simply run the below snippet to enable monit monitoring of tomcat. This method requires the least work and changes to configurations. Typically monit prefers a bid file to monitor a service as described in method b, but this way works just as well so long as the http port connector is enabled.
echo "check host tomcat with address localhost stop program = "/etc/init.d/tomcat stop" start program = "/etc/init.d/tomcat restart" if failed port 8080 and protocol http then start " > /etc/monit.d/tomcat /etc/init.d/monit restart
Tomcat, method b:
Use this method if you dont have a suitable http connector enabled for your tomcat instance, but be aware that pid files can be left in an inconsistent state in some cases. Which may then require manual intervention anyway. Add a tomcat instance into your config for monit that looks like this (change gid/uid tomcat runs )
check process tomcat with pidfile "/var/run/tomcat/tomcat.pid" start program = "/usr/local/tomcat/bin/startup.sh" as uid tomcat gid tomcat stop program = "/usr/local/tomcat/bin/shutdown.sh" as uid tomcat gid tomcat if failed port 8080 then alert if failed port 8080 for 5 cycles then restart
Then edit your catalina.sh and set
CATALINA_PID to be /var/run/tomcat/tomcat.pid JAVA_HOME=/usr/java/jdk
Add in at the top
Then of course create the pid directory
mkdir /var/run/tomcat/ chown tomcat.tomcat /var/run/tomcat/
Add this to your my.cnf under [mysqld]
and this to your monit config
check process mysql with pidfile /var/run/mysqld/mysqld.pid group database start program = "/etc/init.d/mysql start" stop program = "/etc/init.d/mysql stop" if failed host 127.0.0.1 port 3306 protocol mysql then restart if 5 restarts within 5 cycles then timeout depends on mysql_rc check file mysql_rc with path /etc/init.d/mysql group database if failed checksum then unmonitor if failed permission 755 then unmonitor if failed uid root then unmonitor if failed gid root then unmonitor
Add the following to your monit config
check filesystem with path /dev/xvda1 if space usage > 95% then alert if inode usage > 95% then alert
Add this to your monit config (change the port if yours is different)
check process sshd with pidfile /var/run/sshd.pid start program "/etc/init.d/sshd start" stop program "/etc/init.d/sshd stop" if failed port 22 protocol ssh then restart if 5 restarts within 5 cycles then timeout
Now it should all work once you have reloaded your monit config.
For more examples check out http://mmonit.com/wiki/Monit/ConfigurationExamples