Every now and then there is a security issue that has the potential to impact a large number of customers.
RimuHosting has created a server patching tool that automates fixing or mitigating a number of these issues for its customers.
- Automated
- Schedule-able
- Web based
- Permits opt-out per issue
- Works across different Linux distros
Creating this unique service, and updating it for each issue that arises, takes a substantial effort. However, it is something we are quite proud of and in line with our responsibilities to: protect Internet users from exploited servers; provide our customers a secure hosting environment; and to let our systems administration live by their personal and professional ethics.
A tour of the tool
You can access the patch tool at https://rimuhosting.com/cp/serverpatchstate.jsp
Enabling access. For most issues we would need to log into your server to assess if it is vulnerable. If we cannot, then the patch tool will let you know which servers we had an issue with. If you then wish, you can enable our access per the information at https://rimuhosting.com/knowledgebase/rimuhosting/rimuhosting-ssh-access
The page will also link to a notice with more details about the issue. Which should also include a description of how we will be attempting to address the issue.
Disabling patching. If you don’t wish us to patch a server, or check it further for vulnerability to a particular issue, you can disable the tool (which is a per issue setting).
Scheduling patches. This page will also let you set a time for the tool to auto patch your server. This is a handy way to update your fleet of servers at a time that suits you.
If you have more than one server with us we will summarise the patch status for each server.
The excerpt above shows one server where we were unable to login (and therefore, for this issue, we were unable to determine the vulnerability state). The other server was not vulnerable (e.g. due to already being patched by the user, or not being affected by the issue – e.g. not having a particular package installed).
You can click through into an individual server. And that will provide you more details about its state.
Interpreting results. In this example the server was not vulnerable when we checked:
In the following example, the server was vulnerable, and was patched. In this case it let’s us know the issue was resolved via a yum upgrade of the package.
In this following case the server was vulnerable. And we have not attempted to patch it (yet).
On an individual server you can attempt to re-check for a vulnerability. For example you may wish to fix the issue yourself, and have the tool just do the check.
Running the patch. You can also run a patch/fix for the issue. That runs the fix straight away (rather than waiting for the scheduled patch time). And after that you can see whether the server was able to be successfully patched.
How the tool works
A brief outline:
- Log into server
- Collect information. Typically via a script. For example, the script we run to check (and patch) the polkit issue. These scripts try to collect information about the issue. For example, for the polkit issue we try to collect the package version installed, if the exploitable file is present (and when it was last updated), and if the setuid bit is set on that file (which is one way this particular issue can be mitigated).
- Assess vulnerability. After collecting the information our main website processes the information. e.g. checking package versions against ones that distro makers have declared as fixed. We then update the status of the issue for each server. Collecting information about if we were able to log in, the assessed vulnerability.
- Attempt patch. Typically this is done by updating packages.
- Present the status. To give customers an indication of their vulnerability.
Older servers
Since RimuHosting have been providing VMs since 2002 we have a few customers on the same distro for so long that it is no longer receiving security updates.
It would be easy to throw up our hands and say ‘not supported’ and leave customers to the consequence of having an insecure system. However, that isn’t always in the best interest of the customer – who may wish to be on a newer distro, but could be stuck there due to application dependencies, or being unable to upgrade/migrate in a timely fashion.
RimuHosting adopts a wabi-sabi philosophy. Where we accept transience and imperfection. We try to make the best of the situation.
When customers are ready to upgrade distros they have a few options.
- Reinstall the server (with a newer distro), mounting the old disk image, and migrating their data to the new setup. Server will be down until things are all migrated.
- Setup a new server and migrate from one server to another. Services can be up while things are being switched over.
- Try an upgrade in place. We have created a ‘distrorejuve’ script that does a good job in many cases of bringing Ubuntu and Debian distros from as old as 2005-era distros up to the latest version.
Customers are welcome to run these steps themselves, else our support team can perform assist.
We even provide a distro upgrade as a service option for customers running Linux servers with other providers.
Here are old things:
https://poets.org/poem/mending
Fraying edges,
Ravelling threads;
Felled edges,
An expectant thimble,
New thread through an eye;
Here are patches,
Darned threads,
Strengthening old utility,
Pending the coming of the new.
Yes, I have been mending