BOFH Hunter

firmware without the floppy

In recent years I’ve been able to rid myself of floppy drives entirely.  The problem that I run into however is that occasionally my beloved hardware manufacturers still ship firmware or bios updates on a floppy for some insane reason.

Luckily the syslinux people are aware of this vendor oversight, and have a solution. Enter ‘memdisk’. Memdisk allows you to use grub or other bootloader of your choice to boot to your floppy image file, without actually using a floppy.  To do this,  there are a few simple steps you’ll need to perform.  I’ll assume for the sake of argument that your hardware vendor provides you the .img file for the floppy image.

• Create a directory inside /boot called firmware
• Copy the memdisk binary to /boot
• Copy the vendor’s .img file containing your update to /boot/firmware
• Edit grub and add the appropriate entries for memdisk. A sample is below.

title Firmware Update
        root(hd0,0)
        kernel /memdisk
        initrd /firmware/vendor-file.img

The root values for grub will depend on your setup. You’ll have to salt to taste, but you should be able to copy from existing entries in grub. The only limitation I’ve run into with this update style are multi-floppy updates. You can’t really use them with memdisk that I’m aware of. If anyone’s aware of a way to use multiple img files with memdisk, I’ll happily update with new information!

vmware esxi and ssl

For those of you using the free version of VMWare’s ESXi, you may have noticed that it generates a self-signed local cert. If you want to use your own ssl certificates, you can simply follow these quick steps.

1. Enable SSH in ESXi by following the steps outlined here (don’t forget to reboot the server).
2. Generate your ssl cert using openssl, or however else you want to do it. You need a .key and .crt file.
3. Now scp your .key and .crt files to your esxihost and put them in the /etc/vmware/ssl directory
4. Now cd to /etc/sfcb and edit sfcb.cfg. Change the rui.crt and rui.key references to reflect your cert files.
5. run services.sh restart

That’s it. Now when you use VMWare INfrastructure Client to connect, you’ll be using your own custom certificates.

If you want to use the same cert for the web interface (I’m not certain why, since you can’t really do much from it, but hey)  then you’ll need to do the following steps as well.

1. cd to /etc/vmware/hostd/
2. edit config.xml and change the references for rui.key and rui.crt to your own cert names.
3. run services.sh restart again

How far should we take automation?

It’s a given that the world population is growing. It’s also a given that the world economy is currently slowing.  This overall slowing of the economy is triggering job cuts across many levels of industry, and IT is at the heart of many such trimmings. When IT itself is not being cut, the IT professionals are often tasked with finding ways to make things more efficient or to automate various processes so that other positions can be cut. Just how deep should these cuts go, and should IT professionals get any say in the matter? A speaker from Cisco once said that Linux was the lens through which he found the flaws in his networks.  I believe this metaphor can be carried further, and that IT as a whole exposes more about humanity than we realize.  With this in mind, just how far should we carry our automation endeavors?

Is it acceptable to script system management such that 3 admins are needed instead of 4? Should we automate a line of factory jobs to save the company some money, or increase shareholder profits?  Is making a product cheaper an acceptable reason to eliminate jobs?

How much responsibility does IT have in these actions, and how much responsibility should we take for them? Should we seek out ways to trim down the company and take a ‘survival of the fittest’ mentality, or do we seek out a way to preserve the jobs of those around us? Should we question the corporate officers who direct the action, or is our job simply to follow orders and let the company’s leadership decide what’s best?

From automotive plants and wall street offices to the California budget there are examples on both sides of the issue. Where do you stand, and do your actions correspond to your beliefs? I’m genuinely interested in how people feel about this one, so please take a moment and let me know what you think.

Nagios Security Updates

While most shops keep their nagios installs protected, folks with a publicly available nagios instance should update as soon as possible. There’s an interesting pair of security vulnerabilities which admins should be aware of. The first allows for users to submit commands to cmd.cgi that they would not ordinarily have permission to submit. This is basically a priviledge escalation issue and its severity depends on who has access to your nagios instance, and just how disgruntled they are.

The second is the more serious of the issues, and was described best by Andreas Ericsson, a major nagios contributor.  Quoting from Andreas:

Nagios CGI's are vulnerable to a Cross Site Request Forgery attack (csrf).
A CSRF attack requires a couple of things for it to work, and it relies
on the webs abilities (or rather, the browser's abilities) of posting
form-data to a site which is other than that of the site presenting the
form.

Here's how it works:
Unsuspecting Nagios Admin (UNA from now on) logs on to the Nagios server
and checks the status of his/her network. Since everything's ok, UNA
decides to leisurely browse evilsite.com, controlled by Dr Evil.

On evilsite.com, there's a page containing a bog-standard web form, but
with some hidden variables and an 'action' tag that points to UNA's
cmd.cgi on UNA's Nagios server. When UNA submits the form, Dr Evil has
all of a sudden sent data of his/her choice to the responding page
on UNA's site. It's important to note that UNA's browser is being used,
as it leads to a couple of interesting things:
* UNA sees the output from cmd.cgi. It's never sent to evilsite.com,
  which can only guess if the attack was successful or not.
* Firewalls can not be used to defend against this, as UNA requires
  access to the Nagios server in order to work.
* Cookies can't be used either, as they are helpfully sent to the
  Nagios server whenever the browser loads a page from it.

Why is this bad, then? Well, it's not so evil in itself, and the most
horrible thing that it should have lead to was Dr Evil being able to
enable / disable notifications or stuff like that, but in Nagios 3
we gained the ability to change checkcommand arguments and suchlike,
which, combined with the csrf above, ultimately led to Dr Evil being
able to run any command of his/her (who says girl's can't be evil?)
choice on UNA's preacious Nagios server as the Nagios user.

So what's the remedy?
Well, a proper remedy is to implement in-form session tokens, which
makes sure that the form submitted by the user came from the site we
would like it to have come from (namely our humble selves). I'm
working on that right now, and hope to have it done by this afternoon.
It's been loads of fun implementing that in super-paranoid C, by the
way. 

In the mean-time, we've blocked use of the CHANGE_ commands from the
CGI's, and also made sure that multiple commands can't be submitted
as one (fe by using comments with newlines). This interim remedy
brings the worst-case scenario down from remote command execution to
a more prank-like level (acknowledging problems, adding or deleting
comments, etc, etc).

A couple of things to note:
* Information disclosure is not possible. No remote user can see
  anything from your authentication-protected Nagios servers.
* Invalid commands read from the FIFO are always dropped flat by
  Nagios.
* Since commands must be valid, it's not very easy to submit a
  command that has all the information required. Social engineering
  is required.
* You *will* notice if this happens to you, since you all of a
  sudden will end up with cmd.cgi (not in a frame either) saying

  "Command submitted successfully" or some such.

For the full details of this,  you can follow the thread here.  Mostly, If you’re currently using nagios 3, you should update.

Spacewalk’s first steps

A few days ago RedHat announced that they had open-sourced their satellite product under the moniker of Spacewalk, and I’ve taken a few days to play around with it and get some first impressions of what’s been put out. I do not by any means claim to be an expert on the RHN satellite from whence this came, or the current spacewalk incarnation. This is simply meant to be a linux enthusiast’s first look at the product they’ve put out.

The directions to install spacewalk are very clear and relatively simple to follow. It only took about 10 minutes to get it up and running. From there, it’s a whole different story.

Spacewalk requires its own machine

The directions don’t say this, and it’s not really 100% true, but for nearly all real-world cases, it’s much simpler just to give it a box. There are two basic reasons for this. First up is that spacewalk drops a number of packages on top of other things you may be running, like apache. The spacewalk server setup drops in a ‘satellite-httpd’ process instead of using the distro provided httpd package. Since RHN satellite was/is a boxed solution, this fact can be overlooked as I figure it’s probably something that will change as the project matures and gains popularity. The second issue with spacewalk is storage, which is primarily an organization based issue. Sure it’s going to take a few gigs of disk space to mirror your favorite distribution, updates and any associated 3rd party repositories that you might want. However:

Channels cannot cross organisations

This one kind of surprised me considering that RHN seems to do this just fine, though it’s probably due to a different back end.  To illustrate this point a little, lets assume that we’re running the Spacewalk server for a university.  The IT department has their own organization for the university infrastructure, with a CentOS5 channel for base, a child channel for updates, and another child for Extras. A fairly boring example to be sure, but a good foundation to work from. The CS department runs CentOS for this systems as well, using it for both instruction, and the servers related to instruction. They have require the exact same channels the IT department uses, but Spacewalk currently requires them to duplicate the entire tree; Base, Updates, Extras, all of it.  If you expand this out for a few more organizations, and figure 20G or so per channel for the life of the distribution, you’re very easily looking at a few hundred gigs of storage. And while you’re busy pushing these packages to the Spacewalk server, you’ll be doing so manually.

Syncing Repositories

Part of the RHN satellite feature was that it would sync with redhat’s RHN proper, and then you could move out with your updates locally. The old RHN satellite would pull from RedHat’s RHN proper, and then you could manage your machines locally. With Spacewalk, the RHN sync capability was removed, and no base for syncing to other repositories (via yum, rsync or otherwise) currently exists. If you want to keep spacewalk updated with the latest and greatest for your distribution, you’ll have to script something up yourself. The methods for doing so are not difficult, and anyone with a basic grasp of shell scripting should be able to pull this off.

The good stuff

I really don’t want this to seem like I’m simply complaining about Spacewalk as it is a very good product, and RedHat did a good thing by releasing it. It simply has a bit of growing to do as it begins life as an open source project. There’s already a rather vibrant community springing up around it, both as a mailing list and in irc on freenode. Additionally, Spacewalk provides functional centralized management of multiple boxes across different distributions which is indeed quite useful.

If you’re in the market for centralized system management and you have a box with storage to spare, then I would highly recommend folks take a look at Spacewalk. It is definitely a project to keep an eye on as it matures.