BOFH Hunter

Spacewalk’s first steps

A few days ago RedHat announced that they had open-sourced their satellite product under the moniker of Spacewalk, and I’ve taken a few days to play around with it and get some first impressions of what’s been put out. I do not by any means claim to be an expert on the RHN satellite from whence this came, or the current spacewalk incarnation. This is simply meant to be a linux enthusiast’s first look at the product they’ve put out.

The directions to install spacewalk are very clear and relatively simple to follow. It only took about 10 minutes to get it up and running. From there, it’s a whole different story.

Spacewalk requires its own machine

The directions don’t say this, and it’s not really 100% true, but for nearly all real-world cases, it’s much simpler just to give it a box. There are two basic reasons for this. First up is that spacewalk drops a number of packages on top of other things you may be running, like apache. The spacewalk server setup drops in a ’satellite-httpd’ process instead of using the distro provided httpd package. Since RHN satellite was/is a boxed solution, this fact can be overlooked as I figure it’s probably something that will change as the project matures and gains popularity. The second issue with spacewalk is storage, which is primarily an organization based issue. Sure it’s going to take a few gigs of disk space to mirror your favorite distribution, updates and any associated 3rd party repositories that you might want. However:

Channels cannot cross organisations

This one kind of surprised me considering that RHN seems to do this just fine, though it’s probably due to a different back end.  To illustrate this point a little, lets assume that we’re running the Spacewalk server for a university.  The IT department has their own organization for the university infrastructure, with a CentOS5 channel for base, a child channel for updates, and another child for Extras. A fairly boring example to be sure, but a good foundation to work from. The CS department runs CentOS for this systems as well, using it for both instruction, and the servers related to instruction. They have require the exact same channels the IT department uses, but Spacewalk currently requires them to duplicate the entire tree; Base, Updates, Extras, all of it.  If you expand this out for a few more organizations, and figure 20G or so per channel for the life of the distribution, you’re very easily looking at a few hundred gigs of storage. And while you’re busy pushing these packages to the Spacewalk server, you’ll be doing so manually.

Syncing Repositories

Part of the RHN satellite feature was that it would sync with redhat’s RHN proper, and then you could move out with your updates locally. The old RHN satellite would pull from RedHat’s RHN proper, and then you could manage your machines locally. With Spacewalk, the RHN sync capability was removed, and no base for syncing to other repositories (via yum, rsync or otherwise) currently exists. If you want to keep spacewalk updated with the latest and greatest for your distribution, you’ll have to script something up yourself. The methods for doing so are not difficult, and anyone with a basic grasp of shell scripting should be able to pull this off.

The good stuff

I really don’t want this to seem like I’m simply complaining about Spacewalk as it is a very good product, and RedHat did a good thing by releasing it. It simply has a bit of growing to do as it begins life as an open source project. There’s already a rather vibrant community springing up around it, both as a mailing list and in irc on freenode. Additionally, Spacewalk provides functional centralized management of multiple boxes across different distributions which is indeed quite useful.

If you’re in the market for centralized system management and you have a box with storage to spare, then I would highly recommend folks take a look at Spacewalk. It is definitely a project to keep an eye on as it matures.

3ware performance in CentOS

There is an upstream bug report here, which may be of interest to folks using 3ware and aacraid raid cards with CentOS or RHEL. The basics of the bug hinge on setting MWI, which can have a pretty hefty impact on IO performance. If you’re storing your MySQL bits on a 3ware powered array, it’s a safe bet that this fix may help improve your performance and reduce some of the IO wait seen on the system.

The downside with this is that even though the fix is known, Red Hat is sticking to their procedure, and has stated that they will not release the fix for this in the main kernel until 5.3 is released. Since 5.2 is fresh from the factory, it’s not likely that we’ll be seeing this fix pushed mainstream in the next few months.

This leaves a few choices for the RHEL and CentOS communities for how to proceed.

1. Weigh in with your opinion on this bug report. If enough people respond, RH will likely appease them.
2. Help test the patched kernels in the bug report. The more comfortable RH is with the patch, the more likely it is that they’ll tuck it in with a bug fix or security update. See #1.
3. Give CentOS a chance to get 5.2 out the door. Once it’s released, folks will have some time to roll up a kernel repository at http://people.centos.org similar to what was done with bz321111. Since this bug is strictly performance affecting, installs won’t be an issue, and you can update to the modified kernel, or use the stock release as you see fit.

In theory, you could also roll your own kernel with this patch if you didn’t want to wait, however if you do this, you’re accepting responsibility for building it properly and tracking all the kernel security and bug updates until the patch becomes mainstream. I wouldn’t recommend this method since it requires more time and upkeep, but for folks who roll their own it provides another alternative.

gconf voodoo

The gnome desktop has tons of versatility and flexibility to suit just about any desktop type needs. Unfortunately this flexibility has a hidden cost, and a few dozen hidden options. While most options are right where you’d expect them to be in the various gnome applications like nautilus, others can be difficult to nail down. This is where gconf was supposed to come to our rescue, but instead it got all drunk and confused.

While many folks compare it to the Windows registry, this isn’t entirely accurate. GConf is a bit more user friendly than that, although some similarities can be drawn. It’s a binary set of files which requires a special utility to work, and operates in a directory/file structure type. Below, we’ll go through some of the more common changes users are likely to want.

The Basics:

There are two ways to go about playing around in your GConf registry. You can use the gconf-editor gui, which is in the gconf-editor package, or you can use gconftool-2, which is a command line driven application, and a little more cumbersome to maneuver around in. The basic command to help get you around in gconftool-2 is ‘gconftool-2 -R /’. With this command, you’ll see the directory/file structure which makes up the registry, and their associated settings. If you’re new to gconf, it’s probably best to start out with the gui.

Starting small:

A few times a month or so, users will ask how to tell gnome or nautilus to ignore blank CD input, or to at least do something useful with it, like open k3b instead of the default nautilus burn window. Setting this up with gconf-editor is relatively simple. Open it up, and browse to the ‘/desktop/gnome/volume_manager/’ directory. Inside this directory you’ll find a number of settings that you can modify for various automated media handling. You can change the default movie player, dvd player and more from this directory. Incidentally, ‘automount_drives’ and ‘automount_media’ are located here also, so if you’re having trouble with usb drives, this is one thing to check. The two options that we’re concerned with right now are ‘autoburn_data_cd_command’ and ‘autoburn_audiio_cd_command’.

By default these are both set to ‘nautilus –nodesktop burn:‘, but this isn’t the behavior we want. If you’d like to have k3b loaded up instead, simply change the string values to ‘k3b’ and you’re off and running. This is a per user setting, so you don’t have to be root to modify most of these values. If you do happen to launch the app with sudo, you’ll also have the ability to enforce this setting for all users. This can get handy, and we’ll look at it a bit later on.

System Policies

Now that you’ve had a little bit to look at the user side of gconf, if you’re planning to run lab or kiosk systems you might also want to look at enforcing some of your system policies with gconf. This should not be your only security method to lock the boxes down, simply another layer to examine for inclusion.

Inside the /desktop/gnome/lockdown directory, you’ll find several settings which can help you restrict your workstations or kiosk systems, such as ‘disable_command_line’. After launching gconf-editor with sudo, set these options the way that you want, then right click them and choose ‘Set as Mandatory‘. This will enforce these changes for all system users, and disallow the user from changing the settings individually for their accounts. This can be done for many of the settings here, including application specific options. While this means of enforcement is not perfect, it can go a long way toward helping an admin regain some control and a possibly a little sanity. It’s also one EVIL BOFH prank for other admins/users… if one were so inclined… >:-)

The kernel collection

If you maintain a number of older RHEL or CentOS 3 and 4 machines, you’ve probably got a few extra kernels lying around to clutter up your /boot partition. In some instances this can cause update issues, and I ran into one such case today. An admin came to me asking why yum was attempting to install all of his packages to his /boot partition, and when I examined further, I saw this on his screen:

installing package kdegraphics-3.3.1-9.el4_6 needs 3MB on the /boot filesystem

While the error itself does look at first glance as the admin described it, this is not the case. The culprit was a kernel update further up the screen, and a 98% full /boot partition, with around 25 spare kernels. His system was simply informing him that the transaction would not occur because one of the updates was not going to succeed due to limited disk space.

While the InstallOnlyN plugin for yum will handle this quite nicely for the day to day stuff, once it’s happened it can be a little tricky to resolve. The easiest way is with a script called kernel-prune from Seth Vidal’s duke directory.

By piping the output of this script through xargs, you can remove all the tedium of manually removing packages one or two at a time. For RHEL and CentOS 3, where installonlyn isn’t really an option, this is pretty much the easiest way to periodically purge some unwanted fat from your system. Hope this helps at least 1 of you out there.

Feel free to share your own methods or comments below. I’m sure there are other methods out there so let’s hear them!

CentOS reference guide

SElinux is a phenomenal way to protect your systems, and very few people disagree with this. The biggest complaint I hear is that it’s not user friendly. Most people seem to treat it like a binary system, and either leave it on, or turn it off. There’s very little documentation about the ins and the outs of selinux contexts and the targeted rulesets which ship with RHEL and CentOS. After some discussions with Ralph this morning on IRC, he’s graciously put together a list of the base contexts which ship in the targeted rule, and a brief explanation of what they do. If you want to take a few minutes to look through the granular protection possible through selinux, have a quick read of the new documentation at http://wiki.centos.org/TipsAndTricks/SelinuxBooleans

If you’re on IRC, feel free to stop by freenode’s #centos channel and thank Range for putting this list together.