The directions to install spacewalk are very clear and relatively simple to follow. It only took about 10 minutes to get it up and running. From there, it’s a whole different story.

Spacewalk requires its own machine

The directions don’t say this, and it’s not really 100% true, but for nearly all real-world cases, it’s much simpler just to give it a box. There are two basic reasons for this. First up is that spacewalk drops a number of packages on top of other things you may be running, like apache. The spacewalk server setup drops in a ’satellite-httpd’ process instead of using the distro provided httpd package. Since RHN satellite was/is a boxed solution, this fact can be overlooked as I figure it’s probably something that will change as the project matures and gains popularity. The second issue with spacewalk is storage, which is primarily an organization based issue. Sure it’s going to take a few gigs of disk space to mirror your favorite distribution, updates and any associated 3rd party repositories that you might want. However:

Channels cannot cross organisations

This one kind of surprised me considering that RHN seems to do this just fine, though it’s probably due to a different back end.  To illustrate this point a little, lets assume that we’re running the Spacewalk server for a university.  The IT department has their own organization for the university infrastructure, with a CentOS5 channel for base, a child channel for updates, and another child for Extras. A fairly boring example to be sure, but a good foundation to work from. The CS department runs CentOS for this systems as well, using it for both instruction, and the servers related to instruction. They have require the exact same channels the IT department uses, but Spacewalk currently requires them to duplicate the entire tree; Base, Updates, Extras, all of it.  If you expand this out for a few more organizations, and figure 20G or so per channel for the life of the distribution, you’re very easily looking at a few hundred gigs of storage. And while you’re busy pushing these packages to the Spacewalk server, you’ll be doing so manually.

Syncing Repositories

Part of the RHN satellite feature was that it would sync with redhat’s RHN proper, and then you could move out with your updates locally. The old RHN satellite would pull from RedHat’s RHN proper, and then you could manage your machines locally. With Spacewalk, the RHN sync capability was removed, and no base for syncing to other repositories (via yum, rsync or otherwise) currently exists. If you want to keep spacewalk updated with the latest and greatest for your distribution, you’ll have to script something up yourself. The methods for doing so are not difficult, and anyone with a basic grasp of shell scripting should be able to pull this off.

The good stuff

I really don’t want this to seem like I’m simply complaining about Spacewalk as it is a very good product, and RedHat did a good thing by releasing it. It simply has a bit of growing to do as it begins life as an open source project. There’s already a rather vibrant community springing up around it, both as a mailing list and in irc on freenode. Additionally, Spacewalk provides functional centralized management of multiple boxes across different distributions which is indeed quite useful.

If you’re in the market for centralized system management and you have a box with storage to spare, then I would highly recommend folks take a look at Spacewalk. It is definitely a project to keep an eye on as it matures.

The downside with this is that even though the fix is known, Red Hat is sticking to their procedure, and has stated that they will not release the fix for this in the main kernel until 5.3 is released. Since 5.2 is fresh from the factory, it’s not likely that we’ll be seeing this fix pushed mainstream in the next few months.

This leaves a few choices for the RHEL and CentOS communities for how to proceed.

1. Weigh in with your opinion on this bug report. If enough people respond, RH will likely appease them.
2. Help test the patched kernels in the bug report. The more comfortable RH is with the patch, the more likely it is that they’ll tuck it in with a bug fix or security update. See #1.
3. Give CentOS a chance to get 5.2 out the door. Once it’s released, folks will have some time to roll up a kernel repository at http://people.centos.org similar to what was done with bz321111. Since this bug is strictly performance affecting, installs won’t be an issue, and you can update to the modified kernel, or use the stock release as you see fit.

In theory, you could also roll your own kernel with this patch if you didn’t want to wait, however if you do this, you’re accepting responsibility for building it properly and tracking all the kernel security and bug updates until the patch becomes mainstream. I wouldn’t recommend this method since it requires more time and upkeep, but for folks who roll their own it provides another alternative.

While many folks compare it to the Windows registry, this isn’t entirely accurate. GConf is a bit more user friendly than that, although some similarities can be drawn. It’s a binary set of files which requires a special utility to work, and operates in a directory/file structure type. Below, we’ll go through some of the more common changes users are likely to want.

The Basics:

There are two ways to go about playing around in your GConf registry. You can use the gconf-editor gui, which is in the gconf-editor package, or you can use gconftool-2, which is a command line driven application, and a little more cumbersome to maneuver around in. The basic command to help get you around in gconftool-2 is ‘gconftool-2 -R /’. With this command, you’ll see the directory/file structure which makes up the registry, and their associated settings. If you’re new to gconf, it’s probably best to start out with the gui.

Starting small:

A few times a month or so, users will ask how to tell gnome or nautilus to ignore blank CD input, or to at least do something useful with it, like open k3b instead of the default nautilus burn window. Setting this up with gconf-editor is relatively simple. Open it up, and browse to the ‘/desktop/gnome/volume_manager/’ directory. Inside this directory you’ll find a number of settings that you can modify for various automated media handling. You can change the default movie player, dvd player and more from this directory. Incidentally, ‘automount_drives’ and ‘automount_media’ are located here also, so if you’re having trouble with usb drives, this is one thing to check. The two options that we’re concerned with right now are ‘autoburn_data_cd_command’ and ‘autoburn_audiio_cd_command’.

By default these are both set to ‘nautilus –nodesktop burn:‘, but this isn’t the behavior we want. If you’d like to have k3b loaded up instead, simply change the string values to ‘k3b’ and you’re off and running. This is a per user setting, so you don’t have to be root to modify most of these values. If you do happen to launch the app with sudo, you’ll also have the ability to enforce this setting for all users. This can get handy, and we’ll look at it a bit later on.

System Policies

Now that you’ve had a little bit to look at the user side of gconf, if you’re planning to run lab or kiosk systems you might also want to look at enforcing some of your system policies with gconf. This should not be your only security method to lock the boxes down, simply another layer to examine for inclusion.

Inside the /desktop/gnome/lockdown directory, you’ll find several settings which can help you restrict your workstations or kiosk systems, such as ‘disable_command_line’. After launching gconf-editor with sudo, set these options the way that you want, then right click them and choose ‘Set as Mandatory‘. This will enforce these changes for all system users, and disallow the user from changing the settings individually for their accounts. This can be done for many of the settings here, including application specific options. While this means of enforcement is not perfect, it can go a long way toward helping an admin regain some control and a possibly a little sanity. It’s also one EVIL BOFH prank for other admins/users… if one were so inclined… >:-)

While the InstallOnlyN plugin for yum will handle this quite nicely for the day to day stuff, once it’s happened it can be a little tricky to resolve. The easiest way is with a script called kernel-prune from Seth Vidal’s duke directory.

By piping the output of this script through xargs, you can remove all the tedium of manually removing packages one or two at a time. For RHEL and CentOS 3, where installonlyn isn’t really an option, this is pretty much the easiest way to periodically purge some unwanted fat from your system. Hope this helps at least 1 of you out there.

Feel free to share your own methods or comments below. I’m sure there are other methods out there so let’s hear them!

If you’re on IRC, feel free to stop by freenode’s #centos channel and thank Range for putting this list together.

I discovered this a few nights ago while reading through Jeremy Cole’s blog and noticed some of the functionality he was talking about. While I don’t fully trust their repositories (I prefer to stick with the distro mysql releases), I was very interested in the maatkit functionality.

After talking with Karanbir and waiting patiently for a few minutes, I had a nice shiny build of the maatkit MySQL toolkit in the testing repository to play with. It works exactly as advertised, and I could not be happier with it.

The majority of the maatkit tools are geared toward mysql replicated environments, so there’s a load of master/slave tracking capability including some new ways to track latency, lag, and more. This is all neat, but it’s not something I’m overly interested in just yet, because I don’t have any replication problems.

What does interesting me is this: mk-find

mk-find is essentially a perl implementation of find for mysql. I can now search for and sort tables by size, row, create/modify time, and perform actions based on what returns with –exec.

Another tool within maatkit which caught my eye was mk-parallel-dump, which while exceptionally long to type in (yes, I’m lazy) is quite a good tool for speeding up backups. Instead of simply dumping the entire db all at once into a single file and waiting for it to spool through huge amounts of information, I can pass this utility a directory, run a thread per table, and cut my backup time by a significant percentage of time. Simply run: mk-parallel-dump –tab –basedir /path/to/backups/ and you’re off, but if you want to get even more hi-tech, they include some sample scripts for scheduled backups.

If you’re looking for a way to make your mysql management tasks easier, have a look at maatkit, or just go get it from the CentOS-Testing repository and see how it can simplify your life.

Installing Aide
yum install aide
What? You expected it to be harder? Now that we have aide installed, we need to configure it. The default config file should be okay for most folks who haven’t relocated things on the distro too much. Double check to make sure that all the directories you want to scan are listed. If you want to fine-tune the aide config, then you’ll need to edit /etc/aide.conf.

Initializing Aide’s Records

The next thing we need to do is create the initial aide database. For this, you need to run the following command:
# /usr/sbin/aide --init

This will take a little bit of time to run, and you’ll have some disk churn for minute or two while aide investigates your system and creates a baseline. Once this is done, we’re going to run an initial query of the system, just to make sure that everything’s working properly. To do this, run the command below:
# cp /var/lib/aide/aide.db.new.gz /var/lib/aide/aide.db.gz
# /usr/sbin/aide --check


This copies the initial database to the current database, then checks them against each other. In theory you should not have any differences. If you do, investigate them. As we’re still setting this up, they’re likely to be mundane .viminfo files or something similar. Keep in mind that when you update applications via ‘yum update’ that you may see aide go a bit nuts, just as tripwire or others would. You’re replacing files on your system when you update, and this is exactly what aide is designed to warn you about. In a perfect world, you should get some output like the text below:

# aide --check
AIDE, version 0.13.1
### All files match AIDE database. Looks okay!


Once we’re satisfied that aide is working as we expect, it’s time to set up a periodic check of the system. Only you can determine what’s often enough for your servers. I personally run aide as weekly cron, by creating a file in /etc/cron.weekly/ called aide.cron, with the following contents:


#!/bin/bash
/usr/sbin/aide --check | /bin/mail -s "Weekly Aide Data" email@host.com

This runs my check once a week. That’s pretty much it to setting up aide. If you want to see more options for aide, please check out the documentation in /usr/share/doc/aide-*/

Update:

So it seems that by default, aide requires selinux to be enabled, or at least permissive so that it can record the selinux contexts of the files it watches. If for some reason you really, truly want to have selinux disabled, but you still want aide to watch the system, use the config file below. It is identical to the default scan, but with the selinux bits removed.

selinux-free.aide.conf

ssh -X remotehost # yawn. X forwarding through ssh.

ssh -Y remotehost # trusted X forwarding through ssh. Still yawn, let’s do something fun.

ssh -D2222 remotehost # This is okay. This command sets up a SOCKS proxy on port 2222 which can be used with firefox (and Internet Explorer if you really hate yourself) to avoid office internet filters…. not that I condone such anti-social behavior.

ssh -L 3306:database.example.net:3306 # okay, now we’re getting interesting. This generates an ssh tunnel between your machine and the remote box on port 3306. This works for connecting to remote mysql instances when firewalls would ordinarily interfere. Simply point your mysql client to localhost:3306 and you’re off and running. This can also be applied to other applications as well. A slight modification, and you get the string below:

ssh -L3389:remote.win2k3.server:3389 user@remote.linux.box # This command is a variation on the command above, allowing us to connect to those unsightly windows machines via rdesktop for remote administration. Best of all, we do this without opening up the remote desktop ports to the outside world. Remember folks, that windows code is expensive, you have to keep your precious little snowflakes safe after all.

With the commands above, you can alternatively add -f , if you want ssh to go to the background after the authentication portion is handled. Otherwise it’ll just leave you sitting at a remote shell prompt. If you wanted to add a built-in self-destruct, you could add ’sleep 30′ after the ssh command strings above. This tells ssh to exit after 30 seconds if nothing has made a connection via the tunnel created.

ssh -nNT -R 2222:localbox:2222 remotebox # This command lets you create a reverse ssh tunnel, so that if you connect to remotebox:2222, you’ll be connected to the local machine on port 2222 also. This is useful when you really want to go home at night, but your boss demands you keep working. This way, you each get what you want, and you can avoid the firewall your office employs to keep folks from remotely connecting to…well, if you’re using this, you don’t really care.

Crap, what if I already have an ssh session open, but I forgot to create the tunnel? Not to worry, there are escape keys to rescue you. Operating a little like screen, ~C will open an ssh command prompt so that you can start or stop tunnels as needed. It’ll look a bit like the one below:

[jperrin@server ~]$
ssh> help
Commands:
-Lport:host:hostport Request local forward
-Rport:host:hostport Request remote forward
-KRhostport Cancel remote forward


As always, we’re just scratching the surface of what ssh can do, so if you want more information, fire up ‘man ssh’ in your favorite terminal and sit down for a good read.

To get started with your own personal config file, create and open ~/.ssh/config in your favorite text editor. The easiest way to organize this file is to break it up into host based sections, similar to the example below.

Host SessionName # Short Name you can use to start a session
HostName server.example.com # FQDN or ip of the server
User myuser # Unprivileged username for this session

Host Session2
HostName server.example.com
User myadminuser #this is a 2nd user with sudo access to the same server
IdentityFile id_rsa

Host server2
HostName server2.example.com
User yetanother # yet another user on another host.
IdentifyFile second_rsa_key
Compression yes #server has limited bandwidth, so enable compression

Host backup
HostName backups.example.com
User backupuser # backup user
BindAddress 192.168.1.2 #Connect from this ip address.
Port 2222 # Use this port instead of 22.

As you can see from the example above, we change ssh keys so you don’t have to use the same one, ssh usernames to eliminate the @, and the address ssh connects from since we have multiple IP addresses on this system. This isn’t really even scratching the surface of what you can do with ssh, so if you have even more customizations in mind, have a read through the ssh_config manpage in your favorite terminal.

So, for those of you thinking “uh, there’s an example for doing this right in the httpd docs”… let me save you the trouble. It doesn’t work. the instructions at http://httpd.apache.org/docs/2.2/rewrite/rewrite_guide.html cause the url to be rewritten properly, but then promptly 404’s because it looks in /var/www/html/u/username/ for the files. Afer digging around in google, and working with some friendly folks in #apache (yes they do exist) we have a solution:

RewriteEngine On
#RewriteLog logs/rewrite.log # Uncomment for rewrite logging
#RewriteLogLevel 3 # uncomment for verbose logging
RewriteCond %{REQUEST_URI} ^/([^/]+)
RewriteCond /home/%1 -d
RewriteRule ^/([^/]+)(.*) /home/$1/public_html/$2


Take that bit of code, and drop it into /etc/httpd/conf.d/homedir.conf or wherever else you’d like in your httpd configs, and reload apache. From there you’ll be able to use shorter, sexier UserDir urls for your user’s webpages.