There’s been quite a bit of interest in the XFS offerings included in the 5.4 release of RHEL, and unfortunately it hasn’t really lived up to the hype. There are a few things you’ll need to know if you want to use the included xfs support

• It’s not really included: The XFS kernel module is only in the x86_64 version of the kernel. If you’re using the x86 release, you get no XFS module.
• There aren’t any XFS tools included either: The xfsprogs package isn’t included in RHEL 5.4 Server (see RH’s own BZ #521173 about this). So basically mkfs.xfs isn’t included. That’s not very handy if you were hoping to actually USE xfs.
• Anaconda won’t let you use XFS either: If you’re doing a fresh install and you boot anaconda with the XFS option, you get all the nifty little XFS options you’d expect when you set up your partitioning scheme. The downside is that once anaconda has all the information and tries to actually format things like you told it to, it segfaults. Why? Because it can’t actually MAKE the XFS file system. The tools aren’t there, remember?

Now, many of you may be saying “But Red Hat TOLD me to use xfs in their release notes!” and yes, yes they did. Quoting the Release Notes from RHEL 5.4 we see this ->

Users of GFS2 that do not need high availability clustering are encouraged to look at migrating to other file systems like the ext3 or xfs offerings. The xfs file system is specifically targeted at very large file systems (16 TB and above).

I’d really like to see RH fix support for this, because XFS is an excellent file system, and has some excellent performance when paired with things like MySQL databases.

Red Hat: if you’re listening, please reverse the decision on https://bugzilla.redhat.com/show_bug.cgi?id=521173 and include the XFS toolkits. Your users will thank you for it.

For around a year now, I’ve been wanting to move away from IBM’s (okay, LSI’s) rdac mpp drivers used for the ds4XXX  series disk chassis on RHEL and CentOS. When RHEL 5.3 came out boasting support dm-multipath support for the DS4XXX series, I was understandably overjoyed. The only problem was I couldn’t make it work. After several rounds of cursing, muttering and poking folks smarter than me to help out, the problem became immediately clear.

The example configs which work fine for the ’supported’ platforms have a text string mismatch when using the ‘unsupported’ ds4700. Basically you have to change a bit of text slightly because the hardware identifies itself slightly differently. Who’d have thunk it, right?

Below is the multipath.conf snippet I’ve been using now for the past month with some pretty good success.


defaults {
user_friendly_names yes
}
blacklist {
devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
devnode "^(hd|xvd|vd)[a-z]*"
wwid "*"
}
blacklist_exceptions {
wwid "3600XXXX"
}
devices {
device {
vendor "IBM"
product "1814 FAStT"
getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
prio_callout "/sbin/mpath_prio_rdac /dev/%n"
features "0"
hardware_handler "1 rdac"
path_grouping_policy group_by_prio
failback immediate
rr_weight uniform
no_path_retry queue
rr_min_io 1000
path_checker rdac
}


Now clearly you’ll have to modify the wwid to suit your own environment, and you’ll also want to exclude the non-multipath device references (/dev/sdb for example) from lvm.conf if you’re using lvm.  This got things going for me, so hopefully it’ll help others out as well.

I’m sure by now everyone who reads this has heard the news about the goings on of the CentOS development team. Within a few hours following the release of the open letter, the story had been picked up, chewed on, misunderstood, panicked over, and copied to a few hundred other places and blogs too numerous to count. The mailing list, twitter, and the irc channels associated with the centos project were (and still are) hotbeds of advice, discussion, and controversy.

One thing has stood out from the anticipated frenzy surrounding this, and that’s been the overwhelming support of the CentOS community. The number of people stepping forward to volunteer hardware, time, advice, and cash (even though we’re currently asking folks to not do that) has been tremendous, and it’s incredibly refreshing to know that we have the backing of the community in what we’re doing.  I’d like to take a moment to personally say thank you to everyone who has been kind enough to offer their support or has made the effort to thank the developers following the announcement. This kind of support and outreach is what makes the project great, and what keeps people helping out.

Thank you.

There were loads of interesting comments and corrections about the security page on the CentOS wiki.  I’ve made a few updates regarding some missing sections, and some minor corrections. What I’m interested in now are the other security pages that the community might want. I’d like to turn this into a full blown wiki section for security, detailing basic web server security, mail server security, ssh protection, and more. What would the community (or at least those of you who read this) want to see first from a network application standpoint?

This is something that I’ve griped about in the past, but without taking action, griping is basically just annoying other people in an arrogant fashion. Now while I’m very good at doing this, I’d much rather do something to help people provide proper security for their systems.

To this end, I’ve started a page on the CentOS wiki (Hardening CentOS)  that will very likely turn into its own section in the future. This page uses Steve Grubb’s RHEL hardening guide, as well as the NSA’s RHEL 5 security guide as the basis for locking down the operating system in a reasonably secure fashion. I do not claim it to be all-powerful, nor will everything there apply to everyone in all instances. Take from it what you need to. What is covered on the page is the basis for locking down a system with very minimal impact to distribution standards or package changes. I’ll continue to add to this page, and break out other pages to encompass securing the distribution provided httpd, mysqld, etc as I get time.

If you find this information useful, please let me know. Comments and suggestions are warmly welcomed, so long as it’s constructive. As usual, the OMG-U-SUCK comments won’t get approved unless they’re accompanied by facts and/or evidence to support the opinion.

Once again, you can see this page at http://wiki.centos.org/HowTos/OS_Protection

Red Hat appears to be taking virtualization quite seriously, and they appear to be choosing KVM as their virtualization method of choice. There’s a rather interesting article over at Practical Tech discussing the upcoming virtualization additions to the RHEL 5 line in the next point release. Have a read over the full article at http://practical-tech.com/infrastructure/red-hat-makes-kvm-its-linux-virtualization-of-choice/

Well, as most of you may already know, Red Hat has released RHEL version 5 update 3 today, and they appear to have been quite hard at work. So what can you all expect in CentOS 5.3? Here’s a brief rundown of cool stuff to look forward to:

Networking

• NetWorkManager and wpa_supplicant updates mean better wireless security support. NetworkManager has a whole host of updates listed, so loads of good things have been happening there.
• Updated driver support for a number of broadcom, forcedeth, ralink, and realtek cards made it into the kernel, so those of you in irc complaining that your nic wasn’t recognized should be happier after this.
• There are also a few improvements for intel networking, both wired and wireless, so that should give the intel crowd their feel-good too.

Storage

This is where things get interesting, so hang on.

• ext4 support is now included, so you can feel free to play with it. All accounts have it being pretty interesting.
• encrypted block devices are now supported in anaconda for direct install. Anyone with a laptop should be interested in this one. (This one is my personal favorite. A die-hard suse fan always rubs on this when we debate)
• There’s added support for IBM’s DS4xxx series disk systems in the dm_multipath package now. In theory this should rid us of the rdac driver update reboot hell. I’ll be testing this feature out tomorrow.
• 3ware and megaraid_sas also made the cut for driver updates. These two should have a fair bit of performance improvements to them.

One thing I’m still waiting to see sorted out is the httpd fiasco on x86_64. In previous releases, you could install both, but it would cause conflicts when run. RH says they fixed this by removing the x86_64 version of httpd from the x86_64 distro. I’m really hoping they mean that they’ve removed the x86 version from the x86_64 distro, and that the release notes just have a nice little heart-stopping typo. Anyone dealing with multi-arch issues might want to keep an eye on this one between visits to the therapist.

Update: Seems the httpd issue was for ppc, though the arch was not clearly spelled out in the release notes. Have a look at http://www.redhat.com/archives/rhelv5-list/2009-January/msg00098.html for information.

You can get the full reading on what’s coming from this url:  http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5/html/Release_Notes/index.html

What features are you looking forward to the most with the new release? I’m curious to see which features people are most interested in using. Let me know in the comments below.

Computer security is once again becoming a hot topic for administrators.  There are dozens of new sites springing up around the web, and each is slinging their own ‘Perfect’ setup instructions.  They have the usual bell curve of good advice, okay advice, and advice that will effectively leave you with a smoldering pile of rubble where your data used to be. What disturbs me is the growing number of seemingly reputable Linux security websites pitching brittle security advice. I call it ‘brittle’ advice, because while your system is hardened over its previous state, it is also no longer supportable by your distribution.  There are several causes for this, but I’ll outline the top three reasons for this breach in support.

Bad Advice:

Really, who didn’t see this coming as the #1 here? Ignoring the occasional typos, misprints and ramblings, there are quite a few security sites which just plain offer bad advice. Much of the advice has to do with offering incorrect permissions changes, which may inadvertently expose data, or cause system breakage.  Let’s use a CentOS system here for an example:

SecureCentOS.com (no relation to the CentOS project itself) tells you that the security on /tmp should be secured from the default baseline, and they’re right. They recommend mounting a file in loopback and modifying options there. This is fine. They move on to double check the permissions for your newly created /tmp mount and tell you to chmod it 777.   Damn, they were doing so well until they had you do that. The default CentOS permissions, which set the stickybit such that only the user who creates the file can move or delete it. It’s been done with the stickybit in every major linux distro for ages now.  Oh well, I’m leaving hope that it’s a typo and that they’ll change their chmod statement to 1777 any day now.

Ignoring Distro Tools

As we descend into flawed-security-advice hell, the next level we come to involves side-lining the distribution provided security tool-kits. This is commonly done through ignorance of their existence or operation, or just basic willful neglect.  Regardless of how it happens, the end result remains the same. If you bring your own toys to party, you’re responsible for them. Again looking to CentOS for the example, we find that a large number of websites will completely ignore the security tools default in the distribution. They ignore selinux, aide, audit, and pam restrictions, while touting things like grsec (which may possibly be stagnant, see this announcement), ossec.  I don’t want to turn this into a grsec vs selinux holy war ala vim-vs-emacs. Grsec is a good security tool, however it’s not what RHEL and CentOS have chosen to throw their weight behind. Grsec requires a significantly newer kernel than RHEL and centos ship, and this newer kernel can lead to ABI issues with applications or dependency issues for updates. The further you stray from the tools which ship with the distro, the less the distro support mechanisms will be able to help you.

Providing Details

Perhaps the biggest issue I have with supposed security advice is the lack of detail as to what some of the options mean or do. For a rather glaring example,  we can use securecentos.com’s page on tuning sysctl.conf. They provide instructions to wget the sysctl.conf file they supply, however they don’t go into detail as to what they set, or what benefits the options provide. Teaching is (or should be) at the core of security. An admin should understand the changes they’re making to a system rather than blindly pecking at keys because someone told them to. To make matters worse, the provided sysctl.conf file doesn’t contain comments explaining the various options there either. A budding admin is going to have no idea what a martian is in terms of linux networking, or why it may be important to have rp_filter enabled. These sorts of things need to be documented to take the voodoo out of security for new admins looking to do the right thing.

Do the Right Thing

With the plethora of poor security advice showing up, and often interspersed with good advice how are admins supposed to know what to listen to?  There are a few guidelines to work with so that you keep your system secure, and still get support from your distribution channels. If you stick with these steps, you should be in good shape for having a supportable, secure system.

1. Identify the areas to secure based on exposure.  (web server, file server, DNS, etc)
2. Document the changes that you make. For support or duplication later on, it’s important to keep a record of changes you make to the default configurations.
3. Ask your distribution what security tools are included.
4. Use package management tools (see the previous post).
5. Look to 3rd party repositories after examining what the distribution provides and/or recommends.  3rd party vendors may well be an excellent source of security tools, but look at what the distro offers first.
6. Keep major modifications to core components to a minimum. Many distributions support only the software that they ship. If you replace core components like the kernel(for grsec) or go replacing distribution packages with external software, you may have to get your support from the external source. The more you deviate, the more places you’ll have to shop for support.
7. Most importantly, understand the changes you make to the system. If you’re unsure about a particular security setting (I’m thinking of sysctl.conf here) or you don’t fully understand the option means, ask.  The worst thing you as an admin can do is modify a system without understanding the benefit or penalty for a particular change.

It’s a near daily occurrence that I see an administrator building something from source because either they fail to understand how backporting works, or they want to install software which isn’t packaged in the default repositories. Source installs used to be common, but the problems inherent in them drove the creation of package management software.  Since you have package management software, USE IT!   I cannot stress this enough.

Source installs suck for the following reasons:

1. They conflict with packaged installs of software.
2. They do not inform the package management system that they exist, which leads back to #1.
3. They generally leave no easy way to uninstall them later on.
4. There’s no record of what an installation does to your system, or what files are placed where. Searching them becomes a pain (what files did that install of  openssl drop on my system?)
5. They are not easily duplicated across multiple systems over time (think remote support for users)
6. There is no distribution update mechanism. If you updated via source for security reasons then congratulations, the onus is now on you to maintain that install, and track product security for its lifetime on your server.

The benefits of using package management:

1. Reliable reproduction of installs over time, and across multiple systems
2. Easily search for installs, files, and modifications of packages.
3. Easily get package meta-data such as who built it, when, build options used, and scripts required to set up the package for use.
4. Audit trails for allow you to know what files have changed since install, what permissions were modified, etc.
5. Simplified install, update and removal of packages. No hunting for remnants of previous installs.
6. Simplified dependency management and version tracking.
7. Distribution supplied updates. Rather than 1 admin tracking a dozen packages for updates, you have dozens of builders and hundreds of users testing and tracking the software.

Just about every distribution out there has some form of support structure, either via irc, email, forums, or carrier pidgeon.  In many cases this community will be able to point you to a location where you can find packages for the software you’re wanting to install. In the case of CentOS, there are a number of very good 3rd party repositories providing quite literally thousands of packages which are not included in the base distribution. By using your distribution’s support mechanisms, you can provide a voice for the users and potentially guide the development efforts for future software releases.

In short, if you need a particular piece of software, please use your distribution’s package management system and if you cannot find a pre-existing package, ASK.  If one doesn’t exist, and won’t be created, build a package yourself (there are tutorials aplenty online) and submit it for inclusion. Source installs should only be done when all other options have been exhausted.

Update: Well, since this one seems to have generated some interesting responses, I thought I’d post a couple links that I’d neglected to mention. Thanks to toracat for pointing these out to me, since they fit in perfectly with what we’ve discussed here.

• http://wiki.centos.org/PackageManagement/SourceInstalls
• http://www.centos.org/modules/newbb/viewtopic.php?topic_id=14408&forum=47

           *             ,
                       _/^\_
                      <>
     *                 /.-.\         *
              *        `/&\`                   *
                      ,@.*;@,
                     /_o.I %_\    *
        *           (`'--:o(_@;
                   /`;--.,__ `')             *
                  ;@`o % O,*`'`&\
            *    (`'--)_@ ;o %'()\      *
                 /`;--._`''--._O'@;
                /&*,()~o`;-.,_ `""`)
     *          /`,@ ;+& () o*`;-';\
               (`""--.,_0 +% @' &()\
               /-.,_    ``''--....-'`)  *
          *    /@%;o`:;'--,.__   __.'\
              ;*,&(); @ % &^;~`"`o;@();         *
              /(); o^~; & ().o@*&`;&%O\
        jgs   `"="==""==,,,.,="=="==="`
           __.----.(\-''#####---...___...-----._
         '`         \)_`"""""`
                 .--' ')
               o(  )_-\
                 `"""` `

ascii art taken from http://www.geocities.com/SoHo/7373/xmas.htm#xmastree