Configure IP Failover — Part 1

For companies that have grown on an assortment of mismatched machines, the efficiency and flexibility of a server-based workflow can be liberating. Open Directory, managed services, portable home directories, and many other features are impossible to use without a server to to drive them. This dependance on a single, central system can be a detriment as well. After all, what can you do when your all-important server inevitably fails?

You can already have a second server in place. IP failover lets another machine take charge should your primary server cease to function. This feature isn't for novices (and is one of Apple's only Leopard sales points that lacks a graphical interface), but if you can't afford any downtime its a feature the deserves serious consideration.

IP Failover Planning:

Before you begin, there are some significant considerations when designing a failover setup. Most importantly, IP failover doesn't work on OS X between Apple's older PowerPC and newer Intel machines. Ideally, you'll want to failover between identical hardware and operating systems, but at a bare minimum you need to be working with the same architecture and major OS version.

Next, you'll likely be inclined to give your failover server a "real" job, having it offer services on your network to stay busy instead of just waiting for your main server to fail. Unfortunately, those services often become an essential element of users' workflow, meaning a failover that changes the failover server's IP could be as disastrous as losing your main server itself.

While IP failover doesn't require a Directory Service scheme, any authenticated services will need account information from the main server. Setting up your failover server as an Open Directory replica is certainly the simplest method to achieve this, and provides essential authentication should your main server go down.

Finally, IP Failover can provide an available server at an expected address, but it doesn't guarantee the files you need are available as well. You'll need both servers to access data from a shared store of some kind (preferably an XSan volume), a task outside the scope of this article.

IP Failover Basics:

To start your IP failover setup, you'll need to make sure that both servers are networked correctly. It's best if their primary ethernet ports (en0 on the command line) have IPs from the same subnet, making it more likely that anyone who can reach the first can reach the second. In our examples we're using 192.168.0.250 and 192.168.0.251. You'll also want to have a direct connection from the main server to it's failover, rather than connecting them through a switch or other network device, preferably using IP over Firewire. This direct connection will carry the heartbeat signal, which lets the failover know the main server is still functioning.

To provide the heartbeat connection, go to System Preferences and select the "Network" pane. The graphical interface calls this port "Firewire 1", while on the command line it's fw0. Set the IP address and subnet mask on the main server (but no router or DNS servers) on your main server, choosing a private subnet not used by your regular IP network, then set up your failover server in the same fashion with a subsequent IP. Here we'll use 10.0.0.1 and 10.0.0.2 for our heartbeat connection.

Now we'll configure IP failover itself by editing the /etc/hostconfig files. On your main server, add the following:

FAILOVER_BCAST_IPS="192.168.0.251 10.0.0.2"
FAILOVER_EMAIL_RECIPIENT=USER@DOMAIN

Here FAILOVER_BCAST_IPS lets the server know first which IP to failover to and second which IP to send the heartbeat signal. The FAILOVER_EMAIL_RECIPIENT setting tells the server to send mail to the address provided when failover takes place. You'll need to replace the IPs here with the IPs of your failover server and heartbeat connection respectively, and USER@DOMAIN with the appropriate administrative email address. On the second server, edit the same file, adding:

FAILOVER_PEER_IP_PAIRS="en0:192.168.0.250"
FAILOVER_PEER_IP="10.0.0.1"
FAILOVER_EMAIL_RECIPIENT=USER@DOMAIN

The FAILOVER_PEER_IP_PAIRS setting provides the ethernet port to handle failover (using the command line numbering) as well as the IP to adopt should the heatbeat signal fail. FAILOVER_PEER_IP tells the failover server which address to expect the heartbeat connection from. Keep in mind that to send email alerts, you'll need SMTP service configured on both servers.

To test your failover setup, simply ping the main server from a workstation, then disconnect the server's ethernet connection. The ping should stop momentarily, then pick up again once the failover server stops receiving the heartbeat signal.

Next week, in part two of this article, we'll discuss configuring high availability for AFP connections and extending IP failover with custom shell scripts.

Files Corrupted By Windows Sharing

Windows doesn't have a long history of interoperability and standards compliance, but the history it does have is somewhat checkered. As a result, most Macintosh administrators develop a knee-jerk aversion to Microsoft products and protocols. So when the first few Photoshop files wind up corrupted on your SMB file shares, complaining of inconsistent permissions issues or just displaying jumbled nonsense, it's easy to blame the problem on the Windows workstations. More often than not, though, it's a configuration error on Mac OS X Server that renders these files unavailable or unreadable.

The SMB (or Server Message Block) protocol is used for file sharing by Windows operating systems. It's also the basis of Windows-compatible file sharing on OS X Server, using the open source "Samba" implementation. When you configure individual share points for the SMB service, you can configure "Protocol Options" including custom naming, guest access, and permission inheritance. Those options also include enabling oplocks (also known as opportunistic locking) by default, which can be a serious and unexpected cause of file damage.

Opportunistic locking can improve performance by allowing multiple users to edit portions of a file concurrently. Unfortunately, they only work when clients edit files via over SMB, meaning that they can cause conflicts and become destructive when enabled on share points that offer AFP or NFS sharing as well. They can also cause performance issues when used with large binary files, the very kinds of files most often associated with the Macintosh platform.

While this issue can become quite serious over time, the solution is trivial. Uncheck the "Enable oplocks" box from the File Sharing configuration in Server Admin, then restart the service from the "Server" menu. You can turn off opportunistic locking this way on each share offered through multiple protocols, and eliminate the threat of file corruption through misconfiguration.

Recommended Reading: For more information on file locking with SMB, check out the Locks And Oplocks chapter of Oreilly's "Using Samba", available free online.

Open Archives Without Stuffit

In the late eighties, the original Stuffit was an invaluable utility, bringing compression and archiving capabilities to the Macintosh filesystem. In Mac OS X, however, you could soon right-click on any file or folder to create a cross-platform .zip file with the "Make Archive" feature (now called "Compress" in Leopard). While the need to compress files with Stuffit passed long ago, the need to "unstuff" old archives has never disappeared. The Unarchiver can open .sit files, as well as more than thirty other legacy formats (such as Disk Doubler and BinHex), and does it all for free.

While stand-alone expansion utilities for Stuffit archives aren't new, The Unarchiver actually ties this ability into the operating system interface, allowing you to uncompress supported formats by simply opening them in the Finder. Just double-click on the application when you first install it to determine which file formats you'd like The Unarchiver to handle.

The author of The Unarchiver isn't selling anything, so users never get nagged to "upgrade" to a deluxe version, a common complaint with the Stuffit-branded utilities. Plus, once you get your users into the habit of compressing files through the Finder, you'll be able to share archives with Windows users without shelling out for Stuffit licenses year after year.

There are, unfortunately, tasks The Unarchiver can't handle. Most notably, it lacks support for the proprietary .sitx format, which offers better compression ratios for graphics files than the original. It's also essentially a front-end for the open source libxad project, so new file formats aren't added to The Unarchiver unless they're added to libxad first. Despite these weaknesses, The Unarchiver is an invaluable tool for breaking the expensive Stuffit habit.

The Unarchiver is available for free download from its creator, Dag Ågren.

Disable Hardware Components

Security policy means different things to different companies. In some environments, using managed preferences to control external drive access would be considered draconian. In others, leaving the Airport card plugged in (or firewire ports connected) is thought of as irresponsible. What can a systems administrator do to limit hardware use on company machines?

One option is to disable use of certain hardware components at a system level. Kernel extensions, the software which allows the system to access hardware, are kept in the /System/Library/Extensions folder. Removing specific extensions from that folder will disable the associated hardware, preventing it from being used. Remove the wrong ones, unfortunately, and the your operating system won't boot.

If you want to prohibit wireless access, those files are AppleAirPort.kext, AppleAirPort2.kext, and AppleAirPortFW.kext. To disable Bluetooth capabilities (and therefore Bluetooth file transfer), you'll want IOBluetoothFamily.kext and IOBluetoothHIDDriver.kext. Apple's iSight cameras can be disabled to prevent video and photographs of your facility by replacing AppleUSBVideoSupport.kext and Apple_iSight.kext. The use of external hard drives is regulated by IOUSBMassStorageClass.kext and IOFireWireSerialBusProtocolTransport.kext.
For every hardware interface, there's a kernel extension file to match.

If your goal is to disable your chosen hardware permanently (so that software updates won't re-enable it), just replace the missing files with empty text files of the same name, then "Get Info" on each file in the Finder and check the "Locked" option so they won't be reinstalled.

Using this technique, you can insure that even users with administrative privileges can't get around the hardware restrictions on your standard installation.

Recommended Reading: Apple's very technical explanation of kernel extensions can be found at their developer site. You might also consider controlling hardware access through Open Directory's managed preferences, explained previously in this very blog.

Clear System Font Caches

Anyone who supports graphic designers learns to hate fonts. As tiny pieces of software loaded directly into the operating system, they're responsible for more than their fair share of system issues. So it goes with users whose systems freeze up on login, displaying nothing but their desktop background and a lonely spotlight icon in the upper left corner of the screen. The issue is so common, there are a myriad of third-party tools to address it. And like so many common problems, this one comes down to fonts again.

Faced with a machine that consistently lets users log in, but get no further, the problem is very often a corrupt font cache. This causes the system to have issues rendering type, prevents the menu bar from displaying properly, and therefore stops the login process before the user can take control of their work environment (even after reboot). The issue is especially common in later versions of Tiger, but it plagues Leopard systems as well. One way to correct the problem is to boot into single-user mode and clear out the system's font caches entirely.

Reboot the machine while holding down the "Command" and "S" keys. If the frozen workstation runs Tiger, type:

rm -r /Library/Caches/com.apple.ATS/*

If the frozen workstation runs Leopard, instead type:

atsutil databases -remove

Replace USER with the name of any administrative user, and MACHINE with the hostname or IP, belonging to the workstation. You'll be asked for that user's password, after which these commands will remove the Apple Type Server caches from the frozen machine, and with them this issue. If you can login as an administrative user, these commands can also be run from the Terminal by preceding them with "sudo". You can then safely restart, and login (as well as fonts) should function normally again.

Special Thanks: We were reminded of this problem (and its Tiger-specific solution) by Aaron Robinson, systems administrator at Seattle's fine Hornall Anderson Design Works.

Recommended Reading: For more information on corrupt font caches (and some products to clear them without the command line), you can check out the CreativeTechs QuickTips blog.