Mirror Disks After Install

Disk mirroring, where data is written to two disks simultaneously, is a great low cost method to protect against single-disk failure and improve read-intensive performance. Apple's Disk Utility provides an easy way to set two disks up as a RAID mirror at installation, but once the operating system has been installed OS X can't mirror an existing drive without reformatting and reinstalling.

What began as a missed opportunity in Tiger became a buggy implementation in Leopard, and in the past year the procedure for mirroring an existing volume has changed no less than three times. This is an essential feature for storage management, and it inexplicably still only works from the command line. While not the most convenient, this method works with any Tiger or Leopard machine.

This process has the potential to destroy all the data on your machine, so make sure you have a current backup of the drive you're going to mirror and confirm the backup is both restorable and bootable. Next, if you're mirroring your existing startup disk, you'll need to boot off an installation DVD or external drive for the first part of the operation. When you're ready to proceed, type the following into the Terminal:

diskutil list

This will produce a listing of the attached disks, along with their types, names, sizes, and identifiers, like so:

/dev/disk1
#: type name size identifier
0: Apple_partition_scheme *465.8 GB disk1
1: Apple_partition_map 31.5 KB disk1s1
2: Apple_HFS Server Disk 465.6 GB disk1s2

Use the name of the volume you wish to mirror to find it's identifier. If your machine only has one internal hard drive, for instance, the identifier of your boot volume will most likely be disk1s2, the second slice (after the partition scheme and map) of disk 1. Once you've determined which disk you're working with, type the following, replacing IDENTIFIER appropriately:

sudo diskutil enableRAID mirror IDENTIFIER

If all goes well, the disk will unmount, reappearing in the Finder a moment later along with "The disk has been converted into a RAID" reported in the Terminal. At this point you can insert the additional disk you wish to use in the mirrored RAID array (assuming it's not installed already), then reboot the machine off the original drive.

Now when when you run diskutil list from the Terminal you should see two new listings, one for the new virtual RAID array (the one without slice information) and the other for the additional physical disk you installed.

/dev/disk2
#: type name size identifier
0: Apple_HFS Server Disk *465.6 GB disk2
/dev/disk3
#: type name size identifier
0: GUID_partition_scheme *465.6 GB disk3
1: EFI 200.0 MB disk3s1
2: Apple_HFS New Disk 465.4 GB disk3s2

Now with the necessary identifier information, you can assign the physical disk to the mirrored RAID array, replacing RAIDARRAY with the RAID volume (in our example disk2) and NEWDISK with the hard drive being added (here disk3):

sudo diskutil addToRAID member RAIDARRAY NEWDISK &

How long the RAID mirror takes to build depends on how much data is on the original volume, but the ampersand at the end of the command tells it to run in the background. When the process is completed you'll have two fully mirrored and redundant disks, built from your existing installation without ever having to erase the original volume.

Expand Printing Options

It's been a year since Apple first shipped "Leopard", its fifth retail version of OS X. Which means it's been about a year since I started hearing people complain about Leopard's truncated printer controls. Many users still haven't discovered that the printing options in 10.5 are hidden behind that unlabeled blue arrow, and those that have are tired of constantly expanding the print dialog just to do their jobs.

If you never again want your users annoyed and flummoxed by this user interface puzzle, you can easily change the default display setting for the print dialog. Just open the Terminal and type:

defaults write -g PMPrintingExpandedStateForPrint -bool TRUE

This changes the print settings for the current user, defaulting to the expanded print dialog above and exposing the options previously stored in the "Page Setup" dialog.

Configure IP Failover — Part 2

Last week, in part one of this article, we took a look at configuring basic IP failover features on Mac OS X to provide high availability. This week, we'll take a look at building on the process with custom shell scripts, and the secret to successful AFP setup for failover systems.

Extending IP Failover Features:

When IP failover is triggered, the system first checks if any scripts exist in the /Library/IPFailover directory. This script system can be used for additional failover logging, email notification, or even mounting and unmounting volumes in environments without a SAN. The folder doesn't exist by default, but creating it allows you to add functionality to the failover process.

Shell scripts are placed inside a folder named for the main server IP (192.168.0.250 in our example from part one), and must begin with one of four prefixes: PreAcq, PostAcq, PreRel, and PostRel. The PreAcq and PostAcq scripts are run before and after IP acquisition from the main server, while the PreRel and PostRel scripts handle actions surrounding IP release when the main server comes back online.

IP Failover and File Sharing:

If you're configuring IP failover on Macintosh, there's likely one more thing you'll want to know. While there are articles elsewhere on configuring Postfix, BIND, and Apache for high availability, even Apple doesn't currently have support documents for IP failover of AFP. This can make setting up file sharing failover both difficult and frustrating. Should your main file server fail, you can make new connections to an alternate server, but your existing connection is lost (and unfortunately, with it, your data).

The secret is that AFP requires access to a per-session token, established in order to seamlessly reconnect an existing connection. It's this cache that allows AFP to survive network interruptions without data loss. For single-server setups that file is located on the startup volume, but in a failover environment it needs to be accessible to both servers. Open /Library/Preferences/com.apple.AppleFileServer.plist and find these lines near the bottom:

<key>reconnectKeyLocation</key>
<string>/private/etc/AFP.conf</string>

Change the string above, and you change the where OS X Server stores the reconnect key. This allows the to be kept in the shared storage available to both your servers, and provides the ability to "reconnect" to a new server during IP failover.

Configure IP Failover — Part 1

For companies that have grown on an assortment of mismatched machines, the efficiency and flexibility of a server-based workflow can be liberating. Open Directory, managed services, portable home directories, and many other features are impossible to use without a server to to drive them. This dependance on a single, central system can be a detriment as well. After all, what can you do when your all-important server inevitably fails?

You can already have a second server in place. IP failover lets another machine take charge should your primary server cease to function. This feature isn't for novices (and is one of Apple's only Leopard sales points that lacks a graphical interface), but if you can't afford any downtime its a feature the deserves serious consideration.

IP Failover Planning:

Before you begin, there are some significant considerations when designing a failover setup. Most importantly, IP failover doesn't work on OS X between Apple's older PowerPC and newer Intel machines. Ideally, you'll want to failover between identical hardware and operating systems, but at a bare minimum you need to be working with the same architecture and major OS version.

Next, you'll likely be inclined to give your failover server a "real" job, having it offer services on your network to stay busy instead of just waiting for your main server to fail. Unfortunately, those services often become an essential element of users' workflow, meaning a failover that changes the failover server's IP could be as disastrous as losing your main server itself.

While IP failover doesn't require a Directory Service scheme, any authenticated services will need account information from the main server. Setting up your failover server as an Open Directory replica is certainly the simplest method to achieve this, and provides essential authentication should your main server go down.

Finally, IP Failover can provide an available server at an expected address, but it doesn't guarantee the files you need are available as well. You'll need both servers to access data from a shared store of some kind (preferably an XSan volume), a task outside the scope of this article.

IP Failover Basics:

To start your IP failover setup, you'll need to make sure that both servers are networked correctly. It's best if their primary ethernet ports (en0 on the command line) have IPs from the same subnet, making it more likely that anyone who can reach the first can reach the second. In our examples we're using 192.168.0.250 and 192.168.0.251. You'll also want to have a direct connection from the main server to it's failover, rather than connecting them through a switch or other network device, preferably using IP over Firewire. This direct connection will carry the heartbeat signal, which lets the failover know the main server is still functioning.

To provide the heartbeat connection, go to System Preferences and select the "Network" pane. The graphical interface calls this port "Firewire 1", while on the command line it's fw0. Set the IP address and subnet mask on the main server (but no router or DNS servers) on your main server, choosing a private subnet not used by your regular IP network, then set up your failover server in the same fashion with a subsequent IP. Here we'll use 10.0.0.1 and 10.0.0.2 for our heartbeat connection.

Now we'll configure IP failover itself by editing the /etc/hostconfig files. On your main server, add the following:

FAILOVER_BCAST_IPS="192.168.0.251 10.0.0.2"
FAILOVER_EMAIL_RECIPIENT=USER@DOMAIN

Here FAILOVER_BCAST_IPS lets the server know first which IP to failover to and second which IP to send the heartbeat signal. The FAILOVER_EMAIL_RECIPIENT setting tells the server to send mail to the address provided when failover takes place. You'll need to replace the IPs here with the IPs of your failover server and heartbeat connection respectively, and USER@DOMAIN with the appropriate administrative email address. On the second server, edit the same file, adding:

FAILOVER_PEER_IP_PAIRS="en0:192.168.0.250"
FAILOVER_PEER_IP="10.0.0.1"
FAILOVER_EMAIL_RECIPIENT=USER@DOMAIN

The FAILOVER_PEER_IP_PAIRS setting provides the ethernet port to handle failover (using the command line numbering) as well as the IP to adopt should the heatbeat signal fail. FAILOVER_PEER_IP tells the failover server which address to expect the heartbeat connection from. Keep in mind that to send email alerts, you'll need SMTP service configured on both servers.

To test your failover setup, simply ping the main server from a workstation, then disconnect the server's ethernet connection. The ping should stop momentarily, then pick up again once the failover server stops receiving the heartbeat signal.

Next week, in part two of this article, we'll discuss configuring high availability for AFP connections and extending IP failover with custom shell scripts.

Clear System Font Caches

Anyone who supports graphic designers learns to hate fonts. As tiny pieces of software loaded directly into the operating system, they're responsible for more than their fair share of system issues. So it goes with users whose systems freeze up on login, displaying nothing but their desktop background and a lonely spotlight icon in the upper left corner of the screen. The issue is so common, there are a myriad of third-party tools to address it. And like so many common problems, this one comes down to fonts again.

Faced with a machine that consistently lets users log in, but get no further, the problem is very often a corrupt font cache. This causes the system to have issues rendering type, prevents the menu bar from displaying properly, and therefore stops the login process before the user can take control of their work environment (even after reboot). The issue is especially common in later versions of Tiger, but it plagues Leopard systems as well. One way to correct the problem is to boot into single-user mode and clear out the system's font caches entirely.

Reboot the machine while holding down the "Command" and "S" keys. If the frozen workstation runs Tiger, type:

rm -r /Library/Caches/com.apple.ATS/*

If the frozen workstation runs Leopard, instead type:

atsutil databases -remove

Replace USER with the name of any administrative user, and MACHINE with the hostname or IP, belonging to the workstation. You'll be asked for that user's password, after which these commands will remove the Apple Type Server caches from the frozen machine, and with them this issue. If you can login as an administrative user, these commands can also be run from the Terminal by preceding them with "sudo". You can then safely restart, and login (as well as fonts) should function normally again.

Special Thanks: We were reminded of this problem (and its Tiger-specific solution) by Aaron Robinson, systems administrator at Seattle's fine Hornall Anderson Design Works.

Recommended Reading: For more information on corrupt font caches (and some products to clear them without the command line), you can check out the CreativeTechs QuickTips blog.