Upgrade To Leopard Server

If you've tried to upgrade Tiger server to Leopard, you've probably already learned that it's a bumpy road. The process is so fraught with bugs that Apple themselves recommend against it in their Server Essentials manual, stating it should never be preformed on "production" machines. Unfortunately, not every organization has a spare XServe to experiment on, and those sharing files off internal disks may not want the hours of data shuffling that installing from scratch would entail.

The issues inherit in a Leopard Server upgrade can be addressed quickly and easily if you know what to look for. With a little advance preparation, and this article as a checklist, you can make sure the process is safe and successful for most OS X installations.

You'll want to have checked in advance that any third-party applications you need run on Leopard, and you'll definitely need a known-good backup in case something goes awry. Now let's get started.

Document Server Settings:

The big problem with the Leopard upgrade isn't that it loses data. In fact, it's the opposite, with the newly upgraded volume turning on new services, adding duplicate users, or even sharing out random (and sometimes private) directories. The most important part of the process, then, is to document all your Server Admin settings prior to upgrade. You'll want to have notes (or screenshots) detailing all your DNS zones, web sites, firewall configuration, etc. If your team doesn't already document all their server settings in a change log, now's the time to catch up on the paperwork.

Archive Open Directory:

The next step you'll want to take is to export all your Open Directory settings. This includes not just your users and groups, their passwords, their managed preferences, but also all your Kerberos and Password Server data. After you've upgraded your server, you'll restore this information to insure nothing's been corrupted in the upgrade process.

You can do this easily by using the "Archive" pane in the "Open Directory" options of Server Admin. Simply click the "Choose..." button to set the location the collected data should be saved to, then click the "Archive..." button to set the name and password for the disk image that's created. Type very carefully, and since the password isn't verified before the image is created, test that you can open it before you proceed.

Install Leopard Server:

With your preparations made, you can run the Leopard Server install normally. You'll be asked to enter your new Leopard serial number, but otherwise the process shouldn't require any configuration. The upgrade takes about a half hour on the current generation of XServe, and you can fill the time by downloading any recent updates you plan on installing. When your server reboots, it'll likely have a host of services you don't want turned on, so it's best to restart it with the network cable unplugged.

Reconfigure Services:

If all went well (or at least as expected), you should now have a fully functional Leopard Server. Maybe a little too functional, even. Leopard tends to turn on a bunch of undesired services during the upgrade process. Was your XServe an email or NFS server previously? There's a good chance it might be now.

Open Server Admin, and select the your local machine from the left column. Now take a good, hard look at the services listed below it. Turn off the services you weren't running prior to upgrade, then check your remaining service settings to make sure nothing unexpected has been added or changed. If you don't need Apache 1.3, it's a good time to upgrade to version 2.2 in the Web options as well. Now it's time to plug back in the network cable and wrap up the job.

Restore Open Directory:

Now that the rest of the damage has been undone, you'll want to clean up your account information. If your server remained an OD master through this process, you're in luck. Otherwise, you'll need to promote it in the "Settings" pane of the Open Directory options, then return to the "Archive" pane.

Choose the archive disk image you made earlier, and click "Restore..." to return your Open Directory domain to it's pre-upgrade state. Once you've confirmed Open Directory accounts work locally, test each of the services you're offering over the network as well, making sure that everything functions as it did prior to the upgrade process.

And voila! While not exactly fast or automatic, you should now have a newly upgraded and properly functioning Leopard Server. Now you can go find out what's going to break when you upgrade your client machines...

Recommended Reading: While it may contradict some real-world experience, Apple nonetheless publishes an excellent guide on Upgrading and Migrating 10.5, essential reading for anyone planning a complex upgrade to Leopard Server.

Configure And Deploy NetInstall

One of the best features of OS X is the built-in ability to clone a customized installation to other machines. By creating a .dmg image of an existing installation in Disk Utility, then using the "Restore" feature to copy it to another disk, you can install a pre-configured OS onto any number of Macintosh workstations.

As much time as this saves, however, you're still cloning one machine at a time. What if you have tens or even hundreds of machines in your organization? Fortunately, there's a built-in system to handle that as well. All you have to do is deploy NetInstall.

Creating NetInstall Images:

Ten years ago, when Apple's administrative tools were aimed more squarely at academia, schools began using NetBoot to deploy their lab machines. By booting workstations off a central server-side system image, machines could run custom configurations that reset cleanly for each user. NetInstall was built on that same technology, allowing your servers to host similar custom images, but install them over the network onto each machine's local disk.

Creating that image is now done with the System Image Utility, found on OS X Server in the /Applications/Server directory. While an image can be built directly from the OS X installation DVD, they're hard to customize and seldom up-to-date. You're better off creating a customized installation on an external drive (or saved as a disk image), then mounting it on your server. While Leopard theoretically allows network installation onto both Intel and PowerPC hardware from the same image, preparing separate images can save time and frustration in the long run.

When you're ready, launch System Image Utility, and in the "Create an Image" window select "NetInstall Image". Click "Continue", then on the next screen enter a name and description of the image you're preparing. For basic deployment just click "Create", and save the image in the default location of /Library/NetBoot/NetBootSP0. For more complex configurations, click "Customize" to add post-install scripts, filtering by hardware type, or allow more client-side installation options.

You may want to go take lunch while your image builds, but when it's done you can repeat the process for each image you'll need to deploy over your network.

Configuring The NetBoot Service:

Once your images are prepared, the next step is to configure NetInstall on your network. Launch Server Admin and select the NetBoot service from the left column. Click the "Settings" icon in the toolbar, then choose the "General" pane. Here you'll choose the network interface for NetInstall to run on (typically Ethernet 1), and the volume to store images and client data (the system array, in this example, though you may prefer another volume if you're mixing NetBoot and NetInstall).

Now move to the "Images" pane. Check "default" if you've got a single image, or there's an image you'd like to force if one isn't specified. Next check "enable" for each of the images you're planning to deploy, and choose the protocol over which you'll offer them.

The NetBoot service can run over either HTTP or NFS. Given that most companies filter HTTP traffic, or engage some kind of web proxy for port 80, utilizing NFS is usually the better choice. In these circumstances, be sure to allow traffic over port 1049 on your internal network.

With these options set, click "Save", then click "Start NetBoot" to take the service live.

Configuring Clients For NetInstall:

Once NetInstall is running happily on your network, all that's left is to boot your client machines. If you're only offering one image (and you've marked it as the "default" in the "Images" pane), you can simply restart your machines while holding down the "n" key. If you're offering multiple NetInstall images, open System Preferences on your client machines, and choose the appropriate image to install from the "Startup Disk" pane. When you restart, the computer will boot into the installer automatically. You'll need each computer connected to the network during this procedure, and you'll save yourself a world of trouble if they're using ethernet rather than wireless.

That's all there is to it. Allowing you to save hours, if not days, of work on every rollout, NetInstall is one of the most powerful, and easiest, tricks up the Macintosh administrator's sleeve.

Recommended Reading: There's no better, or more extensive, explanation of BSDP (the protocol on which NetInstall is built) than the amazing How NetBoot Works by the even more amazing Mike Bombich.

Configure IP Failover — Part 2

Last week, in part one of this article, we took a look at configuring basic IP failover features on Mac OS X to provide high availability. This week, we'll take a look at building on the process with custom shell scripts, and the secret to successful AFP setup for failover systems.

Extending IP Failover Features:

When IP failover is triggered, the system first checks if any scripts exist in the /Library/IPFailover directory. This script system can be used for additional failover logging, email notification, or even mounting and unmounting volumes in environments without a SAN. The folder doesn't exist by default, but creating it allows you to add functionality to the failover process.

Shell scripts are placed inside a folder named for the main server IP (192.168.0.250 in our example from part one), and must begin with one of four prefixes: PreAcq, PostAcq, PreRel, and PostRel. The PreAcq and PostAcq scripts are run before and after IP acquisition from the main server, while the PreRel and PostRel scripts handle actions surrounding IP release when the main server comes back online.

IP Failover and File Sharing:

If you're configuring IP failover on Macintosh, there's likely one more thing you'll want to know. While there are articles elsewhere on configuring Postfix, BIND, and Apache for high availability, even Apple doesn't currently have support documents for IP failover of AFP. This can make setting up file sharing failover both difficult and frustrating. Should your main file server fail, you can make new connections to an alternate server, but your existing connection is lost (and unfortunately, with it, your data).

The secret is that AFP requires access to a per-session token, established in order to seamlessly reconnect an existing connection. It's this cache that allows AFP to survive network interruptions without data loss. For single-server setups that file is located on the startup volume, but in a failover environment it needs to be accessible to both servers. Open /Library/Preferences/com.apple.AppleFileServer.plist and find these lines near the bottom:

<key>reconnectKeyLocation</key>
<string>/private/etc/AFP.conf</string>

Change the string above, and you change the where OS X Server stores the reconnect key. This allows the to be kept in the shared storage available to both your servers, and provides the ability to "reconnect" to a new server during IP failover.

Configure IP Failover — Part 1

For companies that have grown on an assortment of mismatched machines, the efficiency and flexibility of a server-based workflow can be liberating. Open Directory, managed services, portable home directories, and many other features are impossible to use without a server to to drive them. This dependance on a single, central system can be a detriment as well. After all, what can you do when your all-important server inevitably fails?

You can already have a second server in place. IP failover lets another machine take charge should your primary server cease to function. This feature isn't for novices (and is one of Apple's only Leopard sales points that lacks a graphical interface), but if you can't afford any downtime its a feature the deserves serious consideration.

IP Failover Planning:

Before you begin, there are some significant considerations when designing a failover setup. Most importantly, IP failover doesn't work on OS X between Apple's older PowerPC and newer Intel machines. Ideally, you'll want to failover between identical hardware and operating systems, but at a bare minimum you need to be working with the same architecture and major OS version.

Next, you'll likely be inclined to give your failover server a "real" job, having it offer services on your network to stay busy instead of just waiting for your main server to fail. Unfortunately, those services often become an essential element of users' workflow, meaning a failover that changes the failover server's IP could be as disastrous as losing your main server itself.

While IP failover doesn't require a Directory Service scheme, any authenticated services will need account information from the main server. Setting up your failover server as an Open Directory replica is certainly the simplest method to achieve this, and provides essential authentication should your main server go down.

Finally, IP Failover can provide an available server at an expected address, but it doesn't guarantee the files you need are available as well. You'll need both servers to access data from a shared store of some kind (preferably an XSan volume), a task outside the scope of this article.

IP Failover Basics:

To start your IP failover setup, you'll need to make sure that both servers are networked correctly. It's best if their primary ethernet ports (en0 on the command line) have IPs from the same subnet, making it more likely that anyone who can reach the first can reach the second. In our examples we're using 192.168.0.250 and 192.168.0.251. You'll also want to have a direct connection from the main server to it's failover, rather than connecting them through a switch or other network device, preferably using IP over Firewire. This direct connection will carry the heartbeat signal, which lets the failover know the main server is still functioning.

To provide the heartbeat connection, go to System Preferences and select the "Network" pane. The graphical interface calls this port "Firewire 1", while on the command line it's fw0. Set the IP address and subnet mask on the main server (but no router or DNS servers) on your main server, choosing a private subnet not used by your regular IP network, then set up your failover server in the same fashion with a subsequent IP. Here we'll use 10.0.0.1 and 10.0.0.2 for our heartbeat connection.

Now we'll configure IP failover itself by editing the /etc/hostconfig files. On your main server, add the following:

FAILOVER_BCAST_IPS="192.168.0.251 10.0.0.2"
FAILOVER_EMAIL_RECIPIENT=USER@DOMAIN

Here FAILOVER_BCAST_IPS lets the server know first which IP to failover to and second which IP to send the heartbeat signal. The FAILOVER_EMAIL_RECIPIENT setting tells the server to send mail to the address provided when failover takes place. You'll need to replace the IPs here with the IPs of your failover server and heartbeat connection respectively, and USER@DOMAIN with the appropriate administrative email address. On the second server, edit the same file, adding:

FAILOVER_PEER_IP_PAIRS="en0:192.168.0.250"
FAILOVER_PEER_IP="10.0.0.1"
FAILOVER_EMAIL_RECIPIENT=USER@DOMAIN

The FAILOVER_PEER_IP_PAIRS setting provides the ethernet port to handle failover (using the command line numbering) as well as the IP to adopt should the heatbeat signal fail. FAILOVER_PEER_IP tells the failover server which address to expect the heartbeat connection from. Keep in mind that to send email alerts, you'll need SMTP service configured on both servers.

To test your failover setup, simply ping the main server from a workstation, then disconnect the server's ethernet connection. The ping should stop momentarily, then pick up again once the failover server stops receiving the heartbeat signal.

Next week, in part two of this article, we'll discuss configuring high availability for AFP connections and extending IP failover with custom shell scripts.

Files Corrupted By Windows Sharing

Windows doesn't have a long history of interoperability and standards compliance, but the history it does have is somewhat checkered. As a result, most Macintosh administrators develop a knee-jerk aversion to Microsoft products and protocols. So when the first few Photoshop files wind up corrupted on your SMB file shares, complaining of inconsistent permissions issues or just displaying jumbled nonsense, it's easy to blame the problem on the Windows workstations. More often than not, though, it's a configuration error on Mac OS X Server that renders these files unavailable or unreadable.

The SMB (or Server Message Block) protocol is used for file sharing by Windows operating systems. It's also the basis of Windows-compatible file sharing on OS X Server, using the open source "Samba" implementation. When you configure individual share points for the SMB service, you can configure "Protocol Options" including custom naming, guest access, and permission inheritance. Those options also include enabling oplocks (also known as opportunistic locking) by default, which can be a serious and unexpected cause of file damage.

Opportunistic locking can improve performance by allowing multiple users to edit portions of a file concurrently. Unfortunately, they only work when clients edit files via over SMB, meaning that they can cause conflicts and become destructive when enabled on share points that offer AFP or NFS sharing as well. They can also cause performance issues when used with large binary files, the very kinds of files most often associated with the Macintosh platform.

While this issue can become quite serious over time, the solution is trivial. Uncheck the "Enable oplocks" box from the File Sharing configuration in Server Admin, then restart the service from the "Server" menu. You can turn off opportunistic locking this way on each share offered through multiple protocols, and eliminate the threat of file corruption through misconfiguration.

Recommended Reading: For more information on file locking with SMB, check out the Locks And Oplocks chapter of Oreilly's "Using Samba", available free online.