Portable Home Directories — Part 1

Available since version 10.4, Portable Home Directories have become one of the most elegant and well-implemented features of a full Mac OS X Server deployment. Functioning much like Windows' roaming profiles (or earlier Solaris NFS/NIS environments), they allow a user to log in from any computer on your network while retaining their personal data and settings. Unlike entirely network-based systems, however, they do so by synchronizing user data to the server (so that a full copy of the home directory exists in both locations), eliminating the need for constant connectivity.

Portable Home Directories make for simpler backup of user data, both by copying off the server rather than each client machine, and by allowing remote users to synchronize via VPN. They also free users from being tied to a single machine, allowing for greater flexibility and less service down-time. It's because this functionality is so powerful that it's often assumed to be difficult to put into practice. Instead, with the proper infrastructure already in place, deploying Portable Home Directories is practically the reward for having done everything else right.

Planning For Portable Home Directories:

Before you actually implement any kind of server-based account storage, you'll want to make sure you have sufficient storage and bandwidth on an available OS X server. This may seem obvious to some, but for reasonable performance, fifty users with a 40GB quota requires at least 2TB of relatively high-speed (and hopefully redundant) disk attached to a gigabit network switch. This isn't an exotic setup by any means, but it may be more than you just have lying around.

You'll also need clients bound to a functioning Open Directory environment, complete with internal DNS. If you don't yet have this set up, refer to our earlier series on how to master Open Directory. Once Directory Service users and groups are in place, Portable Home Directories are nothing more than cleverly deployed managed account preferences. There's a lot to keep track of, but very little you wouldn't already know how to do.

Configuring Portable Home Directory Preferences:

In Workgroup Manager, browse to the "LDAPv3" directory (as opposed to the local user directory), then choose the multi-headed "Groups" button on the left and the "Preferences" icon from the toolbar. Select the group (or groups) you're offering Portable Home Directories, then click the "Mobility" icon in the center of the window to configure that group's settings. If you're deploying this feature to all your users, you're better off creating an all-encompassing "Employees" group to do so.

Beginning in the "Account Creation" tab with the "Creation" pane, choose to manage these Preferences "Always", the check "Create mobile account when user logs in to network account". Uncheck the box which requires confirmation, as this allows the user to skip the Portable Home Directory set up for their individual account. Below that, choose to "Create home" directories "with default sync settings".

Next comes the Account Expiry tab, new to 10.5. By allowing you to set a time limit after which the client-side copy of a home directory expires, it helps clean up the occasional "orphaned" set of user data (a full home directory left, for instance, on a machine only used once by that user during maintenance on their own machine. This feature can reduce the chance of accidentally filling client machines with multiple unused accounts, but does so at the risk of letting the computer determine when data should be disposed of. If you enable it, do so with caution.

Finally, the "Rules" tab lets you set what data will synchronize and when. Start with the "Login & Logout Sync" pane and once again click the button to "Always" manage, then check the box to "Sync at login and logout". The first list above allows you to set which directories you'll sync, and unless you feel you can fully predict your users' behavior the best approach is usually to select the entire home directory (as represented by the tilde symbol). You can then choose what not to sync in the second list below, including full paths, partial names, and even regular expressions. Be careful if you delete any of Apple's pre-configured items to skip, especially ~/Library/Application Support/SyncServices, which can result in synchronization issues and potentially data loss. The "Merge with user's settings" box allows you to decide if individuals can add or subtract to the list of data being synchronized.

The Background Sync pane, functions identically, and in most cases makes sense to configure identically as well. The only exceptions would be huge local files which change often, or live databases which won't sync properly. The Entourage database, for instance, sits both criteria and should be excluded from background synchronization. The "Options" pane also allows you to choose how often background sync takes place. With your configuration decided, click the "Apply Now" button to save your settings.

Next week, in part two, we'll set up the AFP share where your new Portable Home Directories reside and configure your Open Directory accounts to store user data there.

Recommended Reading: While I might not recommend implementing it in a production environment, Greg Neagle's multi-part article on Portable Home Directory Without Open Directory provides fantastic under-the-hood information on exactly how Portable Home Directories function at his "Managing OS X" blog.

CS3 Won’t Save To 10.5.3 Server

Despite Apple's encouragement to install OS upgrades as soon as they're released, most systems administrators test updates for a couple of weeks to see if any obvious or significant issues occur in their environment. It's been two weeks since Apple released Leopard 10.5.3, and while the update fixes a laundry list of problems (including Active Directory, AFP, iCal, Time Machine, and SMB issues), it breaks one simple feature that most Mac users simply can't live without: With 10.5.3 on client or server machine, some Adobe CS3 applications (primarily Photoshop, but occasionally InDesign) can no longer save to network shares.

It's hard to tell who to blame for this disaster, if blame is important to you. The fact that saving documents to the server worked just fine in 10.5.2 (and that multiple sources have reported that the functionality returns in 10.5.4) makes Apple look like the bad guy. On the other hand, Adobe has very publicly resisted modernizing portions of its underlying application code, and their antiquated position of not officially supporting direct server usage is bewildering to anyone computing in the 21st century.

The obvious solution is to avoid installing 10.5.3 at all (or roll back to 10.5.2 if you've kept a Time Machine backup of your system volume) until the problem is solved. The immediate (and far-less obvious answer) is to always use the "Save As" option, which continues to work perfectly on servers of all types.

Update: On June 30th, Apple released their 10.5.4 update. Among it's improvements, they list "Resolves an issue with saving and reopening Adobe Creative Suite 3 files on a remote server".

Recommended Reading: The normally-reasonable John Nack (Senior Product Manager, Adobe Photoshop) first reported issues saving CS3 files to 10.5.3 servers in his otherwise entertaining blog. For those with strong stomachs, some very angry commentary can be found on Apple's own discussion boards.

Restart Kerberos Manually

The Kerberos authentication protocol is an encrypted ticketing system at the heart of Apple's Open Directory. It is the basis for Mac OS X's "Single Sign On" features, and a required component for integration with Windows Active Directory domains. Unfortunately, it's possible for the Kerberos service to stop functioning properly, and when it dies, a good number of your network services die with it.

Ideally, you can simply click the "Kerberize" button in the "General" pane of Server Admin's Open Directory settings. That should restart the service. This article for when you find yourself in less than ideal circumstances.

Why Kerberos Can Stop On OS X Server:

When the Kerberos service fails, the likely culprit is a DNS issue. This isn't the only reason Kerberos can break, but it's most common and the easiest to test for. The Kerberos KDC service (or Key Distribution Center), requires matching forward and reverse DNS lookups, and changing a server's hostname or IP manually (or losing connection to your DNS servers) can require the machine to be re-Kerberized. Since Tiger, Mac OS X has also been designed to dynamically set a machine's hostname on boot using a reverse lookup, meaning that external changes to a server's DNS listings can cause it to change names and break its original Kerberos realm.

Before you try to restart Kerberos, you'll want to know which (if any) of these things went wrong. If you aren't sure how to go about this, check with whomever in your organization controls Domain Name Service, or use the procedure from the beginning of our article on setting up Open Directory.

How To Start Kerberos From The Command Line:

Once your server is properly registered in DNS, you can begin reconstructing your Kerberos realm. The first step is to create a new edu.mit.Kerberos file, which Mac OS X can fortunately do (somewhat) automatically. Open the Terminal and type:

sudo kerberosautoconfig -r REALM -m HOSTNAME

For this command, HOSTNAME is the fully qualified domain name of your server, such as server.example.com. REALM is the server's DNS name again, but this time in all capital letters, like SERVER.EXAMPLE.COM. Next you'll create a KDC database for the new Kerberos realm:

kdcsetup -f /LDAPv3/127.0.0.1 -w -a DIRADMIN -p PASSWORD REALM

Here DIRADMIN is the name of any user with directory administration privileges, and PASSWORD is their actual login password. Once that's done, you can Kerberize the server's Open Directory domain with the following:

sudo slapconfig -kerberize -f DIRADMIN REALM

Finally, configure the Single Sign On mechanism to utilize Kerberos authentication for all applicable OS X services.

sudo sso_util configure -r REALM -a DIRADMIN -p PASSWORD all

This should once again leave you with a fully-functioning Kerberos installation on your Open Directory server, all without ever having had to reboot the machine.

If you're following this procedure, keep one additional things in mind: Because the kdcsetup and sso_util commands unfortunately requires that you provide a password on the command line (and therefore store that password in your command history), it's good security practice to change that user's password immediately after this process.

Recommended Reading: For an illustrated overview of the Kerberos authentication Process, there's the handy Kerberos Chart [PDF - 72KB] at Computer World. For more depth into how Kerberos works on OS X, there's an excellent three-part article at AFP548.com.

Host Corporate Email — Part 4

In part three of this four-part series, we took a look at MX records, and how properly-configured DNS is essential for email hosting. In this final installment, we'll take a look at ways to insure uninterrupted service and handle high volume for your corporate Macintosh-based email server.

Deploying A Backup MX Server:

If you've followed along with this series, you should now have a functioning well-configured mail server, available to the internet and exchanging messages from inside and outside your company. That's an adequate solution until a power outage, a loss of internet connectivity, or a hardware failure takes place. When disaster strikes, this machine becomes a single point of failure for your entire communications system, with employees unable to send or receive mail.

Historically, the answer to this problem has been to always have a backup MX, a secondary server that handles mail in the event the primary server is unreachable or inoperative. For most organizations that's still the best solution, as it allows not just redundancy but lets you geographically separate the machines on separate power grids and ISPs if you have a second office.

To set up a backup MX server, first add an additional MX record in both your internal and external DNS. The record name traditionally indicates that it's your secondary MX (such as mx2.makemacwork.com) and is set with a lower priority (and therefore, a higher number) like 20. This tells anyone sending your domain email to attempt delivery to your backup MX should connection to the primary MX fail.

Next, configure a second Mac OS X Server as you did the first, but using the second server's name as both the "Domain name" and "Host name" in the "General" pane of the Server Admin mail settings. Make sure IMAP is disabled, but check "Relay outgoing mail through host" and fill in the name of your primary mail server.

As a last step, you'll need to go back to your original server, and configure it to allow relaying from your backup host. In the "Relay" pane of the Server Admin mail settings, add the IP address of your secondary mail server. This allows any mail stored on your backup MX while your primary is down to be accepted without further authentication. Restart the mail service on your main host, and you should now have redundant mail receipt and delivery.

This time-proven method is how businesses and universities have avoided mail outages for the last twenty years, and few operating systems make it as easy to set up as Mac OS X. Ideally, your backup MX server will be housed, powered, and connected to the internet separately. If that's not possible, simply putting the machine on a separate circuit with a cheap cable or DSL connection can still get you through a host of unexpected problems (and more importantly, through routine maintenance).

XSan Mail Clustering:

From the old-school to the new-fangled, the more modern way of handling the need for mail service redundancy is mail clustering, where mail service can be handled by multiple servers all sharing a common real-time storage pool which holds incoming and outgoing messages. Mail clustering itself isn't a new idea, but Apple's version of it (requiring both Leopard and XSan 2) is so recent it doesn't yet even have documentation.

Nonetheless, if you're familiar with XSan implementation, the new clustering features aren't hard to take advantage of. The XSan 2 Admin tool allows you to create new Volumes optimized for Mail and assign them Affinities through a simple configuration wizard. You can then choose "Clustering" from the "Advanced" pane in Server Admin's mail settings, and click the "Change" button to launch another wizard to reconfigure your mail setup across multiple XSan client servers.

The advantage to Xsan mail clustering over the backup MX approach is primarily one of performance, with huge arrays of disks able to process mail more quickly and store it more efficiently. The disadvantage in most cases is that those arrays must be physically attached to multiple servers in the same location. Still, this setup eliminates the majority of reasons you might outsource your company's email service. If your organization needs redundant delivery and scalable performance, the only thing now standing in the way is next year's budget.

Special Thanks: Our friend and colleague Eddie Kelley, technician extraordinaire at Portland's Mac Pac, volunteered how to set up a backup MX without ever resorting to the command line.

Recommended Reading: If you're planning on maintaining a Macintosh mail server, it's important to have at least a passing understanding of Postfix, the open-source mail system OS X uses. By far the best reference is Ralf Hildebrandt and Patrick Koetter's Book of Postfix, available in PDF from publishers No Starch Press.

Host Corporate Email — Part 3

In part two of this four-part series, you set up user accounts, storage restrictions, and authentication methods for new email hosting on an OS X server. This week, we'll look at what it takes to bring your new mail server up smoothly on your internet domain.

Testing An Isolated Email Configuration:

As much work as you've done up until now, the only machine that knows you're running mail service is the email server itself. Take advantage of that fact to adequately test your existing configuration, sending messages to a test account (from both inside and outside your network), collecting them using an IMAP client configured specifically for this purpose, and using the machine as an SMTP server to send mail out to other addresses. It's ten minutes of mild annoyance, but it could save you hours of frustration if there are problems you don't find until the system is live.

MX Records And The Domain Name System:

Once you know your server can send and receive mail properly, it's time to let the whole world know it's there. Like most internet services, successful communication between machines requires that the appropriate DNS information be available to every computer involved in the transaction. While it's unlikely you'd implement your own mail system without a strong working knowledge of the Domain Name System, email is unusual in that you'll need to adjust DNS both inside your own network and out on the internet.

On your internal DNS servers, you'll have to point to your mail servers (in the form of MX records). In the DNS Zones settings of Server Admin, simply click the "plus" button next to the "Mail Exchangers" field and fill in the hostnames for the email servers you're about to configure. Then assign the priority (the order mail delivery is attempted) to those hosts, usually 10 for the primary MX (the only server set up so far).

If the DNS servers you administer control the DNS for your external domain as well, then mail will start moving towards your new servers as the new MX records propagate across the internet Domain Name System. More likely, your external DNS is controlled by a parent company or bandwidth vendor. In those cases, you'll need to have them change your external MX records as well.

Testing Your MX Records:

When an email client sends mail to an address like user@example.com, it queries it's DNS servers for the MX record to the example.com domain, and makes an SMTP connection to send mail to the server address that's returned. So once you hit "Save" in the bottom right of the Server Admin window (or your ISP does the equivalent on their end), every machine that receives your updated information will start sending mail to your new servers.

Once you've flipped the switch, it makes sense to check the MX records for your local environment immediately. Open the Network Utility application, select the "Lookup" pane, then "MX Record" from the pull-down menu. Enter your domain name and click "Lookup" to see the current data offered by your immediate DNS provider.

Be aware, however, that DNS is propagated in fits and starts, as each server synchronizes MX record changes at their own pre-determined schedules across the internet. Machines which share your primary name server may begin using the updated MX records immediately, with large ISPs following suit in a matter of minutes, while international providers or smaller services could take hours or (in a worse case but not uncommon scenario) days.

As a rule of thumb, you'll want to have your DNS settings in effect at least 48 hours before your users need to rely on dependable mail delivery (with a common strategy being to enact changes Friday evening one the office is empty). Come Monday morning, your users should have a brand new mail system, fully operational and under your company's control.

Next Week: In our fourth and final installment, we'll take a look at strategies for redundancy and high-availability to make OS X email truly suitable for corporate use.

Recommended Reading: Paul Kincaid-Smith's article Configuring DNS For Standards-Based Mail Service gives an excellent overview of the issues administrators most often face with MX records.