rsync+hfsmode

first released: 12Sep2004
last updated: 30Jan2012

rsync+hfsmode is a version of rsync which recognizes the HFS+ resource forks and Finder metadata in Mac OS X files, and properly copies them to remote filesystems. The destination can be any OS and filesystem that supports rsync, so you can use rsync+hfsmode to safely archive all of your Macintosh files[*] to servers running FreeBSD, Solaris, Linux, et cetera.

Newer versions of standard rsync (3.0.0+) will copy resource forks and Finder metadata between two Mac OS X systems (only). If that suits your requirements, by all means use that instead — rsync 3 also adds many other useful new features. Otherwise, read on.


Downloads

rsync-2.6.3+hfsmode-1.2b2 (STABLE on PPC or Intel/Leopard, BROKEN and WON'T BUILD on Intel/Snow Leopard+) rsync-2.6.3+hfsmode-1.4a1 (ALPHA! this will supersede 1.2, but is presently for TESTING ONLY!) Rsync source is available from rsync.samba.org

To build rsync-2.6.3+hfsmode-1.4a1
Requires Xcode [4.1 GB]. Downloading the proper rsync+hfsmode binary from the links above is recommended for non-developers.

   bash$ tar xzf rsync-2.6.3.tar.gz
   bash$ patch -p0 -u < rsync-2.6.3+hfsmode-1.4a1.diff
   bash$ cd rsync-2.6.3
   bash$ LDFLAGS="-framework CoreServices" ./configure
   bash$ make proto
   bash$ make man
   bash$ make
NOTE: make man will fail, unless you happen to have yodl2man installed. Don't worry about it.


A Short History of the Problem

Mac OS X uses the HFS+ filesystem. HFS+ files are often composed of a data fork, a resource fork, and Finder metadata. The data and resource forks contain what you normally think of when you think of file information: data, program code, etc. Finder metadata includes information like file type and creator, comments, modification dates, locked and invisible status, and Finder colors.

Traditional UNIX filesystems only store a single stream of file data (the HFS+ equivalent of the data fork). Mac OS X (or Darwin, more precisely) is a genuine BSD UNIX, but with a nontraditional UNIX filesystem. Because of this, standard UNIX tools can only see certain portions of OS X files.

The difficulties caused by HFS/HFS+ aren't new. Early Mac users of BBSes and the Internet had the same problems when uploading files from a Mac for storage by other operating systems. Since the foreign OS had no way to store the additional HFS data, the uploads would be incomplete and useless. To solve this problem, Apple and others invented serialization formats that collected the full set of file information into a single data stream. Examples (some with varying and/or additional design goals) include AppleSingle, AppleDouble, MacBinary, BinHex, and Stuffit.

Apple released Mac OS X in 2001, and suddenly decades of software written for traditional UNIX systems could be used on Macs, but some of that software would need to be updated before they would be able to work with all of the components of HFS+ files.

One such standard UNIX tool is rsync, an excellent file synchronization utility which is also great for use as filesystem backup software. Rsync builds and runs without errors on OS X, but because it is unaware of resource forks and other metadata, it creates incomplete (and therefore corrupt) backups. [UPDATE: Newer versions of rsync (3.0.0+) are now aware of resource forks and metadata, but still cannot transfer them to non-Mac OS X systems.]


The Solution

rsync+hfsmode is HFS+ metadata-aware. Resource forks and Finder metadata are assembled on the sender into a synthetic file in standard AppleDouble format before being sent to the destination.

This method preserves disk space on both sides, with zero redundant data and only a small amount of overhead per file [*]. It works with any destination filesystem and operating system (tested primarily with Solaris and Linux), and even with older or unpatched versions of rsync.

This is currently a one-way, backup-only process. To restore from the backup to an HFS+ filesystem, reassemble the AppleDouble files with an Apple utility named FixupResourceForks (see simple instructions below).


Filename mapping details

rsync+hfsmode names AppleDouble files according to the Netatalk naming scheme. This means that if your remote filesystem is served over Netatalk to other Macs, reverse conversion is done automagically. It's also convenient for the rest of us.

For a source file named filename, the destination filesystem will store two files: filename, containing the source file's data fork, and ._filename, containing the resource fork and Finder metadata, in AppleDouble format.

Reassembly of restored files

Apple's /System/Library/CoreServices/FixupResourceForks will reassemble the two component files into a single HFS+ file.

   bash$ ls -a
      ._filename
      filename
   bash$ /System/Library/CoreServices/FixupResourceForks filename
   bash$ ls -a
      filename

FixupResourceForks works recursively, so to reassemble a large number of files just specify the directory name (or ".") in place of filename, above. FixupResourceForks will reassemble each pair of files beneath that point.


Warnings

Backups are important, test them regularly!

Using rsync option --hard-links and --hfs-mode together would cause file corruption on the destination filesystem. The current version of rsync+hfsmode will print an error message and exit if you try. Future versions might resolve this problem more elegantly.


Usage Examples

To back up your home directory:

   bash$ rsync --archive --delete --verbose \
               --hfs-mode=appledouble \
               --delete-excluded \
               --exclude=.Trash \
               --exclude=Cache \
               --exclude=Caches \
               /Users/username fileserver:/backups/username
To back up your whole machine:
   bash$ rsync --archive --delete --verbose \
               --hfs-mode=appledouble \
               --delete-excluded \
               --exclude=/dev/fd \
               --exclude=/.Trashes \
               --exclude=/Network \
               --exclude=.Trash \
               --exclude=Cache \
               --exclude=Caches \
               / fileserver:/backups/machinename

Marion Bates has also written an article explaining how to use rsync+hfsmode in a backup strategy.


Possible Future Enhancements


Contact Info

Let me know if you have any problems (or great successes) with rsync+hfsmode: reynhout@quesera.com

thanks