Filesystem caching is a great tool for improving performance, but it is important to balance performance with data safety. Caching over NFS involves caches at several different levels, so it is not immediately obvious which combination of options ensures a good compromise between performance and safety.
the NFS client has the
async mount option, which caches writes in the client's RAM until
certain conditions are met:
delays sending application writes to the server until any of these events occur:
The NFS client treats the sync mount option differently than some other file systems (refer to mount(8) for a description of the generic sync and async mount options). If neither sync nor async is specified (or if the async option is specified), the NFS client delays sending application writes to the server until any of these events occur:
- Memory pressure forces reclamation of system memory resources.
- An application flushes file data explicitly with sync(2), msync(2), or fsync(3).
- An application closes a file with close(2).
- The file is locked/unlocked via fcntl(2).
In other words, under normal circumstances, data written by an application may not immediately appear on the server that hosts the file.
If the sync option is specified on a mount point, any system call that writes data to files on that mount point causes that data to be flushed to the server before the system call returns control to user space. This provides greater data cache coherence among clients, but at a significant performance cost.
See nfs(5) for more details. In other words, when writing data to a file or set of files, rather than flush to the server on each write(2) call, the system waits until the file is closed or the application expliticly calls fsync(3) or another sync function. Since you're relying on the application correctly request its data to be synced, I was concerned about relying on this cache in a general circumstance, when potentially poorly-written applications could be never syncing their data. However, given that close(2) causes the data to be synced, this seems like a non-issue, and asking on the linux-nfs mailing list clarified in more detail how this works:
In NFSv3, the close() will cause the client to flush all data to stable storage. The client will also flush data to stable storage on a chmod, since that could potentially affect its ability to write back the data. It will not bother to do so for rename. An application should normally be able to rely on the data being safely on disk in both these situations provided that the server honours the NFS protocol (with a caveat that an ill-timed 'kill -9' could interrupt the process of flushing).
All metadata operations such as create, chmod, rename, etc. will cause the server to flush the file metadata to disk assuming that you set the (highly recommended) "sync" export option. If "sync" is set, the server will also honour COMMIT requests by flushing the data to stable storage.
If, OTOH, your server lists the "async" export option as being set, then COMMIT is considered a no-op, and it will not bother to explicitly flush metadata operations to stable storage. Performance will scream, but be prepared to lose data if that server crashes. This is all technically a violation of the NFS spec, however you have been given rope...
async on the client is safe and will provide a pretty significant performance boost.
It's also important to look at
hard mounts. A soft mount will give up attempting to write to
a server that is unavailable after a specific timeout and number of retries. In my experience, this hasn't
worked well and I often end up with processes stuck in uninterruptable sleep blocking on an NFS mountpoint
anyway. As per the manpage,
hard is highly recommended to ensure data integrity:
Determines the recovery behavior of the NFS client after an NFS request times out. If neither option is specified (or if the hard option is specified), NFS requests are retried indefinitely. If the soft option is specified, then the NFS client fails an NFS request after retrans retransmissions have been sent, causing the NFS client to return an error to the calling application.
NB: A so-called "soft" timeout can cause silent data corruption in certain cases. As such, use the soft option only when client responsiveness is more important than data integrity. Using NFS over TCP or increasing the value of the retrans option may mitigate some of the risks of using the soft option.
Note that the
intr option allows you to interrupt a request waiting on a
hard NFS mount by sending it
SIGKILL signal. However, on kernels newer than 2.6.25 this is provided by default, and the
option is deprecated. You should still be aware of it though in case you are working with an older kernel.
Given my poor experience using
soft (the timeouts don't seem to actually work) and the increased
risk of data loss,
hard seems like the most appropriate option to use. The common problem mentioned
hard is if the server goes away (e.g hardware failure and it is down for an extended period
of time), there used to be no way to unmount that mountpoint or let processes blocking on it complete. There
are now a few ways to mitigate this:
- bring up a fake NFS server on the same IP address as the offline server, which can then reject the requests that are waiting
for a response. I've even seen this done with a local ethernet alias interface, e.g
ifconfig eth0:nfstmp <server ip>/32 up
- use the
fsid=<unique number>on the server side in
/etc/exports. This creates a static unique identifier for the export, so you won't get a "Stale NFS File Handle" error on the client if the server is restarted or goes offline. These ID numbers must be unique and be greater than 1, since 1 is used by NFSv4 as the root export.
- try "lazy" unmounting the mountpoint with
umount -l /path/to/mountpoint
If the above fails to work, you will probably have to reboot the client in order to clear the stuck mountpoint.
Confusingly, the NFS server options (found in
/etc/exports) are also called
async, see exports(5) for details:
async This option allows the NFS server to violate the NFS protocol and reply to requests before any changes made by that request have been committed to stable storage (e.g. disc drive).
Using this option usually improves performance, but at the cost that an unclean server restart (i.e. a crash) can cause data to be lost or corrupted.
sync Reply to requests only after the changes have been committed to stable storage (see async above).
In releases of nfs-utils up to and including 1.0.0, the async option was the default. In all releases after 1.0.0, sync is the default, and async must be explicitly requested if needed. To help make system administrators aware of this change, exportfs will issue a warning if neither sync nor async is specified.
Thus if you use
async on the server side, the data will be confirmed to be written as soon as it hits
the server's RAM. In the case of a power failure, this data would be lost. Conversely,
sync waits for
the data to be written to the disk or other stable storage (and confirmed) before returning a success. It is
sync is the appropriate option to use on the server side.
In conclusion, these options seem to provide a good balance of stability and performance when using NFS:
- Client Side:
- hard - forces requests to retry indefinitely to avoid corruption
- intr - this allows
hardmounts to be interrupted (though is unnecessary on kernels newer than 2.6.25)
- async - queue up writes and flush them in logical groups for more efficient writing
- tcp - using TCP is more reliable than UDP since it requires confirmation of receipt of packets
- Server Side:
- fsid - specifies a unique, static identifier for this export; see above for more details
- sync - ensures that data is really flushed to stable storage when the server says it is