1Reexporting NFS filesystems
2===========================
3
4Overview
5--------
6
7It is possible to reexport an NFS filesystem over NFS.  However, this
8feature comes with a number of limitations.  Before trying it, we
9recommend some careful research to determine whether it will work for
10your purposes.
11
12A discussion of current known limitations follows.
13
14"fsid=" required, crossmnt broken
15---------------------------------
16
17We require the "fsid=" export option on any reexport of an NFS
18filesystem.  You can use "uuidgen -r" to generate a unique argument.
19
20The "crossmnt" export does not propagate "fsid=", so it will not allow
21traversing into further nfs filesystems; if you wish to export nfs
22filesystems mounted under the exported filesystem, you'll need to export
23them explicitly, assigning each its own unique "fsid= option.
24
25Reboot recovery
26---------------
27
28The NFS protocol's normal reboot recovery mechanisms don't work for the
29case when the reexport server reboots because the source server has not
30rebooted, and so it is not in grace.  Since the source server is not in
31grace, it cannot offer any guarantees that the file won't have been
32changed between the locks getting lost and any attempt to recover them.
33The same applies to delegations and any associated locks.  Clients are
34not allowed to get file locks or delegations from a reexport server, any
35attempts will fail with operation not supported.
36
37Filehandle limits
38-----------------
39
40If the original server uses an X byte filehandle for a given object, the
41reexport server's filehandle for the reexported object will be X+22
42bytes, rounded up to the nearest multiple of four bytes.
43
44The result must fit into the RFC-mandated filehandle size limits:
45
46+-------+-----------+
47| NFSv2 |  32 bytes |
48+-------+-----------+
49| NFSv3 |  64 bytes |
50+-------+-----------+
51| NFSv4 | 128 bytes |
52+-------+-----------+
53
54So, for example, you will only be able to reexport a filesystem over
55NFSv2 if the original server gives you filehandles that fit in 10
56bytes--which is unlikely.
57
58In general there's no way to know the maximum filehandle size given out
59by an NFS server without asking the server vendor.
60
61But the following table gives a few examples.  The first column is the
62typical length of the filehandle from a Linux server exporting the given
63filesystem, the second is the length after that nfs export is reexported
64by another Linux host:
65
66+--------+-------------------+----------------+
67|        | filehandle length | after reexport |
68+========+===================+================+
69| ext4:  | 28 bytes          | 52 bytes       |
70+--------+-------------------+----------------+
71| xfs:   | 32 bytes          | 56 bytes       |
72+--------+-------------------+----------------+
73| btrfs: | 40 bytes          | 64 bytes       |
74+--------+-------------------+----------------+
75
76All will therefore fit in an NFSv3 or NFSv4 filehandle after reexport,
77but none are reexportable over NFSv2.
78
79Linux server filehandles are a bit more complicated than this, though;
80for example:
81
82        - The (non-default) "subtreecheck" export option generally
83          requires another 4 to 8 bytes in the filehandle.
84        - If you export a subdirectory of a filesystem (instead of
85          exporting the filesystem root), that also usually adds 4 to 8
86          bytes.
87        - If you export over NFSv2, knfsd usually uses a shorter
88          filesystem identifier that saves 8 bytes.
89        - The root directory of an export uses a filehandle that is
90          shorter.
91
92As you can see, the 128-byte NFSv4 filehandle is large enough that
93you're unlikely to have trouble using NFSv4 to reexport any filesystem
94exported from a Linux server.  In general, if the original server is
95something that also supports NFSv3, you're *probably* OK.  Re-exporting
96over NFSv3 may be dicier, and reexporting over NFSv2 will probably
97never work.
98
99For more details of Linux filehandle structure, the best reference is
100the source code and comments; see in particular:
101
102        - include/linux/exportfs.h:enum fid_type
103        - include/uapi/linux/nfsd/nfsfh.h:struct nfs_fhbase_new
104        - fs/nfsd/nfsfh.c:set_version_and_fsid_type
105        - fs/nfs/export.c:nfs_encode_fh
106
107Open DENY bits ignored
108----------------------
109
110NFS since NFSv4 supports ALLOW and DENY bits taken from Windows, which
111allow you, for example, to open a file in a mode which forbids other
112read opens or write opens. The Linux client doesn't use them, and the
113server's support has always been incomplete: they are enforced only
114against other NFS users, not against processes accessing the exported
115filesystem locally. A reexport server will also not pass them along to
116the original server, so they will not be enforced between clients of
117different reexport servers.
118