Presentation on theme: "Directory Delegations in NFSv4 CITI University of Michigan Ann Arbor Brian Wickman."— Presentation transcript:
Directory Delegations in NFSv4 CITI University of Michigan Ann Arbor Brian Wickman
page2 NFS version 4 u NFSv4 introduces state! u OPEN and CLOSE u integrated locking u file delegations u Delegation Paradigm u Server-driven optimizations visible to the client but transparent to applications. u More client responsibility; but better performance!
page3 NFSv4 File Delegations u Server-driven optimization: Server has final authority over delegating ability to clients u Transparent to applications: No special application-level tooling required, otherwise delegation utility would be undermined. u Better performance: Local locking! Smart caching! Less latency! Less traffic!
page4 NFS Directory Delegations u Want to delegate some portion of directory state to client u What to delegate? u Potential state u Filename to filehandle bindings (DNLC) u File attributes u Anything else?
page5 How to Enable Directory Delegations? u Introduce pair of ops, OPENDIR/CLOSEDIR? u Better mirroring of file delegation semantics u Same interoperability concerns as delegating v4.0 servers speaking with v2/v3 clients. u Introduce single-op? GIVE_ME_DELEGATION_PRETTY_PLEASE? u Half as many ops to implement. Works, too. u How information-rich? Notifications? Asynchronous, synchronous, whaaaa?
page6 draft-ietf-nfsv4-directory-delegation-01.txt u Saadia’s proposal u New op (GET_DIR_DELEGATION) u New callback (CB_NOTIFY) u Delegations and notifications, all in one u Read-only, as specified u Information rich, fine-grained u Lots of new functionality to code, especially for server implementer
page7 draft-wickman-nfsv4-directory-delegations-00.txt u My proposal u New op (REQUEST_DELEGATION) u No new callbacks u No notifications, existing data structures unchanged u Read and write delegations specified u Minimal information, coarse-grained u Implementation more straightforward?
page8 Write directory delegations u Server doesn’t need to change its behavior, relies more on client implementer u If the client is clever it can u Issue directory updates asynchronously u Perform whole-tree write delegation on new directories u Effectively own file write delegations on all files within a write delegated directory
page9 Write delegations and STATEIDs u Overlooked in my draft, need errata u Directory ops such as READDIR or MKDIR do not reference a delegation stateid u Need PUT_STATEID function and “current” STATEID so that we can modify directories on which we hold delegations u Discussed at Connectathon…?
page10 Evaluating the proposals u For starters, how do you define better? u My take: how many ops can you service locally and at what extra cost, given common workloads? u So far, I’ve investigated this for read-only directory delegations and for two common(?) workloads.
page11 Steps to Evaluating Delegation Behavior u Passively capture traces from NFSv3 deployments with thousands of users (v4 deployments, anyone?) u Build an NFS (v3) metadata simulator u Instrument extra information into traces (e.g., OPEN, CLOSE, REQUEST_DELEGATION) u Reconstruct filesystem image on the fly u Observe delegation behavior! u Measure what you see
page12 Traces Used u From Dan Ellard’s PhD research u CAMPUS trace u Read dominated workload u Good test of notification behavior? (taking pulse of mail spool) u Digital UNIX server, one week, late October 2001, 242M v3 ops u EECS trace u Write-dominated research workload, lots of small files u NetApp filer, one week, late October 2001, 41M v3 ops
page13 Simple use of Directory Delegations u For fun, I emulated the v3 cache algorithm using the REQUEST_DELEGATION op u When first caching directory entries, REQUEST_DELEGATION u If clock expires and no revocation, do nothing u If revocation occurs, issue all LOOKUPs to server, synchronously until clock expiration u Reacquire delegation at clock expiration
page14 Simple use of Directory Delegations u Use liberal server policy for granting directory delegations (i.e., always-grant) u In both traces, >90%** of LOOKUPs are serviced locally and guaranteed consistent u In both traces, >95%** of directory cache validation traffic is eliminated ** - with unbounded lookup caches
page15 What about with notifications? u I also implemented the same algorithm but with GET_DIR_DELEGATION u We can service a bunch of extra LOOKUPs (practically all) because the server notifies us when our lookup cache changes and what is entailed in each change, rather than revoke u How much can we service and at what cost?
page16 Not much CAMPUSEECS Extra LOOKUPs serviced locally 162K14K Percentage of total LOOKUPs2.1%0.4% Notifications received9M164K Ratio of Notifications to extra LOOKUPs 56x12x
page17 FILE_ATTR notifications u If FILE_ATTR notifications are added, the ratio goes from 56x to 500x for CAMPUS u Millions of DIR_ATTR notifications to service tens of thousands of LOOKUPs u Tens of millions of FILE_ATTR notifications (67M) to service millions of GETATTRs (2.3M)
page18 Reducing Notification Traffic u Can we win if using asynchronous notifications? u If nothing else, we can reduce traffic. u Because the client can specify a notification window, the server has the opportunity to eliminate self- nullifying updates (e.g., ADD fh; REMOVE fh) within that window. u Asynchronous notifications would still mean less LOOKUP consistency than no notifications at all, as long as revocations are synchronous
page19 Reducing Notification Traffic
page20 Another Drawback of Notifications u How many notifications refer to directory objects whose data or metadata we are actively accessing? u CAMPUS: 4% u EECS:<1% u Are we taking sips from a fire-hose? u Also, knowing the existence of a change may be sufficient more often than knowing the change itself.
page21 Summary u A simple directory delegation scheme without notifications gives immediate improvements in latency and caching, in the read-only case u Notifications have the tendency to create lots of information that is otherwise unused by the client u Still much more work to be done…
page22 TODO u Start looking at v4 traces. Of course, I need v4 traces, especially Windows v4 traces Anyone want to help? u Simulating write directory delegations u Looking at other lookup cache maintenance algorithms (instead of just clock-based) u Adding more variables to the simulator (cache eviction, for example) u Implementing directory delegations for real