+ + Servers each specify a server scope value in the form + of an opaque string eir_server_scope returned as part of + the results of an EXCHANGE_ID operation. The purpose of + the server scope is to allow groups of servers to + indicate to clients that a set of servers sharing the + same server scope value have arranged to use compatible + values of otherwise opaque identifiers so that the identifiers + generated by one server of that set may be presented to + another of that same scope. + + + The use of such compatible values does not imply that + a value generated by one server will always be accepted + by another. In most cases, it will not. However, a + server will not accept a value generated by another + inadvertently. When it does accept it, it will be because + it is recognized as valid and carrying the same meaning + as on another server of the same scope. + + + When servers are of the same server scope, this compatibility + of values applies to the follow identifiers: + + + Filehandle values. A filehandle value accepted by two + servers of the same server scope denotes the same object. + A write done to one server is reflected immediately + in a read done to the other and locks obtained on one + server conflict with those requested on the other. + + + Session ID values. A session ID value accepted by two + server of the same server scope denotes the same session. + + + Client ID values. A client ID value accepted as valid by + two servers of the same server scope is associated with + two clients with the same client owner and verifier. + + + State ID values when the corresponding client ID is recognized + as valid. If the same stateid value is accepted as valid + on two servers of the same scope and the client ID's on + the two servers represent the same client owner and + verifier, then the two state ID values designate the + same set of locks and are for the same file + + + Server owner values. When the server scope values are + the same, server owner value may be validly compared. + In cases where the server scope are different, server + owner values are treated as different even if they + contain all identical bytes. + + + + + The co-ordination among servers required to provide such + compatibility can be quite minimal, and limited to a simple + partition of the ID space. The recognition of common values + requires additional implementation, but this can be tailored + to the specific situations in which that recognition is + desired. + + + Clients will have occasion to compare the server scope values + of multiple servers under a number of circumstances, each of + which will be discussed under the appropriate functional + section. + + + When server owner values received in response to + EXCHANGE_ID operations issued to multiple network + addresses are compared for the purpose of determining + the validity of various forms of trunking, as described + in . + + + When network or server reconfiguration causes the same + network address to possibly be directed to different + servers, with the necessity for the client to determine + when lock reclaim should be attempted, as described + in + + + When file system migration causes the transfer of + responsibility for a file system between servers and + the client needs to determine whether state has been + transferred with the file system and whether a + client may reclaim it on a similar basis as in the + case of server reboot. + + + + + When two replies from EXCHANGE_ID each from two different + server network addresses have the same server scope, there + are a number of ways a client can validate that the common + server scope is by benign intent. + + + If both EXCHANGE_ID requests were sent with RPCSEC_GSS + authentication and the server principal is the same for + both targets, the equality of server scope is validated. + + + A second option for verification is to use SP4_SSV + protection in a fashion similar to the verification + of client ID trunking. When the client sends + EXCHANGE_ID it specifies SP4_SSV protection. The + first EXCHANGE_ID the client sends always has to + be confirmed by a CREATE_SESSION call. The client + then sends SET_SSV. Later the client + sends EXCHANGE_ID to a second destination + network address different form the one the first + EXCHANGE_ID was sent to. + The client checks that each EXCHANGE_ID reply has the + same eir_server_scope. If so, the client verifies the + claim by issuing a CREATE_SESSION to the second + destination address, protected with RPCSEC_GSS integrity + using an RPCSEC_GSS handle returned by the second + EXCHANGE_ID. If the server accepts the CREATE_SESSION + request, and if the client verifies the RPCSEC_GSS + verifier and integrity codes, then the client has + proof the second server knows the SSV, and thus + the two servers are co-operating for the purposes of + maintaining compatible ID spaces as indicated by a + common server scope. + + + If neither of the two methods provides verification, + the client may accept the appearance of the second + server in fs_locations or fs_locations_info attribute + for a relevant file system. For example, if there is + migration event for a particular particular file system + or there are locks to be reclaimed on a particular file + system, the attributes for that particular file system + may be used. The client sends the GETATTR request to + the first server for the fs_locations or + fs_locations_info attribute with RPCSEC_GSS + authentication. It may need to do this in advance + of the need to verify the common server scope. + If the client successfully authenticates the reply + to GETATTR, and the GETATTR request and reply containing + the fs_locations or fs_locations_info attribute refers + to the second server, then the equality of server scope + is supported. A client may choose to limit the use of + this form of support to information relevant to the + specific file system involved (e.g. a file system + being migrated). + + + +

Trunking is the use of multiple connections between a client and server in order to increase the speed of data transfer. NFSv4.1 supports two types of trunking: - session trunking and client ID trunking. NFSv4.1 - repliers and requesters MUST support session trunking. - NFSv4.1 servers MAY support client ID trunking. - NFSv4.1 clients MUST support client ID trunking. - + session trunking and client ID trunking. + + + NFSv4.1 + servers MUST support both forms of trunking within + the context of a single server network address and + MUST support both forms within the context of the + set of network addresses used to access a single server. + NFSv4.1 servers in a clustered configuration MAY allow + network addresses for different servers to use client ID + trunking. + + + Clients may use either form of trunking as long as they + do not, when trunking between different server network + addresses, violate the servers' mandates as to the + kinds of trunking to be allowed (see below). With regard + to callback channels, the client MUST allow the server to + choose among all callback channels valid for a given + client ID and MUST support trunking when the connections + supporting the backchannel allow session or client ID + trunking to be used for callbacks Session trunking is essentially the association of multiple connections, each with potentially different target and/or source - network addresses, to the same session. + network addresses, to the same session. When the target network + addresses (server addresses) of the two connections are the same, + the server MUST + support such session trunking. When the target network addresses + are different, the server MAY indicate such support using the + data returned by the EXCHANGE_ID operation (see below). Client ID trunking is the association of multiple - sessions to the same client ID, major server owner ID + sessions to the same client ID. Servers MUST support client ID + trunking for two target network addresses whenever they allow + session trunking for those same two network addresses. + In addition, a server MAY, by presenting the same + major server owner ID (), and server scope - (). When two + () allow an additional + case of client ID trunking. When two servers return the same major server owner and server - scope it means the two servers are cooperating on + scope, it means that the two servers are cooperating on locking state management which is a prerequisite for client ID trunking. - Understanding and distinguishing session and client ID + Understanding and distinguishing when the client is + allowed to use session and client ID trunking requires understanding how the results of the EXCHANGE_ID () operation identify a server. Suppose a client sends EXCHANGE_ID over two different connections each with a possibly different target - network address but each EXCHANGE_ID with the same + network address but each EXCHANGE_ID operation has the same value in the eia_clientowner field. If the same NFSv4.1 server is listening over each connection, then each EXCHANGE_ID result MUST return the same @@ -1749,8 +1940,8 @@ to verification, see later in this section), and it can use each connection to trunk requests and - replies. The question is whether session trunking - and/or client ID trunking applies. + replies. The client's choice is whether session trunking + or client ID trunking applies. @@ -1770,12 +1961,16 @@ of the connections, which associates the connection to the session. If there is a session for the tuple, the client can send BIND_CONN_TO_SESSION to associate - the connection to the session. (Of course, if the client - does not want to use session trunking, it can - invoke CREATE_SESSION on the connection. This will result - in client ID trunking as described below.) - - + the connection to the session. + + Of course, if the client + does not desire to use session trunking, it is not + required to do so. It can invoke + CREATE_SESSION on the connection. This will result + in client ID trunking as described below. It can also + decide to drop the connection if it does not choose to + use trunking. + @@ -1792,24 +1987,26 @@ each connection with different sessions, where each session is associated with the same server. - + Of course, even if the eir_server_owner.so_minor_id fields do match, the client is free to employ client ID trunking instead of session trunking. - + The client completes the act of client ID trunking by invoking CREATE_SESSION on each connection, using the same client ID that was returned in eir_clientid. These invocations create two sessions and also associate - each connection with each session. + each connection with its respective session. The client + is free to choose not to use client ID trunking by simply + dropping the connection at this point. - + When doing client ID trunking, locking state - is shared across sessions associated with the same + is shared across sessions associated with that same client ID. This requires the server to coordinate state across sessions. @@ -1818,6 +2015,19 @@ + + The client should be prepared for the possibility + that eir_server_owner values may be different on + subsequent EXCHANGE_ID requests made to the same + network address, as a result of various sorts of + reconfiguration events. When this happens and the + changes result in the invalidation of previously + valid forms of trunking, the client should cease + to use those forms, either by dropping connections + or by adding sessions. For a discussion of lock + reclaim as it relates to such reconfiguration events, + see . + When two servers over two connections claim @@ -1903,8 +2113,8 @@ be confirmed by a CREATE_SESSION call. The client then sends SET_SSV. Later the client sends EXCHANGE_ID to a second destination - network address than the first EXCHANGE_ID was - sent with. + network address different from the one the first + EXCHANGE_ID was sent to. The client checks that each EXCHANGE_ID reply has the same eir_clientid, eir_server_owner.so_major_id, and eir_server_scope. If so, the client verifies the @@ -1915,8 +2125,8 @@ request, and if the client verifies the RPCSEC_GSS verifier and integrity codes, then the client has proof the second server knows the SSV, and thus - the two servers are the same for the purposes of - client ID trunking. + the two servers are co-operating for the purposes of + specifying server scope and client ID trunking. @@ -3799,6 +4009,14 @@ The server purges the session of a client that has been inactive for a very extended period of time. + + As a result of configuration changes among a set of clustered + servers, a network address previously connected to one + server becomes connected to a different server which has + no knowledge of the session in question. Such a configuration + change will generally only happen when the original server + ceases to function for a time. + Loss of reply cache is equivalent to loss of session. The replier indicates loss of session to the requester @@ -3815,10 +4033,111 @@ invokes BIND_CONN_TO_SESSION using the session ID. Otherwise, it invokes SEQUENCE. If BIND_CONN_TO_SESSION or SEQUENCE returns NFS4ERR_BADSESSION, the - client knows the session was lost. If the connection survives + client knows the session is not available to it when communicating + with that network address. If the connection survives session loss, then the next SEQUENCE operation the client sends over the connection will get back NFS4ERR_BADSESSION. The client again knows the session was lost. + + + Here is one suggested algorithm for the client when it gets + NFS4ERR_BADSESSION. It is not obligatory in that, if a + client does not want to take advantage of such features as + trunking, it may omit parts of it. However, it is a useful + example which draws attention to various possible recovery + issues: + + + If the client has other connections to + other server network addresses + associated with the same session, attempt + a COMPOUND with a single operation, SEQUENCE, + on each of the other connections. + + + If the attempts succeed, the session is still alive, + and this is a strong indicator the server's + network address has moved. + The client might send an EXCHANGE_ID on the + connection that returned NFS4ERR_BADSESSION + to see if there are opportunities for client ID + trunking (i.e. the same client ID and so_major are + returned). The client might use DNS to see if + the moved network address was replaced with another, + so that the performance and availability benefits of + session trunking can continue. + + + If the SEQUENCE requests fail with NFS4ERR_BADSESSION + then the session no longer exists on any of the + server network addresses the client has connections + associated with that session ID. It is possible the + session is still alive and available on other + network addresses. The client sends an EXCHANGE_ID + on all the connections to see if the server owner + is still listening on those network addresses. + If the same server owner is returned, but a new + client ID is returned, this is a strong + indicator of a server restart. If both the same + server owner and same client ID are + returned, then this is a strong indication + that the server did delete the session, and the + client will need to send a CREATE_SESSION if it + has no other sessions for that client ID. + If a different server owner is returned, + the client can use DNS to find + other network addresses. If it does not, or if + DNS does not find any other addresses for the server, + then the client will be unable to provide NFSv4.1 + service, and fatal errors should be returned + to processes that were using the server. If the + client is using a "mount" paradigm, unmounting + the server is advised. + + + If the client knows of no other connections associated + with the session ID, and server network addresses that + are, or have been associated with the session ID, + then the client can use DNS to find + other network addresses. If it does not, or if + DNS does not find any other addresses for the server, + then the client will be unable to provide NFSv4.1 + service, and fatal errors should be returned + to processes that were using the server. If the + client is using a "mount" paradigm, unmounting + the server is advised. + + + + + If there is a reconfiguration event which results in the + same network being assigned to servers where the + server_scope value is different, it cannot be guaranteed + that a session ID generated by the first will be recognized + as invalid by the first. Therefore, in managing server + reconfigurations among servers with different server scope + values, it is necessary to make sure that all clients have + disconnected from the first server before effecting + the reconfiguration. Nonetheless, clients should not + assume that this requirement will always be adhered to + in effecting server reconfigurations to deal with unexpected + events. Even where a session ID is inappropriately + recognized as valid, it is likely that either the connection + will not be recognized as valid, or that a sequence value + for a slot will not be correct. Therefore, when a client + receives results indicating such unexpected errors, the use of + EXCHANGE_ID to determine the current server configuration + and present the client to the server is recommended. + + + A variation on the above is that after a server's network + address moves, there is no NFSv4.1 server listening. E.g. no + listener on port 2049, the NFSv4 server returns + NFS4ERR_MINOR_VERS_MISMATCH, the NFS server server returns a + PROG_MISMATCH error, the RPC listener on 2049 returns + PROG_MISMATCH, or attempts to re-connect to the network address + timeout. These should be treated as equivalent to SEQUENCE + returning NFS4ERR_BADSESSION for these purposes. When the client detects session loss, it must call CREATE_SESSION diff -rbwu draft-25/nfsv41_middle_datatypes.xml draft-26/nfsv41_middle_datatypes.xml --- draft-25/nfsv41_middle_datatypes.xml Thu Aug 21 20:31:08 2008 +++ draft-26/nfsv41_middle_datatypes.xml Fri Aug 29 08:28:37 2008 @@ -1,4 +1,4 @@ - +

@@ -170,7 +170,7 @@ The netaddr4 data type is used to identify network transport endpoints. The r_netid and r_addr fields respectively contain a netid and uaddr. The netid and uaddr concepts are defined in - in . The netid and uaddr formats for + . The netid and uaddr formats for TCP over IPv4 and TCP over IPv6 are defined in , specifically Tables 2 and 3 and Sections 3.2.3.3 and 3.2.3.4. diff -rbwu draft-25/nfsv41_middle_errors.xml draft-26/nfsv41_middle_errors.xml --- draft-25/nfsv41_middle_errors.xml Thu Aug 7 06:30:01 2008 +++ draft-26/nfsv41_middle_errors.xml Wed Sep 3 11:32:50 2008 @@ -1,4 +1,4 @@ - + @@ -899,9 +899,11 @@

- A stateid designates recallable locking state of any type - that has been revoked due to the failure of the client - to return the lock, when it was recalled. + A stateid designates recallable locking state of + any type (delegation or layout) that has been + revoked due to the failure of the client to return + the lock, when it was recalled. +

- A session ID was specified which does not exist. + A session ID that was specified is not known to the server + to which the operation is addressed.

@@ -1078,18 +1078,20 @@ cooperated in state management. If the scope values match, then this indicates the servers have cooperated in assigning client IDs to the point that they will reject client IDs that - refer to state they do not know about. + refer to state they do not know about. See + for more information about + the use of server scope. In the case of migration, the servers involved in the migration of a file system SHOULD transfer all server state - from the original to the - new server. When this is done, it must be done in a way that - is transparent to the client. - With replication, such a degree of common state is typically not - the case. Clients, however - should use the information provided by the eir_server_scope - returned by EXCHANGE_ID + from the original to the new server. When this is done, + it must be done in a way that is transparent to the client. + With replication, such a degree of common state is + typically not the case. Clients, however should use + the information provided by the eir_server_scope + returned by EXCHANGE_ID (as modified by the validation + procedures described in ) to determine whether such sharing may be in effect, rather than making assumptions based on the reason for the transition. @@ -1103,7 +1105,13 @@ client ID, then the client may use the existing stateids associated with that client ID for the old file system instance in connection with that same client ID in connection with - the transitioned file system instance. + the transitioned file system instance. If the client in + question already had a client ID on the target system, it + may interrogate the state ID values from the source system + under that new client ID, with the assurance that if they + are accepted as valid, then they represent validly transferred + lock state for the source file system, transferred to the + target server. When the two servers belong to the same @@ -1129,10 +1137,18 @@ When the two file system instances are on servers that do not share a server scope value, the client must establish a new client ID on the destination, if it does not - have one already, and reclaim locks if - possible. In this case, old stateids and client IDs should + have one already, and reclaim locks if allowed by the server. + In this case, old stateids and client IDs should not be presented to the new server since there is no assurance that they will not conflict with IDs valid on that server. + Note that in this case lock reclaim may be attempted even + when the servers involved in the transfer have different + server scope values (see + for the contrary case of reclaim after server reboot. + Servers with different server scope values may co-operate + to allow reclaim for locks associated with the transfer of + a filesystem even if they do not co-operate sufficiently + to share a server scope. In either case, when actual locks are not known to be maintained, @@ -1286,9 +1302,8 @@ If state has not been transferred transparently, either because the associated servers are shown as having different - eir_server_scope strings - or because the client ID is rejected when presented to - the new server, + eir_server_scope strings or because the client ID + is rejected when presented to the new server, the client should fetch the value of lease_time on the new (i.e. destination) server, and use it for subsequent locking requests. However the server diff -rbwu draft-25/nfsv41_middle_pnfs.xml draft-26/nfsv41_middle_pnfs.xml --- draft-25/nfsv41_middle_pnfs.xml Thu Aug 21 20:31:09 2008 +++ draft-26/nfsv41_middle_pnfs.xml Fri Aug 29 08:28:37 2008 @@ -1,4 +1,4 @@ - +

@@ -1515,8 +1515,8 @@ If sr_status_flags from the metadata server has - SEQ4_STATUS_RESTART_RECLAIM_NEEDED set (or SEQUENCE returns - NFS4ERR_STALE_CLIENTID, or SEQUENCE returns NFS4ERR_BAD_SESSION and + SEQ4_STATUS_RESTART_RECLAIM_NEEDED set + (or SEQUENCE returns NFS4ERR_BAD_SESSION and CREATE_SESSION returns NFS4ERR_STALE_CLIENTID) then the metadata server has restarted, and the client SHOULD recover using the methods described in . diff -rbwu draft-25/nfsv41_middle_state_mgmt.xml draft-26/nfsv41_middle_state_mgmt.xml --- draft-25/nfsv41_middle_state_mgmt.xml Thu Aug 21 20:31:09 2008 +++ draft-26/nfsv41_middle_state_mgmt.xml Wed Sep 3 12:11:25 2008 @@ -1,4 +1,4 @@ - +

@@ -700,7 +700,7 @@ The client MUST either recover from the ensuing - NFS4ERR_NOGRACE errors, or it MUST ensure that + NFS4ERR_NO_GRACE errors, or it MUST ensure that despite transport level retransmission intervals that exceed the lease_time, nonetheless a SEQUENCE operation is sent that renews the lease @@ -807,7 +807,7 @@ associated with a now invalid client ID. In this case the client will receive either an NFS4ERR_BADSESSION or NFS4ERR_DEADSESSION error, and any attempt to attach a new - session to the existing client ID will result in an + session to that invalid client ID will result in an NFS4ERR_STALE_CLIENTID error. @@ -1077,6 +1077,42 @@ instantiation. This allows the client state obtained during the previous server instance to be reliably re-established. + + The possibility exists, that because of server configuration + events, the client will be communicating with a server + different than the one on which the locks were obtained, as + shown by the combination of eir_server_scope and + eir_server_owner. This leads to the issue of if and when + the client should attempt to reclaim locks previously obtained + on what is being reported as a different server. The rules + to resolve this question are as follows: + + + If the server scope is different the client should not + attempt to reclaim locks. In this situation no lock + reclaim is possible. Any attempt to re-obtain the locks + with non-reclaim operations is problematic since there is + no guarantee that the existing filehandles will be recognized + by the new server, or that if recognized, they denote the + same objects. It is best to treat the locks as having been + revoked by the reconfiguration event. + + + If the server scope is the same, the client should attempt + to reclaim locks, even if the eir_server_owner value is + different. In this situation, it is the responsibility + of the server to return NFS4ERR_NO_GRACE if it cannot + provide correct support for lock reclaim operations, + including the prevention of edge conditions. + + + + + The eir_server_owner field is not used in making this + determination. Its function is to specify trunking + possibilities for the client (see ) + and not to control lock reclaim. +

When the server adopts a finer-grained approach to revocation - of locks when lease have expired, only a subset of stateids + of locks when leases have expired, only a subset of stateids will normally become invalid during a network partition. - When the client can communicate with - the server after such a network partition heals, the status returned by - the SEQUENCE operation will indicate a partial loss of locking state (SEQ4_STATUS_EXPIRED_SOME_STATE_REVOKED). + When the client can communicate with the server after such a + network partition heals, the status returned by the SEQUENCE + operation will indicate a partial loss of locking state + (SEQ4_STATUS_EXPIRED_SOME_STATE_REVOKED). In addition, operations, including I/O submitted by the client, with the now invalid stateids will fail with the server returning the error NFS4ERR_EXPIRED. Once the client learns of - the loss of locking state, it will use the TEST_STATEID operation on - all of its stateids to + the loss of locking state, it will use the TEST_STATEID operation + on all of its stateids to determine which locks have been lost and then suitably notify the applications that held the invalidated locks. The client can then release the invalidated locking @@ -1186,7 +1223,8 @@ Network partition between client A and server heals. - Client A connects to new server instance and finds out about server restart. + Client A connects to new server instance and finds out about + server restart. Client A reclaims its lock within the server's grace period. @@ -1230,7 +1268,8 @@ Network partition between client A and server heals. - Client A connects to new server instance and finds out about server restart. + Client A connects to new server instance and finds out about + server restart. Client A reclaims its lock within the server's