CA2205725C - Preventing conflicts in distributed systems - Google Patents
Preventing conflicts in distributed systems Download PDFInfo
- Publication number
- CA2205725C CA2205725C CA002205725A CA2205725A CA2205725C CA 2205725 C CA2205725 C CA 2205725C CA 002205725 A CA002205725 A CA 002205725A CA 2205725 A CA2205725 A CA 2205725A CA 2205725 C CA2205725 C CA 2205725C
- Authority
- CA
- Canada
- Prior art keywords
- token
- requestor
- requestee
- server
- data item
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2308—Concurrency control
- G06F16/2336—Pessimistic concurrency control approaches, e.g. locking or multiple versions without time stamps
- G06F16/2343—Locking methods, e.g. distributed locking or locking implementation details
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L9/00—Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols
- H04L9/40—Network security protocols
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y10—TECHNICAL SUBJECTS COVERED BY FORMER USPC
- Y10S—TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y10S707/00—Data processing: database and file management or data structures
- Y10S707/99951—File or database maintenance
- Y10S707/99952—Coherency, e.g. same view to multiple users
Abstract
A token-based protocol used to prevent conflicts between copies of the same data items residing in different servers of a replicated distributed system is provided.
The servers in the system with database replicas are provided with token pointers that are each associated with a token. Token pointers are used to track the current possessor of the token. Whenever a server receives or transfers a token, its corresponding token pointer is updated to reflect which server possesses the token. The use of token pointers advantageously eliminates the need for global cooperation by the servers to regenerate a lost token.
The servers in the system with database replicas are provided with token pointers that are each associated with a token. Token pointers are used to track the current possessor of the token. Whenever a server receives or transfers a token, its corresponding token pointer is updated to reflect which server possesses the token. The use of token pointers advantageously eliminates the need for global cooperation by the servers to regenerate a lost token.
Description
PREVENTING CONFLICTS IN DISTRIBUTED SYSTEMS
FIELD OF INVENTION:
The present invention relates to a distributed processing system and, more particularly, to preventing conflicts among copies of the same data item within the distributed system.
BACKGROUND OF INVENTION:
In distributed systems, access to data items is often facilitated by database servers. A database server provides storage of data items grouped into databases. An application running on a client station, such as a personal computer, can access a data item by connecting to the database server that stores the corresponding database. A
common scenario associated with distributed systems is that some clients are geographically located a great distance away from the server. Access to the database by these clients is generally by remote connections, e.g., dial up connections. Remote connections can incur long distance charges which result in added cost. Also, remote connections tend to increase system response time, thereby decreasing system performance. To improve performance and reduce cost in distributed systems, copies or replicas of the database are stored on multiple servers, each advantageously located in proximity to various groups of clients.
Such systems are known as replicated distributed systems.
However, the existence of multiple copies of the same data item requires database management protocols to prevent conflicts between copies.
"Conflicts", as used herein, refer to the scenario where two or more copies of the same data item each contain different modifications. One technique of preventing conflicts from occurring is to ensure 2 0 that all changes to the same data item are serialized and always applied to the most current version. This involves assigning a unique token, as a permission to make changes, for all copies of the same data item. Thus, the number of tokens in the system is equal to the number of data items in a database replica Unless the server possesses the token associated with a specific data item, it cannot perform modifications to that data item.
A problem with token-bases systems is that the token can be lost as a result of a server- failure or communication delays during token transfer.
Recovery of the lost token requires regeneration of the token by the system. Conventional token regeneration techniques, such as those described in S. Nishio, K. F. Li and E.
G. Manning, "A Time-Out Based Resilient Token Transfer Algorithm for Mutual Exclusion in Computer Networks", Proceedings of the IEEE Int. Conf. on Distributed Computing Systems, 1989, pp. 386-393 (broadcast-based regeneration), involve executing a distributed consensus protocol to identify the server that has the most current copy of the data item and to agree on which server regenerates the token. This necessitates numerous exchanges of broadcast messages among the servers, which increases message traffic in the network. Clearly, such protocols are impractical, particularly in low bandwidth networks (or where the network is operating at or near its bandwidth capacity) or where communication cost for connections to the network is high. In addition, broadcast regeneration protocols require global cooperation among the servers in the system, which means that all servers in the network must be simultaneously operational before the token can be regenerated. Thus, the unavailability of one or more servers creates problems in regenerating the token. Also, these protocols force all servers on the system to be interrupted from their normal operations while the token is being regenerated.
From the above discussion,_it is apparent that there is a need for an improved token-based protocol.
SUMMARY OF INVENTION:
In a replicated distributed system having copies of a database located on different servers, a token-based protocol for preventing conflicts between copies of the same data item residing on different servers is provided. As used herein, the term "servers" refers to servers with database replicas as well as to those servers on which 2 5 conflicts are to be prevented. The protocol advantageously enables regeneration of a lost token without the need for global cooperation among the servers. This advantage is achieved by providing servers with token pointers to keep track of the server possessing the token. In one embodiment, a token pointer is associated with each token in the system.
FIELD OF INVENTION:
The present invention relates to a distributed processing system and, more particularly, to preventing conflicts among copies of the same data item within the distributed system.
BACKGROUND OF INVENTION:
In distributed systems, access to data items is often facilitated by database servers. A database server provides storage of data items grouped into databases. An application running on a client station, such as a personal computer, can access a data item by connecting to the database server that stores the corresponding database. A
common scenario associated with distributed systems is that some clients are geographically located a great distance away from the server. Access to the database by these clients is generally by remote connections, e.g., dial up connections. Remote connections can incur long distance charges which result in added cost. Also, remote connections tend to increase system response time, thereby decreasing system performance. To improve performance and reduce cost in distributed systems, copies or replicas of the database are stored on multiple servers, each advantageously located in proximity to various groups of clients.
Such systems are known as replicated distributed systems.
However, the existence of multiple copies of the same data item requires database management protocols to prevent conflicts between copies.
"Conflicts", as used herein, refer to the scenario where two or more copies of the same data item each contain different modifications. One technique of preventing conflicts from occurring is to ensure 2 0 that all changes to the same data item are serialized and always applied to the most current version. This involves assigning a unique token, as a permission to make changes, for all copies of the same data item. Thus, the number of tokens in the system is equal to the number of data items in a database replica Unless the server possesses the token associated with a specific data item, it cannot perform modifications to that data item.
A problem with token-bases systems is that the token can be lost as a result of a server- failure or communication delays during token transfer.
Recovery of the lost token requires regeneration of the token by the system. Conventional token regeneration techniques, such as those described in S. Nishio, K. F. Li and E.
G. Manning, "A Time-Out Based Resilient Token Transfer Algorithm for Mutual Exclusion in Computer Networks", Proceedings of the IEEE Int. Conf. on Distributed Computing Systems, 1989, pp. 386-393 (broadcast-based regeneration), involve executing a distributed consensus protocol to identify the server that has the most current copy of the data item and to agree on which server regenerates the token. This necessitates numerous exchanges of broadcast messages among the servers, which increases message traffic in the network. Clearly, such protocols are impractical, particularly in low bandwidth networks (or where the network is operating at or near its bandwidth capacity) or where communication cost for connections to the network is high. In addition, broadcast regeneration protocols require global cooperation among the servers in the system, which means that all servers in the network must be simultaneously operational before the token can be regenerated. Thus, the unavailability of one or more servers creates problems in regenerating the token. Also, these protocols force all servers on the system to be interrupted from their normal operations while the token is being regenerated.
From the above discussion,_it is apparent that there is a need for an improved token-based protocol.
SUMMARY OF INVENTION:
In a replicated distributed system having copies of a database located on different servers, a token-based protocol for preventing conflicts between copies of the same data item residing on different servers is provided. As used herein, the term "servers" refers to servers with database replicas as well as to those servers on which 2 5 conflicts are to be prevented. The protocol advantageously enables regeneration of a lost token without the need for global cooperation among the servers. This advantage is achieved by providing servers with token pointers to keep track of the server possessing the token. In one embodiment, a token pointer is associated with each token in the system.
Each time a server receives or forwards a token to another server, it updates its token pointer associated with the token to identify the current possessor of that token. The protocol, in some instances, may not necessarily inform the server (server,) which requested the token as to which server actually possesses the token. For example, S if the server (serverb) identified in the requesting server's token pointer has subsequently transferred the token to another server (server) other than the requesting server, then servera's token pointer will no longer point to the server that actually possesses the token.
In this case, serverb requests the token from server on behalf of server,.
According to one embodiment, server sends the token to serverb which forwards it to server" the original token requestor. Thus, a token pointer provides to the token requestor a nexus, directly or indirectly (via an intermediate server), to the server possessing corresponding token.
The protocol involves at most two servers at any instant during a token request; i.e., the server issuing the token request (requestor) and the server receiving the token request (requestee). Due to this feature, there is no need for broadcasting messages to the servers in the system to determine which server has the token.
Furthermore, the protocol determines that a token is lost when the requestor or requestee thinks that the other has the token, but neither does. If the token is lost, then it last existed in either the requestor or requestee. In accordance with one embodiment, the server which possesses the more current copy of the data item is the server which last had the token and is 2 0 responsible to regenerate the token. As such, successful regeneration of a lost token need only to involve the requestor and requestee. Therefore, the protocol eliminates the need for global cooperation among the servers in the system.
BRIEF DESCRIPTION OF DRAWINGS:
FIG. 1 is a simplified illustration of a conventional replicated distributed system; .
2 5 FIG. 2 is an exemplary site within the distributed system of FIG. l;
Figs. 3a-f are examples of servers and the contents of its respective token pointer;
FIG. 4 is a flow diagram depicting the steps obtaining a token;
In this case, serverb requests the token from server on behalf of server,.
According to one embodiment, server sends the token to serverb which forwards it to server" the original token requestor. Thus, a token pointer provides to the token requestor a nexus, directly or indirectly (via an intermediate server), to the server possessing corresponding token.
The protocol involves at most two servers at any instant during a token request; i.e., the server issuing the token request (requestor) and the server receiving the token request (requestee). Due to this feature, there is no need for broadcasting messages to the servers in the system to determine which server has the token.
Furthermore, the protocol determines that a token is lost when the requestor or requestee thinks that the other has the token, but neither does. If the token is lost, then it last existed in either the requestor or requestee. In accordance with one embodiment, the server which possesses the more current copy of the data item is the server which last had the token and is 2 0 responsible to regenerate the token. As such, successful regeneration of a lost token need only to involve the requestor and requestee. Therefore, the protocol eliminates the need for global cooperation among the servers in the system.
BRIEF DESCRIPTION OF DRAWINGS:
FIG. 1 is a simplified illustration of a conventional replicated distributed system; .
2 5 FIG. 2 is an exemplary site within the distributed system of FIG. l;
Figs. 3a-f are examples of servers and the contents of its respective token pointer;
FIG. 4 is a flow diagram depicting the steps obtaining a token;
FIG. 5 is a flow diagram depicting the steps for GetToken routine;
FIG. 6 is a flow diagram depicting the steps for TokenHandler routine;
FIG. 7 is a flow diagram depicting the steps for ReleaseToken routine; and FIG. 8 is a flow diagram depicting the steps for RequestHandler routine.
DETAILED DESCRIPTION OF THE INVENTION:
The invention relates to a token-based protocol which prevents conflicts.
The protocol provides efficient regeneration of a token after if has been lost by eliminating the need for lost global cooperation of all servers with a database replica. As previously discussed, conventional token-based protocols require both global participation by all the servers as well as several rounds of broadcast messages to regenerate the token after if has been lost. This can lead to increased cost and system downtime.
FIG. 1 is a simplified illustration of a conventional replicated distributed system 100. Such systems are described in Berson, Client/Server Architecture, McGraw-Hill, Inc., 1992. Distributed system 100 comprises sites 110, 120, 130, and networked together by communication links 150a-150f. The physical implementation of the communication links is not important. For example, the communication links may be a local area network (LAN) of a wide area network (WAN) interconnecting sites at 2 0 geographically dispersed locations. The sites may also be "loosely connected", i.e., connected through dial-up connections of wireless connections such as satellite links.
FIG. 2 is an exemplary site 200 of the distributed system. As shown, the site includes a plurality of clients 240a-240c network to a server 210 by a LAN, MAN, or WAN. The clients, for example, may be personal computers, workstations, or terminals.
2 5 Other clients (not shown) may be connected via remote links. While the clients are shown to be associated with a particular site, it is understood that they may connect to other servers at other sites within the system using remote links, such as dial-up connections using modems, wireless connections such as satellite links, or connections via the Internet. Furthermore, it is understood that clients may have similar or different 3 0 configurations from each other, and other clients (not shown) may be included as desired.
In addition, one or more clients may be located on the server. Server 210 is, in some embodiments, a mainframe computer system, a workstation, or a personal computer that includes non-volatile storage device 225, such as magnetic disks or optical disks for data storage. The data being stored in the storage device 225 includes data files which are 5 organized into one or more groups called databases. Each individual data file in the databases is referred to as a data item.
Referring back to FIG. l, a copy or replica of the same database is stored in the memory of the servers located at sites 110, 120, 130 and 140. For sake of brevity, servers 110, 120, and 130 refer to servers located at sites 110, 120, 130 and 140, respectively. Furthermore, the servers at each site may also be referred to as nodes. The clients at the various nodes provide users with an interface to communicate with servers to read and update data items. An "update" refers to an operation which modifies of one or more data items. When updates are made to one replica, these updates are propagated to other servers during a replication session to make all replicas consistent with each other.
Conventional update propagation techniques such as those described in Lotus Notes Release 3 Application Developers Reference Manual, 1993, Lotus Development Corporation, Cambridge, MA., can be employed to maintain replica consistency.
Typically, update propagation is scheduled between a source and recipient server. The source or recipient server identifies which copies of data items in the source 2 0 replica are more current than their corresponding copies in the recipient replica. The source server then propagates the more recent copies or updates to the recipient server.
To facilitate identification as to which copy of the same data items are more recent, version information such as version numbers described in D. K. Gifford, "Weighted Voting for Replicated Data", Proceedings of the 7th ACM SIGOPS Symposium on 2 5 Operating Systems Principles, ( 1979), pp. 150-159, or other conventional techniques for identifying versions of data items are used Version information are associated with the data items. Each time a copy of the data item is modified, its corresponding version information is updated. By comparing version information of two copies of the same data item, a determination as to which copy is more current can be made. Whenever a server acquires a more current copy of a data item from another server, it also acquires the version information associated with that copy of the data item In accordance with the protocol of the invention, servers in the system having a database replica maintain pointers that are associated with each token. These pointers are herein referred to as TokenHolders. Since the number of tokens is equal to the number of data items in a replica, the number of TokenHolders in each server is also equal to the number of data items. In order to simplify the description of the protocol, illustrative embodiments are described using one TokenHolder per server (i.e., 1 data item per database replica). However, it will be understood that the protocol is applied on a per-data item basis. Also, to facilitate ease of discussion, a subscript variable (e.g.,) is used to denote any server in the system and its corresponding components.
The pointers are used to keep track of the server possessing the token. In one embodiment, TokenHolder identifies either 1) to which server server;
forwarded the token or 2) itself in the case where server; has the token. Server; modifies TokenHolder;
each time it transfers or receives the token. For example, after server 110 forwards the token to server 120, TokenHolder"o will contain information identifying server 120, such as server 120's server identification (SID) and TokenHolder,ZO will contain its own Sm.
When a client connected to server, sends a request to update, for example server;'s copy of data item x in server; it determines whether or not it has the token 2 0 associated with x. If server, has the token, it services the update request. Otherwise, server; sends a token request message to the server (server,) identified in TokenHolder;. If server, does not have the token, server, examines its TokenHoldera and forwards a token request message to serverb (which is identified in TokenHolderb) on behalf of the server;.
This request chain continues until the server possessing the token is reached.
Assuming 2 5 that serverb has the token, one embodiment of the protocol calls for serverb sends the token to server, which then sends the token to server;. After the transfer of the token by each server, its respective TokenHolder is appropriately updated. As a result, TokenHolderb will contain server,'s SID while TokenHoldera and TokenHolder;
will contain server;'s SID.
FIG. 6 is a flow diagram depicting the steps for TokenHandler routine;
FIG. 7 is a flow diagram depicting the steps for ReleaseToken routine; and FIG. 8 is a flow diagram depicting the steps for RequestHandler routine.
DETAILED DESCRIPTION OF THE INVENTION:
The invention relates to a token-based protocol which prevents conflicts.
The protocol provides efficient regeneration of a token after if has been lost by eliminating the need for lost global cooperation of all servers with a database replica. As previously discussed, conventional token-based protocols require both global participation by all the servers as well as several rounds of broadcast messages to regenerate the token after if has been lost. This can lead to increased cost and system downtime.
FIG. 1 is a simplified illustration of a conventional replicated distributed system 100. Such systems are described in Berson, Client/Server Architecture, McGraw-Hill, Inc., 1992. Distributed system 100 comprises sites 110, 120, 130, and networked together by communication links 150a-150f. The physical implementation of the communication links is not important. For example, the communication links may be a local area network (LAN) of a wide area network (WAN) interconnecting sites at 2 0 geographically dispersed locations. The sites may also be "loosely connected", i.e., connected through dial-up connections of wireless connections such as satellite links.
FIG. 2 is an exemplary site 200 of the distributed system. As shown, the site includes a plurality of clients 240a-240c network to a server 210 by a LAN, MAN, or WAN. The clients, for example, may be personal computers, workstations, or terminals.
2 5 Other clients (not shown) may be connected via remote links. While the clients are shown to be associated with a particular site, it is understood that they may connect to other servers at other sites within the system using remote links, such as dial-up connections using modems, wireless connections such as satellite links, or connections via the Internet. Furthermore, it is understood that clients may have similar or different 3 0 configurations from each other, and other clients (not shown) may be included as desired.
In addition, one or more clients may be located on the server. Server 210 is, in some embodiments, a mainframe computer system, a workstation, or a personal computer that includes non-volatile storage device 225, such as magnetic disks or optical disks for data storage. The data being stored in the storage device 225 includes data files which are 5 organized into one or more groups called databases. Each individual data file in the databases is referred to as a data item.
Referring back to FIG. l, a copy or replica of the same database is stored in the memory of the servers located at sites 110, 120, 130 and 140. For sake of brevity, servers 110, 120, and 130 refer to servers located at sites 110, 120, 130 and 140, respectively. Furthermore, the servers at each site may also be referred to as nodes. The clients at the various nodes provide users with an interface to communicate with servers to read and update data items. An "update" refers to an operation which modifies of one or more data items. When updates are made to one replica, these updates are propagated to other servers during a replication session to make all replicas consistent with each other.
Conventional update propagation techniques such as those described in Lotus Notes Release 3 Application Developers Reference Manual, 1993, Lotus Development Corporation, Cambridge, MA., can be employed to maintain replica consistency.
Typically, update propagation is scheduled between a source and recipient server. The source or recipient server identifies which copies of data items in the source 2 0 replica are more current than their corresponding copies in the recipient replica. The source server then propagates the more recent copies or updates to the recipient server.
To facilitate identification as to which copy of the same data items are more recent, version information such as version numbers described in D. K. Gifford, "Weighted Voting for Replicated Data", Proceedings of the 7th ACM SIGOPS Symposium on 2 5 Operating Systems Principles, ( 1979), pp. 150-159, or other conventional techniques for identifying versions of data items are used Version information are associated with the data items. Each time a copy of the data item is modified, its corresponding version information is updated. By comparing version information of two copies of the same data item, a determination as to which copy is more current can be made. Whenever a server acquires a more current copy of a data item from another server, it also acquires the version information associated with that copy of the data item In accordance with the protocol of the invention, servers in the system having a database replica maintain pointers that are associated with each token. These pointers are herein referred to as TokenHolders. Since the number of tokens is equal to the number of data items in a replica, the number of TokenHolders in each server is also equal to the number of data items. In order to simplify the description of the protocol, illustrative embodiments are described using one TokenHolder per server (i.e., 1 data item per database replica). However, it will be understood that the protocol is applied on a per-data item basis. Also, to facilitate ease of discussion, a subscript variable (e.g.,) is used to denote any server in the system and its corresponding components.
The pointers are used to keep track of the server possessing the token. In one embodiment, TokenHolder identifies either 1) to which server server;
forwarded the token or 2) itself in the case where server; has the token. Server; modifies TokenHolder;
each time it transfers or receives the token. For example, after server 110 forwards the token to server 120, TokenHolder"o will contain information identifying server 120, such as server 120's server identification (SID) and TokenHolder,ZO will contain its own Sm.
When a client connected to server, sends a request to update, for example server;'s copy of data item x in server; it determines whether or not it has the token 2 0 associated with x. If server, has the token, it services the update request. Otherwise, server; sends a token request message to the server (server,) identified in TokenHolder;. If server, does not have the token, server, examines its TokenHoldera and forwards a token request message to serverb (which is identified in TokenHolderb) on behalf of the server;.
This request chain continues until the server possessing the token is reached.
Assuming 2 5 that serverb has the token, one embodiment of the protocol calls for serverb sends the token to server, which then sends the token to server;. After the transfer of the token by each server, its respective TokenHolder is appropriately updated. As a result, TokenHolderb will contain server,'s SID while TokenHoldera and TokenHolder;
will contain server;'s SID.
As apparent from the above description, TokenHolder; , in some instances, may not necessarily inform server; who exactly is the possessor of the token.
However, the pointers, at the very least, provide a nexus to the server that actually holds the token.
This technique of updating the pointers enables a requesting server to obtain the token from the possessor, either directly or indirectly via an intermediate server.
As such, token requests are serviced without broadcasting messages system wide and waiting for the token possessor to respond to the message. To avoid loss of information during, for example, server failures, TokenHolders may be stored in non-volatile memory of the servers.
In the case where the token is lost, the request chain continues indefinitely because no server in the system has the token. Eventually, a situation occurs where the requester thinks that the requestee has the token and vice-versa, but neither does. At this point, the system is aware that the token is lost. Furthermore, the token last existed in either the requester and requestee. Since this is the case, regeneration of the token need only involve these two servers. The server which possesses the more recent version of x is the server which last had the token. As such, this server regenerates the token. Thus, the protocol advantageously regenerates the token without requiring global cooperation of all servers in the system with database replicas.
Figs. 3a-3f are illustrative examples showing the dynamics of TokenHolder 2 0 in replicated distributed system 100. The arrows identify the server to which each server's TokenHolder points. During initialization of system 100, a server is designated to generate tokens for the data item in the database. At this time, it is not important which server generates the tokens since all data item replicas should be identical.
Referring to FIG. 3a, server 140 is chosen to generate the token associated with data item x. As a 2 5 result, the TokenHolder associated with x at server 110, 120, and 130 all point to server 140 while the TokenHolderl,o points to itself.
Referring to FIG. 3b, server 110 requests the token from server 140. After server 140 releases the token to server 110, it updates its TokenHolder,~ to reflect that server 110 now has the token. Upon receipt of the token, server 110 updates its 3 0 TokenHolder,~a to show that it has the token. However, the TokenHolders for servers 120 and 130 remain unchanged, pointing to server 140 as the possessor of the token even though it is not. -As a result, any token request message originating at either 120 or 130 is sent to server 140, which then relays the request on behalf of the requesting server to server I 10. Unless the token is lost, the request chain eventually reaches the possessor of the token.
In FIG. 3c, server I20 subsequently requests the token from server 140.
Upon receiving the request, server 140 relays this request to server 110 which has the token. Server 140 releases the token to server 140 from which it received the request.
Server 140, upon receiving the token from server 1 Z0, releases it to server I20. The sequence of token transfers causes servers 110, 120 and 140 to update their respective TokenHolders as follows. Server 110's TokenHolder"o points to server 140;
server 140's TokenHolder,4o points to server I20; and server 120's TokenHolder,lo points to itself.
Since server 130 was not involved in the token transfer process, its TokenHolder,3o remains unchanged.
The token request by server 140 results in a request chain which includes server 140 and 110. However, this request chain may be collapsed when necessary to include only server 110 if the update propagation between any two servers includes propagating the contents of TokenHolder. FIG. 3d illustrates an example of how the request chain is collapsed during update propagation. As shown, the contents of the 2 0 TokenHolders are similar to those shown in FIG. 3c. As part of the scheduled update propagation between source server 140 and recipient server 130, data item version information as well as its associated TokenHolder are propagated to the recipient server.
The TokenHolder associated with the copy of the data item which has the most recent version information is the more current pointer. For example, if TokenHolder"o is more 2 5 current, its information is adopted by server 130 in its TokenHolder, jo during update propagation. As a result, server 130 now points to server 120 (dotted arrow).
Thus, the request chain which previously included servers 110 (crossed out arrow) has been collapsed to point directly to server 120. Likewise, an update propagation session between servers 110 and 140, as shown in FIG. 3e, collapses the request chain, which 3 0 previously included servers 140 and 120, to include only server 120.
From Figs. 3a-3c, the server possessing the token releases the token to the previous server in the token request chain, regardless of whether or not it is the original token requester. As a result, the token reaches the original token requester by traveling back along the request path. Alternatively, if information identifying the original token requester is included in the token request message, the token can be forwarded directly to the original requester, thus eliminating the need for the token to reverse the request path in order to reach the original requesting server. After the transfer of the token, the TokenHolder of the server possessing the token will contain the SID of the original token requester. However, a side effect of this technique is that under some situations, it results in request chains that is longer than if information identifying the original requester is not included in the request message.
A token can become obsolete. An "obsolete token" is a token which the system cannot ensure that it is associated with the latest copy of a data item. This may create potential conflicts in copies of the data item. Situations which may make a token obsolete include, for example, arrival of a token at the requesting server after an error has occurred. Such error may be a server failure or a timeout error. A "timeout error" is defined as an error which results when the time it takes for a requester to receive the token exceeds the amount of time allotted by the system. This usually happens when a token message experiences unusually long delays or when the token is lost. The timeout 2 0 en or thus prevents a server from waiting indefinitely for a token which may never arrive.
To prevent an obsolete token from being inadvertently received by the requesting server, the protocol provides servers with a counter (tnumber) associated with each token. The tnumber rtcords the number of times that the respective token had arrived at the respective server or the number of times that an error had resulted from a 2 5 token request. The current value of tnumber is included in the request message sent by the requester (server,). In addition, the tnumber of the requestee (serverb) is included in the token message which is sent back to server,. When a token arrives at the server" it compares the contents of its tnumbera with that of tnumberb. If tnumbera=
tnumberb, then the token is not obsolete. Otherwise, server, discards the obsolete token. As such, the 3 0 counter enables a server to detect and ignore an unexpected token message (i.e., obsolete) that can occur as a result of a server failure or error. 'The contents of the counters, in some embodiments, are stored in non-volatile memory to avoid axing convpted or lost during a server failure.
Storage of token counters in non-volatile storage requires a disk access 5 every time a token counter. is modified. To avoid the overhead associated with disk accesses, a different structure for token counters is employed. In accordance with one embodiment, each token counter comprises two components such as (age, counter). Age, which is stored in non-volatile storage, is incremented each time a server reboots, for example, from a server failure. Counter is stored in unstable storage and is incremented 10 in the same manner as tnumber. Using the format, token counter, is defined to be greater than token counters if and only if age; z age= and counter, > counter Z. As such, monotonicity of the token counters aro maintained as required by the protocol However, modifying or reading the counters does. not require a disk access.
In mull-processing environments, multiple processes can execute simultaneously. Concurrent execution of processes can cause potential conflicts between different copies of same data item in a replica. For exampk, client, and clientb, which are connected to the same node, request independent updates to the sarcx data item at the same time. Unless some controls are placed in the system to serialize the update requests, a conflict will exist between client;'s and cliant~'s copy of the data item.
These conflicts 2 0 are referred to as infra-node conflicts. To avoid infra node conflicts, conventional concurrency controls such as lock variables are provided for the server.
Concurrency controls and locks are described in J. Gray and A. Reuter, Transaction Processing:
Concepts and Techniques, Morgan Kaufmann, San Mateo, California, sass.
2 5 Locks are associac~ed with node resources, such as routines used to service token requests. Typically, two operations are associated with each lock variable, lock and unlock. To serialize parallel activities, only one process can successfully lock (or acquire) a lock variable at any one time. If other processes desire to acquire that lock variable, their requests are placed in a queue: Once the process which has aoquired the-lock variable unlocks (or releases) it, a waiting process is chosen from the lock queue as the next process by which to acquire the lock variable..
In accordance with one embodiment, the servers are provided with two lock variables, requesting and token associated with each token The use of a requesting lock variable ensures that a request for a specific token from the same server is not issued unless the previous request for that token has been serviced. This prevents requesting the token twice. This lock variable also prevents a token from being transferred to another server until it is released by the server possessing it. The token lock prevents concurrent accesses of a token by multiple processes. (This is a standard concurrency control within the server.) Referring to FTG. 4, a flow diagram depicting how a server acquires a token in accordance with an illustrative embodiment of the invention is shown.
The flow diagram is separated into sections 401, 402, and 403. Section 401 reflects actions executed by the server (server,) issuing the original token request, while section 402 reflects action executed by the server (serverb) receiving the request.
Section 403 indicates actions executed by an additional server (server), if necessary, in the request chain. At step 410, server, receives an update request for a data item x. At step 420, in response to the update request, server executes a routine (GetToken) to obtain the token.
The GetToken routine examines server,'s TokenHoldera associated with x to determine 2 0 which server in the system possesses x' token. If server, already has the token, it proceeds to step 475. If GetToken determines that another server (serverb ) possesses the token, it sends a token request message at step 430 to the servers. This request message includes server,'s 1D, server,'s copy of x' version vector (vvector,), and tnumber.
Thereafter, server, waits for the token to return. To prevent server, from waiting 2 5 indefinitely for the token, such as when the token is lost, a timeout mechanism is used. At the expiration of a predefined timeout period, GetToken informs server, that it is unable to retrieve the requested token.
Upon successful return of the token, GetToken proceeds to step 475. At step 475, after server, obtains the token, it executes a TokenHandler routine.
3 0 TokenHandler checks the tnumber of the token to make sure that it is not obsolete by comparing tnumber with tnumber~. If the token is not obsolete, servers updates it.
Otherwise the token is ignored, TokenHolderQ to reflect that server, now has the token.
At step 480, server, performs the client's update request and releases the token to a requesting server at step 490 by executing a ReleaseToken routine. The ReleaseToken routine identifies the requesting server and updates TokenHoldera to point to that server.
When serverb receives the token request message from server,, it can be in one of three states: 1 ) serverb thinks servers has the token, 2) serverb has the token, or 3) serverb has forwarded the token to server. In response to the token request message, serverb executes a RequestHandler routine at step 440. The RequestHandler examines TokenHolder,, to identify the state of serverb. State 1 indicates a problem because neither server, nor serverb has the token (both servers think the other has the token). This state typifies the scenario where a server which possesses the token experiences a failure or unusually long message delay. For example, server, issues a token request to serverb which successfully fulfills this request, i.e, serverb sends the token to server, and updates its TokenHolder to reflect this action. However, the token message to server, was delayed long enough for server,'s request to timeout. The timeout causes server,'s tnumberQ to be incremented. Then, even if server, receives the token, it would be ignored because its associated tnumberb is obsolete (tnumbera ~ tnumberb) As a result, the token becomes lost during transfer.
2 0 If RequestHandler determines that serverb is in state l, token regeneration is initiated. This includes comparing the vvectorb of serverb's copy of x to that of server, at step 445. If serverb's copy of x is more recent, then serverb is assumed to be the last server which had the token. Serverb then regenerates and forwards the token to server,.
Token regeneration may be achieved by causing serverb to imitate a token arrival by 2 5 executing the TokenHandler routine at step 450. RequestHandler then proceeds to steps 455 and 460 to retrieve and release the token by executing the GetToken and ReleaseToken routines. The token, along with its copy of x, since it is more current, is sent to server, at step 470. If, on the other hand, server,'s replica of x is more current, then serverb assumes that server, was the last server that had the token, resulting in serverb 3 0 sending a message at step 453 to inform server, to regenerate the token.
If serverb is in either state 2 or 3, RequestHandler executes the GetToken routine at step 455, which either executes ReleaseToken routine at step 460 and transfers the token to servers (state 2) or sends a token request message at step 458 to server (state 3). Server receives the token request message and processes it in a similar manner as serverb at step 440. This process continues until the request chain reaches the server possessing the token.
Referring to FIG. 5, a detailed flow diagram of the GetToken routine is shown. At steps 500 and 510, the requesting and token Lock variables are set to their lock states. Setting requesting to the lock state prevents server, from serving multiple token requests for the same token concurrent. In addition, the token variable is locked to avoid other processes from obtaining the token until the current process is finished updating the data item. At step 520, TokenHoldera is examined to identify which server in the system possesses the token. If servers already has the token, GetToken proceeds to step 570. If TokenHolderQ points to serverb, GetToken sends a token request message to the serverb at step 530. The message includes server,'s Sll~, tnumberQ, information identifying the data item being updated and its' associated version vector.
At step 550, GetToken then waits for the token to arrive or for a timeout to occur. ~f the token arrives before a timeout occurs, GetToken, at step 570 returns a success message to inform server, that the token has been successfully acquired. If a 2 0 timeout occurs, GetToken increments tnumber by 1 and notifies server, that it failed to receive the token at step 590. An unsuccessful acquisition of the token causes GetToken to relinquish system resources to another process by releasing the concurrency controls, i.e., unlocking the requesting lock variables.
FIG. 6 is a flow diagram of the TokenHandler routine in accordance with 2 5 one embodiment of the invention. At step 600, server, executes the TokenHandler routine when a token message arrives from serverb. The token lock variable at server, is locked at step 710. At step 620, TokenHandler validates the token by comparing the token counter (tnumber) from serverb with that of server,. The token is valid when both counters are equal. If the vvector, token is valid, TokenHandler proceeds to step 650 and compares 3 0 the vvector, with the vvectorb. If the comparison indicates that serverb's copy is more is current, server, adopts serverb's copy of x at step 660. Server, then updates vvector, to equal vvectorb at step 670 and proceeds to step 680. If serverb's copy of the document is less current or identical to server,'s copy, then server, does not need serverb's copy of x.
As such TokenHandler proceeds to step 680. At step 680, tnumberQ is incremented to reflect the arrival of the token. At step 685, TokenHoldera is also updated to reflect that server, now has the token. After TokenHolder is updated, token lock is unlocked at step 690.
FIG. 7 is a flow diagram depicting the ReleaseToken routine. At step 700, serverb receives a token request message from another server, and executes the ReleaseToken routine. The token request contains SID" tnumber4 and vvector, from server,. At step 710, the ReleaseToken determines whether the token request message originated from another server or from itself by comparing the SID, with its own S117. If they are equal, the request originated from itself and then proceeds to step 770.
Otherwise, the request originated from another server. At step 720, the TokenHolder is updated with SID,. Optionally, at step 730, the updated contents of TokenHolder is written to non-volatile memory. As previously discussed, this prevents TokenHolder from being lost if a server failure occurs.
At step 740, ReleaseToken compares the vvector, with vvectorb. If the serverb's copy is more current than server,'s copy, serverb sends the token message which 2 0 includes its tnumberb and its copy of the data item along with the vvectorb information to server,. Otherwise, serverb only sends the tnumberb and its vvectorb associated with its copy of the data item with the token message at step 750 because server, already has a more current copy of the data item. After the token message is sent, the request lock is reset to the unlock state.
2 5 FIG. 8 is a flow diagram depicting the RequestHandler routine. At step 800, serverb receives a token request message from server, and executes the RequestHandler routine. The message includes SID, , tnumbera, and vvector,. At step 810, the TokenHandler routine compares SID, with the contents of TokenHolderb.
If the two values are not equal, TokenHandler proceeds to step 820, where it calls GetToken.
3 0 However, if the values where it calls GerToken are equal, the system proceeds to step 860.
1~
This indicates that the token for x is lost and needs to be regenerated. At step-860, server, compares the vvector, of its copy of x with vvectorb to determine which server had the ' token last. If vvectorb's is more current than vvector,, a RequestHandler calls TokenHandler at step 870 to initiate regeneration of the token and proceeds to step 820.
If server,'s copy of x is more current, server, sends a token message to serverb at step 880.
The token message includes a zero vector, which is a vvector with all its components set to zero. This ensures that the data item associated with the zero vector is the least current. The token message and zero vector inform server, to regenerate the token since it was the last server to have the token.
At step 820, server, calls GetToken routine to generate the token. After the requested token returns successfully, at step 830 TokenHandler calls ReleaseToken routine which sends the token to server. However, if serverb fails to retrieve, the token, i.e., timeout, TokenHandler sends a failure message to server,.
While the invention has been particularly shown and described with reference to various embodiments, it will be recognized by those skilled in the art that modifications and changes may be made to the present invention without departing from the spirit and scope thereof. The scope of the invention should therefore be determined not with reference to the above description, but instead should be determined with reference to the appended claims, along with their full scope of equivalents.
However, the pointers, at the very least, provide a nexus to the server that actually holds the token.
This technique of updating the pointers enables a requesting server to obtain the token from the possessor, either directly or indirectly via an intermediate server.
As such, token requests are serviced without broadcasting messages system wide and waiting for the token possessor to respond to the message. To avoid loss of information during, for example, server failures, TokenHolders may be stored in non-volatile memory of the servers.
In the case where the token is lost, the request chain continues indefinitely because no server in the system has the token. Eventually, a situation occurs where the requester thinks that the requestee has the token and vice-versa, but neither does. At this point, the system is aware that the token is lost. Furthermore, the token last existed in either the requester and requestee. Since this is the case, regeneration of the token need only involve these two servers. The server which possesses the more recent version of x is the server which last had the token. As such, this server regenerates the token. Thus, the protocol advantageously regenerates the token without requiring global cooperation of all servers in the system with database replicas.
Figs. 3a-3f are illustrative examples showing the dynamics of TokenHolder 2 0 in replicated distributed system 100. The arrows identify the server to which each server's TokenHolder points. During initialization of system 100, a server is designated to generate tokens for the data item in the database. At this time, it is not important which server generates the tokens since all data item replicas should be identical.
Referring to FIG. 3a, server 140 is chosen to generate the token associated with data item x. As a 2 5 result, the TokenHolder associated with x at server 110, 120, and 130 all point to server 140 while the TokenHolderl,o points to itself.
Referring to FIG. 3b, server 110 requests the token from server 140. After server 140 releases the token to server 110, it updates its TokenHolder,~ to reflect that server 110 now has the token. Upon receipt of the token, server 110 updates its 3 0 TokenHolder,~a to show that it has the token. However, the TokenHolders for servers 120 and 130 remain unchanged, pointing to server 140 as the possessor of the token even though it is not. -As a result, any token request message originating at either 120 or 130 is sent to server 140, which then relays the request on behalf of the requesting server to server I 10. Unless the token is lost, the request chain eventually reaches the possessor of the token.
In FIG. 3c, server I20 subsequently requests the token from server 140.
Upon receiving the request, server 140 relays this request to server 110 which has the token. Server 140 releases the token to server 140 from which it received the request.
Server 140, upon receiving the token from server 1 Z0, releases it to server I20. The sequence of token transfers causes servers 110, 120 and 140 to update their respective TokenHolders as follows. Server 110's TokenHolder"o points to server 140;
server 140's TokenHolder,4o points to server I20; and server 120's TokenHolder,lo points to itself.
Since server 130 was not involved in the token transfer process, its TokenHolder,3o remains unchanged.
The token request by server 140 results in a request chain which includes server 140 and 110. However, this request chain may be collapsed when necessary to include only server 110 if the update propagation between any two servers includes propagating the contents of TokenHolder. FIG. 3d illustrates an example of how the request chain is collapsed during update propagation. As shown, the contents of the 2 0 TokenHolders are similar to those shown in FIG. 3c. As part of the scheduled update propagation between source server 140 and recipient server 130, data item version information as well as its associated TokenHolder are propagated to the recipient server.
The TokenHolder associated with the copy of the data item which has the most recent version information is the more current pointer. For example, if TokenHolder"o is more 2 5 current, its information is adopted by server 130 in its TokenHolder, jo during update propagation. As a result, server 130 now points to server 120 (dotted arrow).
Thus, the request chain which previously included servers 110 (crossed out arrow) has been collapsed to point directly to server 120. Likewise, an update propagation session between servers 110 and 140, as shown in FIG. 3e, collapses the request chain, which 3 0 previously included servers 140 and 120, to include only server 120.
From Figs. 3a-3c, the server possessing the token releases the token to the previous server in the token request chain, regardless of whether or not it is the original token requester. As a result, the token reaches the original token requester by traveling back along the request path. Alternatively, if information identifying the original token requester is included in the token request message, the token can be forwarded directly to the original requester, thus eliminating the need for the token to reverse the request path in order to reach the original requesting server. After the transfer of the token, the TokenHolder of the server possessing the token will contain the SID of the original token requester. However, a side effect of this technique is that under some situations, it results in request chains that is longer than if information identifying the original requester is not included in the request message.
A token can become obsolete. An "obsolete token" is a token which the system cannot ensure that it is associated with the latest copy of a data item. This may create potential conflicts in copies of the data item. Situations which may make a token obsolete include, for example, arrival of a token at the requesting server after an error has occurred. Such error may be a server failure or a timeout error. A "timeout error" is defined as an error which results when the time it takes for a requester to receive the token exceeds the amount of time allotted by the system. This usually happens when a token message experiences unusually long delays or when the token is lost. The timeout 2 0 en or thus prevents a server from waiting indefinitely for a token which may never arrive.
To prevent an obsolete token from being inadvertently received by the requesting server, the protocol provides servers with a counter (tnumber) associated with each token. The tnumber rtcords the number of times that the respective token had arrived at the respective server or the number of times that an error had resulted from a 2 5 token request. The current value of tnumber is included in the request message sent by the requester (server,). In addition, the tnumber of the requestee (serverb) is included in the token message which is sent back to server,. When a token arrives at the server" it compares the contents of its tnumbera with that of tnumberb. If tnumbera=
tnumberb, then the token is not obsolete. Otherwise, server, discards the obsolete token. As such, the 3 0 counter enables a server to detect and ignore an unexpected token message (i.e., obsolete) that can occur as a result of a server failure or error. 'The contents of the counters, in some embodiments, are stored in non-volatile memory to avoid axing convpted or lost during a server failure.
Storage of token counters in non-volatile storage requires a disk access 5 every time a token counter. is modified. To avoid the overhead associated with disk accesses, a different structure for token counters is employed. In accordance with one embodiment, each token counter comprises two components such as (age, counter). Age, which is stored in non-volatile storage, is incremented each time a server reboots, for example, from a server failure. Counter is stored in unstable storage and is incremented 10 in the same manner as tnumber. Using the format, token counter, is defined to be greater than token counters if and only if age; z age= and counter, > counter Z. As such, monotonicity of the token counters aro maintained as required by the protocol However, modifying or reading the counters does. not require a disk access.
In mull-processing environments, multiple processes can execute simultaneously. Concurrent execution of processes can cause potential conflicts between different copies of same data item in a replica. For exampk, client, and clientb, which are connected to the same node, request independent updates to the sarcx data item at the same time. Unless some controls are placed in the system to serialize the update requests, a conflict will exist between client;'s and cliant~'s copy of the data item.
These conflicts 2 0 are referred to as infra-node conflicts. To avoid infra node conflicts, conventional concurrency controls such as lock variables are provided for the server.
Concurrency controls and locks are described in J. Gray and A. Reuter, Transaction Processing:
Concepts and Techniques, Morgan Kaufmann, San Mateo, California, sass.
2 5 Locks are associac~ed with node resources, such as routines used to service token requests. Typically, two operations are associated with each lock variable, lock and unlock. To serialize parallel activities, only one process can successfully lock (or acquire) a lock variable at any one time. If other processes desire to acquire that lock variable, their requests are placed in a queue: Once the process which has aoquired the-lock variable unlocks (or releases) it, a waiting process is chosen from the lock queue as the next process by which to acquire the lock variable..
In accordance with one embodiment, the servers are provided with two lock variables, requesting and token associated with each token The use of a requesting lock variable ensures that a request for a specific token from the same server is not issued unless the previous request for that token has been serviced. This prevents requesting the token twice. This lock variable also prevents a token from being transferred to another server until it is released by the server possessing it. The token lock prevents concurrent accesses of a token by multiple processes. (This is a standard concurrency control within the server.) Referring to FTG. 4, a flow diagram depicting how a server acquires a token in accordance with an illustrative embodiment of the invention is shown.
The flow diagram is separated into sections 401, 402, and 403. Section 401 reflects actions executed by the server (server,) issuing the original token request, while section 402 reflects action executed by the server (serverb) receiving the request.
Section 403 indicates actions executed by an additional server (server), if necessary, in the request chain. At step 410, server, receives an update request for a data item x. At step 420, in response to the update request, server executes a routine (GetToken) to obtain the token.
The GetToken routine examines server,'s TokenHoldera associated with x to determine 2 0 which server in the system possesses x' token. If server, already has the token, it proceeds to step 475. If GetToken determines that another server (serverb ) possesses the token, it sends a token request message at step 430 to the servers. This request message includes server,'s 1D, server,'s copy of x' version vector (vvector,), and tnumber.
Thereafter, server, waits for the token to return. To prevent server, from waiting 2 5 indefinitely for the token, such as when the token is lost, a timeout mechanism is used. At the expiration of a predefined timeout period, GetToken informs server, that it is unable to retrieve the requested token.
Upon successful return of the token, GetToken proceeds to step 475. At step 475, after server, obtains the token, it executes a TokenHandler routine.
3 0 TokenHandler checks the tnumber of the token to make sure that it is not obsolete by comparing tnumber with tnumber~. If the token is not obsolete, servers updates it.
Otherwise the token is ignored, TokenHolderQ to reflect that server, now has the token.
At step 480, server, performs the client's update request and releases the token to a requesting server at step 490 by executing a ReleaseToken routine. The ReleaseToken routine identifies the requesting server and updates TokenHoldera to point to that server.
When serverb receives the token request message from server,, it can be in one of three states: 1 ) serverb thinks servers has the token, 2) serverb has the token, or 3) serverb has forwarded the token to server. In response to the token request message, serverb executes a RequestHandler routine at step 440. The RequestHandler examines TokenHolder,, to identify the state of serverb. State 1 indicates a problem because neither server, nor serverb has the token (both servers think the other has the token). This state typifies the scenario where a server which possesses the token experiences a failure or unusually long message delay. For example, server, issues a token request to serverb which successfully fulfills this request, i.e, serverb sends the token to server, and updates its TokenHolder to reflect this action. However, the token message to server, was delayed long enough for server,'s request to timeout. The timeout causes server,'s tnumberQ to be incremented. Then, even if server, receives the token, it would be ignored because its associated tnumberb is obsolete (tnumbera ~ tnumberb) As a result, the token becomes lost during transfer.
2 0 If RequestHandler determines that serverb is in state l, token regeneration is initiated. This includes comparing the vvectorb of serverb's copy of x to that of server, at step 445. If serverb's copy of x is more recent, then serverb is assumed to be the last server which had the token. Serverb then regenerates and forwards the token to server,.
Token regeneration may be achieved by causing serverb to imitate a token arrival by 2 5 executing the TokenHandler routine at step 450. RequestHandler then proceeds to steps 455 and 460 to retrieve and release the token by executing the GetToken and ReleaseToken routines. The token, along with its copy of x, since it is more current, is sent to server, at step 470. If, on the other hand, server,'s replica of x is more current, then serverb assumes that server, was the last server that had the token, resulting in serverb 3 0 sending a message at step 453 to inform server, to regenerate the token.
If serverb is in either state 2 or 3, RequestHandler executes the GetToken routine at step 455, which either executes ReleaseToken routine at step 460 and transfers the token to servers (state 2) or sends a token request message at step 458 to server (state 3). Server receives the token request message and processes it in a similar manner as serverb at step 440. This process continues until the request chain reaches the server possessing the token.
Referring to FIG. 5, a detailed flow diagram of the GetToken routine is shown. At steps 500 and 510, the requesting and token Lock variables are set to their lock states. Setting requesting to the lock state prevents server, from serving multiple token requests for the same token concurrent. In addition, the token variable is locked to avoid other processes from obtaining the token until the current process is finished updating the data item. At step 520, TokenHoldera is examined to identify which server in the system possesses the token. If servers already has the token, GetToken proceeds to step 570. If TokenHolderQ points to serverb, GetToken sends a token request message to the serverb at step 530. The message includes server,'s Sll~, tnumberQ, information identifying the data item being updated and its' associated version vector.
At step 550, GetToken then waits for the token to arrive or for a timeout to occur. ~f the token arrives before a timeout occurs, GetToken, at step 570 returns a success message to inform server, that the token has been successfully acquired. If a 2 0 timeout occurs, GetToken increments tnumber by 1 and notifies server, that it failed to receive the token at step 590. An unsuccessful acquisition of the token causes GetToken to relinquish system resources to another process by releasing the concurrency controls, i.e., unlocking the requesting lock variables.
FIG. 6 is a flow diagram of the TokenHandler routine in accordance with 2 5 one embodiment of the invention. At step 600, server, executes the TokenHandler routine when a token message arrives from serverb. The token lock variable at server, is locked at step 710. At step 620, TokenHandler validates the token by comparing the token counter (tnumber) from serverb with that of server,. The token is valid when both counters are equal. If the vvector, token is valid, TokenHandler proceeds to step 650 and compares 3 0 the vvector, with the vvectorb. If the comparison indicates that serverb's copy is more is current, server, adopts serverb's copy of x at step 660. Server, then updates vvector, to equal vvectorb at step 670 and proceeds to step 680. If serverb's copy of the document is less current or identical to server,'s copy, then server, does not need serverb's copy of x.
As such TokenHandler proceeds to step 680. At step 680, tnumberQ is incremented to reflect the arrival of the token. At step 685, TokenHoldera is also updated to reflect that server, now has the token. After TokenHolder is updated, token lock is unlocked at step 690.
FIG. 7 is a flow diagram depicting the ReleaseToken routine. At step 700, serverb receives a token request message from another server, and executes the ReleaseToken routine. The token request contains SID" tnumber4 and vvector, from server,. At step 710, the ReleaseToken determines whether the token request message originated from another server or from itself by comparing the SID, with its own S117. If they are equal, the request originated from itself and then proceeds to step 770.
Otherwise, the request originated from another server. At step 720, the TokenHolder is updated with SID,. Optionally, at step 730, the updated contents of TokenHolder is written to non-volatile memory. As previously discussed, this prevents TokenHolder from being lost if a server failure occurs.
At step 740, ReleaseToken compares the vvector, with vvectorb. If the serverb's copy is more current than server,'s copy, serverb sends the token message which 2 0 includes its tnumberb and its copy of the data item along with the vvectorb information to server,. Otherwise, serverb only sends the tnumberb and its vvectorb associated with its copy of the data item with the token message at step 750 because server, already has a more current copy of the data item. After the token message is sent, the request lock is reset to the unlock state.
2 5 FIG. 8 is a flow diagram depicting the RequestHandler routine. At step 800, serverb receives a token request message from server, and executes the RequestHandler routine. The message includes SID, , tnumbera, and vvector,. At step 810, the TokenHandler routine compares SID, with the contents of TokenHolderb.
If the two values are not equal, TokenHandler proceeds to step 820, where it calls GetToken.
3 0 However, if the values where it calls GerToken are equal, the system proceeds to step 860.
1~
This indicates that the token for x is lost and needs to be regenerated. At step-860, server, compares the vvector, of its copy of x with vvectorb to determine which server had the ' token last. If vvectorb's is more current than vvector,, a RequestHandler calls TokenHandler at step 870 to initiate regeneration of the token and proceeds to step 820.
If server,'s copy of x is more current, server, sends a token message to serverb at step 880.
The token message includes a zero vector, which is a vvector with all its components set to zero. This ensures that the data item associated with the zero vector is the least current. The token message and zero vector inform server, to regenerate the token since it was the last server to have the token.
At step 820, server, calls GetToken routine to generate the token. After the requested token returns successfully, at step 830 TokenHandler calls ReleaseToken routine which sends the token to server. However, if serverb fails to retrieve, the token, i.e., timeout, TokenHandler sends a failure message to server,.
While the invention has been particularly shown and described with reference to various embodiments, it will be recognized by those skilled in the art that modifications and changes may be made to the present invention without departing from the spirit and scope thereof. The scope of the invention should therefore be determined not with reference to the above description, but instead should be determined with reference to the appended claims, along with their full scope of equivalents.
Claims (14)
1. In a computer network with a database comprising a plurality of data items, the network comprises a plurality of servers corresponding to a plurality of database replicas, each comprising copies of the plurality of data items, a method for preventing conflicts among the different copies of the same data item in the database replicas comprising the steps of:
providing a plurality of tokens corresponding to the plurality of data items, wherein possession of a token by one of the plurality of servers (possessor) grants permission to the possessor to update a data item corresponding to the token in the server's database replica;
providing a plurality of token pointers in the plurality of servers corresponding to the plurality of tokens; and updating the token pointers to provide a nexus to the possessor of the corresponding token.
providing a plurality of tokens corresponding to the plurality of data items, wherein possession of a token by one of the plurality of servers (possessor) grants permission to the possessor to update a data item corresponding to the token in the server's database replica;
providing a plurality of token pointers in the plurality of servers corresponding to the plurality of tokens; and updating the token pointers to provide a nexus to the possessor of the corresponding token.
2. The method of claim 1 further comprising the step of servicing a request by a client connected to the server, referred to a requestor, to update one of the plurality of data items in the requestor's replica, wherein the step comprises the steps of:
retrieving token associated with the data item; and performing updates on the data item after the token has been retrieved.
retrieving token associated with the data item; and performing updates on the data item after the token has been retrieved.
3. The method of claim 2 wherein the step of retrieving the token comprises the steps of:
examining the requestor's token pointer corresponding to the token associated with the data item and if the requestor's token pointer identifies the requestor as the possessor, then it already has the token; and sending a token request message to a requestee server if the requestor's token pointer identifies the requestee as the possessor, where the requestee is not the same as the requestor.
examining the requestor's token pointer corresponding to the token associated with the data item and if the requestor's token pointer identifies the requestor as the possessor, then it already has the token; and sending a token request message to a requestee server if the requestor's token pointer identifies the requestee as the possessor, where the requestee is not the same as the requestor.
4. The method of claim 3 wherein the step the token request comprises information identifying the requestor (SID) and version information associated with the data item.
5. The method of claim 4 wherein the version information associated with the data item is a version vector.
6. The method of claim 5 further comprises the step of servicing the token request wherein the step comprises:
examining the requestee's to determine the possessor of the token; and i. releasing the token to the requestor if the requestee's token pointer identifies the requestee as the possessor, ii. forwarding an intermediate token request message on behalf of the requestor to a second requestee if the requestee's token pointer identifies the second requestee as the possessor, where the second requestee is not equal to the requestee, and wherein the intermediate token request message comprises the requestee's SID
and the version vector of the requestee's copy of the data item, and iii. regenerating the token if the requestee's token pointer identifies the requestor as the possessor.
examining the requestee's to determine the possessor of the token; and i. releasing the token to the requestor if the requestee's token pointer identifies the requestee as the possessor, ii. forwarding an intermediate token request message on behalf of the requestor to a second requestee if the requestee's token pointer identifies the second requestee as the possessor, where the second requestee is not equal to the requestee, and wherein the intermediate token request message comprises the requestee's SID
and the version vector of the requestee's copy of the data item, and iii. regenerating the token if the requestee's token pointer identifies the requestor as the possessor.
7. The method of claim 6 where the step of releasing the token to the requestor comprises:
updating the requestee's token holder to reflect that the requestor is the possessor;
determining which copy of the data item is more current by comparing the version vector of the requestee's copy of the data item with the version vector of the requestor's copy of the data item; and sending a token message by the requestee to the requestor, wherein the token message comprises the version vector of the requestee's copy of the data item if the requestor's copy of the data item is more current and further comprising the requestee's copy of the data item to the requestee's copy if the data item is more current.
updating the requestee's token holder to reflect that the requestor is the possessor;
determining which copy of the data item is more current by comparing the version vector of the requestee's copy of the data item with the version vector of the requestor's copy of the data item; and sending a token message by the requestee to the requestor, wherein the token message comprises the version vector of the requestee's copy of the data item if the requestor's copy of the data item is more current and further comprising the requestee's copy of the data item to the requestee's copy if the data item is more current.
8. The method of claim 7 wherein the regenerating step comprises:
comparing requestee's version vector with requestor's version vector to determine which copy of the data item is more current;
sending a token regeneration message from the requestee to the requestor informing the requestor to regenerate the token if the requestee's copy of the data item is more current; and if the requestee's copy of the data item is more current, regenerating the token by the requestee and releasing the token to the requestor.
comparing requestee's version vector with requestor's version vector to determine which copy of the data item is more current;
sending a token regeneration message from the requestee to the requestor informing the requestor to regenerate the token if the requestee's copy of the data item is more current; and if the requestee's copy of the data item is more current, regenerating the token by the requestee and releasing the token to the requestor.
9. The method of claim 8 wherein the token regeneration message comprises a zero version vector wherein the components of the zero version vector are set to zero.
10. The method of claim 7 further comprising the step of handling the token message by the requestor comprising:
comparing the version vector from the token message with the version vector of the requestor's copy of the data item to determine which copy of the data item is more current;
if the requestee's copy of the data item is more current, then i. adopting the requestee's copy of the data item into the requestor's database replica, ii. updating the requestor's version vector associated with the data item to equal the version vector of the requestee's copy of the data item, and iii. updating the requestor's token pointer to reflect that the requestor is the possessor of the token; or if the requestor's copy of the data item is more current, then updating the requestor's token pointer to reflect that the requestor is the possessor of the token.
comparing the version vector from the token message with the version vector of the requestor's copy of the data item to determine which copy of the data item is more current;
if the requestee's copy of the data item is more current, then i. adopting the requestee's copy of the data item into the requestor's database replica, ii. updating the requestor's version vector associated with the data item to equal the version vector of the requestee's copy of the data item, and iii. updating the requestor's token pointer to reflect that the requestor is the possessor of the token; or if the requestor's copy of the data item is more current, then updating the requestor's token pointer to reflect that the requestor is the possessor of the token.
11. The method of claim 10 further comprises the steps of:
providing a plurality of token numbers in the plurality of servers, the plurality of token numbers associated with the plurality of tokens; and incrementing the requestor's token number in response to the requestor receiving the token.
providing a plurality of token numbers in the plurality of servers, the plurality of token numbers associated with the plurality of tokens; and incrementing the requestor's token number in response to the requestor receiving the token.
12. The method of clam 11 further comprises the step of:
incrementing the requestor's token number after a token request timeout occurs.
incrementing the requestor's token number after a token request timeout occurs.
13. The method of claim 11 wherein the step of incrementing further comprises the steps of:
comparing the requestor's token number with the requestee's token number associated with the token;
receiving the token if the requestor's and requestee's token numbers are equal; and ignoring the token if the requestor's and requestee's token numbers are not equal.
comparing the requestor's token number with the requestee's token number associated with the token;
receiving the token if the requestor's and requestee's token numbers are equal; and ignoring the token if the requestor's and requestee's token numbers are not equal.
14. A distributed system having a database comprising a plurality of data items, the system comprises:
a plurality of servers;
a plurality of replicas of the database wherein the replicas have copies of the plurality of data items, the plurality of replicas corresponding to the plurality of servers;
a plurality of tokens associated with the plurality of data items, wherein possession of a token granted by a server (possessor) grants the possessor permission to update a data item corresponding to the token;
a plurality of token pointers are provided in the plurality of servers, the plurality of token pointers corresponding to the plurality of tokens, the plurality of token pointers are updated in the plurality of servers to provide a nexus to the possessor of the corresponding token.
a plurality of servers;
a plurality of replicas of the database wherein the replicas have copies of the plurality of data items, the plurality of replicas corresponding to the plurality of servers;
a plurality of tokens associated with the plurality of data items, wherein possession of a token granted by a server (possessor) grants the possessor permission to update a data item corresponding to the token;
a plurality of token pointers are provided in the plurality of servers, the plurality of token pointers corresponding to the plurality of tokens, the plurality of token pointers are updated in the plurality of servers to provide a nexus to the possessor of the corresponding token.
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| US666,772 | 1996-06-19 | ||
| US08/666,772 US5802062A (en) | 1996-06-19 | 1996-06-19 | Preventing conflicts in distributed systems |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CA2205725A1 CA2205725A1 (en) | 1997-12-19 |
| CA2205725C true CA2205725C (en) | 2000-08-22 |
Family
ID=24675412
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CA002205725A Expired - Fee Related CA2205725C (en) | 1996-06-19 | 1997-05-20 | Preventing conflicts in distributed systems |
Country Status (4)
| Country | Link |
|---|---|
| US (1) | US5802062A (en) |
| EP (1) | EP0814590A3 (en) |
| JP (1) | JPH10187519A (en) |
| CA (1) | CA2205725C (en) |
Families Citing this family (43)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6122631A (en) * | 1997-03-28 | 2000-09-19 | International Business Machines Corporation | Dynamic server-managed access control for a distributed file system |
| US6031833A (en) * | 1997-05-01 | 2000-02-29 | Apple Computer, Inc. | Method and system for increasing throughput in a wireless local area network |
| US6161145A (en) * | 1997-05-08 | 2000-12-12 | International Business Machines Corporation | Updating server-related data at a client |
| US6341339B1 (en) * | 1998-03-26 | 2002-01-22 | Compaq Computer Corporation | Apparatus and method for maintaining data coherence within a cluster of symmetric multiprocessors |
| US6484109B1 (en) * | 1998-05-20 | 2002-11-19 | Dli Engineering Coporation | Diagnostic vibration data collector and analyzer |
| US6526448B1 (en) | 1998-12-22 | 2003-02-25 | At&T Corp. | Pseudo proxy server providing instant overflow capacity to computer networks |
| US6298308B1 (en) | 1999-05-20 | 2001-10-02 | Reid Asset Management Company | Diagnostic network with automated proactive local experts |
| US6401103B1 (en) | 1999-08-06 | 2002-06-04 | International Business Machines Corporation | Apparatus, method, and article of manufacture for client-side optimistic locking in a stateless environment |
| US7596563B1 (en) * | 1999-10-28 | 2009-09-29 | Hewlett-Packard Development Company, L.P. | Computerized file system and method |
| US6421686B1 (en) | 1999-11-15 | 2002-07-16 | International Business Machines Corporation | Method of replicating data records |
| US7039670B2 (en) * | 2000-03-30 | 2006-05-02 | United Devices, Inc. | Massively distributed processing system with modular client agent and associated method |
| US7254607B2 (en) * | 2000-03-30 | 2007-08-07 | United Devices, Inc. | Dynamic coordination and control of network connected devices for large-scale network site testing and associated architectures |
| US20090216641A1 (en) | 2000-03-30 | 2009-08-27 | Niration Network Group, L.L.C. | Methods and Systems for Indexing Content |
| US7092985B2 (en) * | 2000-03-30 | 2006-08-15 | United Devices, Inc. | Method of managing workloads and associated distributed processing system |
| US6963897B1 (en) * | 2000-03-30 | 2005-11-08 | United Devices, Inc. | Customer services and advertising based upon device attributes and associated distributed processing system |
| US20010027467A1 (en) * | 2000-03-30 | 2001-10-04 | Anderson David P. | Massively distributed database system and associated method |
| US7082474B1 (en) | 2000-03-30 | 2006-07-25 | United Devices, Inc. | Data sharing and file distribution method and associated distributed processing system |
| US6891802B1 (en) | 2000-03-30 | 2005-05-10 | United Devices, Inc. | Network site testing method and associated system |
| US7020678B1 (en) | 2000-03-30 | 2006-03-28 | United Devices, Inc. | Machine generated sweepstakes entry model and associated distributed processing system |
| US8010703B2 (en) * | 2000-03-30 | 2011-08-30 | Prashtama Wireless Llc | Data conversion services and associated distributed processing system |
| US6654783B1 (en) | 2000-03-30 | 2003-11-25 | Ethergent Corporation | Network site content indexing method and associated system |
| US20040148336A1 (en) * | 2000-03-30 | 2004-07-29 | Hubbard Edward A | Massively distributed processing system architecture, scheduling, unique device identification and associated methods |
| US6847995B1 (en) | 2000-03-30 | 2005-01-25 | United Devices, Inc. | Security architecture for distributed processing systems and associated method |
| USRE42153E1 (en) * | 2000-03-30 | 2011-02-15 | Hubbard Edward A | Dynamic coordination and control of network connected devices for large-scale network site testing and associated architectures |
| US7003547B1 (en) | 2000-03-30 | 2006-02-21 | United Devices, Inc. | Distributed parallel processing system having capability-based incentives and associated method |
| US7136906B2 (en) * | 2000-04-07 | 2006-11-14 | Clarity Visual Systems, Inc. | System for electronically distributing, displaying and controlling the play scheduling of advertising and other communicative media |
| WO2001078273A1 (en) * | 2000-04-07 | 2001-10-18 | Adspace Networks | System for electronically distributing, displaying and controlling advertising and other communicative media |
| US6671699B1 (en) * | 2000-05-20 | 2003-12-30 | Equipe Communications Corporation | Shared database usage in network devices |
| US7177866B2 (en) | 2001-03-16 | 2007-02-13 | Gravic, Inc. | Asynchronous coordinated commit replication and dual write with replication transmission and locking of target database on updates only |
| US7103586B2 (en) * | 2001-03-16 | 2006-09-05 | Gravic, Inc. | Collision avoidance in database replication systems |
| US7243369B2 (en) | 2001-08-06 | 2007-07-10 | Sun Microsystems, Inc. | Uniform resource locator access management and control system and method |
| US20050240763A9 (en) * | 2001-08-06 | 2005-10-27 | Shivaram Bhat | Web based applications single sign on system and method |
| US7594256B2 (en) * | 2003-06-26 | 2009-09-22 | Sun Microsystems, Inc. | Remote interface for policy decisions governing access control |
| US7634566B2 (en) | 2004-06-03 | 2009-12-15 | Cisco Technology, Inc. | Arrangement in a network for passing control of distributed data between network nodes for optimized client access based on locality |
| US8185893B2 (en) * | 2006-10-27 | 2012-05-22 | Hewlett-Packard Development Company, L.P. | Starting up at least one virtual machine in a physical machine by a load balancer |
| US9092250B1 (en) | 2006-10-27 | 2015-07-28 | Hewlett-Packard Development Company, L.P. | Selecting one of plural layouts of virtual machines on physical machines |
| US8732699B1 (en) | 2006-10-27 | 2014-05-20 | Hewlett-Packard Development Company, L.P. | Migrating virtual machines between physical machines in a define group |
| US8296760B2 (en) * | 2006-10-27 | 2012-10-23 | Hewlett-Packard Development Company, L.P. | Migrating a virtual machine from a first physical machine in response to receiving a command to lower a power mode of the first physical machine |
| US8341626B1 (en) | 2007-11-30 | 2012-12-25 | Hewlett-Packard Development Company, L. P. | Migration of a virtual machine in response to regional environment effects |
| US20110029432A1 (en) * | 2009-07-30 | 2011-02-03 | Hildred Richard N | Computer-implemented methods of processing payments for a merchant selling goods or services to a consumer |
| WO2012140916A1 (en) * | 2011-04-15 | 2012-10-18 | 日本電気株式会社 | Log information recording system, device, method, and program |
| US9594579B2 (en) | 2011-07-29 | 2017-03-14 | Hewlett Packard Enterprise Development Lp | Migrating virtual machines |
| CN112835537A (en) * | 2021-03-12 | 2021-05-25 | 东莞中国科学院云计算产业技术创新与育成中心 | Distributed data access method and device and computer equipment |
Family Cites Families (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US4949302A (en) * | 1986-11-17 | 1990-08-14 | International Business Machines Corporation | Message file formation for computer programs |
| JPS63138439A (en) * | 1986-12-01 | 1988-06-10 | Hitachi Ltd | Processing system for access request to decentralized data base |
| US5175851A (en) * | 1989-05-15 | 1992-12-29 | International Business Machines Corporation | System and method for controlling client machine access to a portion of a file with a variable length |
| JP2519390B2 (en) * | 1992-09-11 | 1996-07-31 | インターナショナル・ビジネス・マシーンズ・コーポレイション | DATA COMMUNICATION METHOD AND DEVICE |
| GB2273182A (en) * | 1992-12-04 | 1994-06-08 | Ibm | Currency period of replicated data objects. |
| US5634122A (en) * | 1994-12-30 | 1997-05-27 | International Business Machines Corporation | System and method for multi-level token management for distributed file systems |
| US5644698A (en) * | 1996-05-30 | 1997-07-01 | International Business Machines Corporation | Configurable reuse delay criterion for storage volumes |
-
1996
- 1996-06-19 US US08/666,772 patent/US5802062A/en not_active Expired - Lifetime
-
1997
- 1997-05-20 CA CA002205725A patent/CA2205725C/en not_active Expired - Fee Related
- 1997-06-16 EP EP97109793A patent/EP0814590A3/en not_active Ceased
- 1997-06-19 JP JP9161746A patent/JPH10187519A/en active Pending
Also Published As
| Publication number | Publication date |
|---|---|
| US5802062A (en) | 1998-09-01 |
| CA2205725A1 (en) | 1997-12-19 |
| EP0814590A2 (en) | 1997-12-29 |
| JPH10187519A (en) | 1998-07-21 |
| EP0814590A3 (en) | 1999-11-17 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CA2205725C (en) | Preventing conflicts in distributed systems | |
| US7149761B2 (en) | System and method for managing the synchronization of replicated version-managed databases | |
| AU640029B2 (en) | Distributed data processing systems | |
| US5459871A (en) | Detection and resolution of resource deadlocks in a distributed data processing system | |
| Amir et al. | Robust and E cient Replication using Group Communication | |
| EP1543420A2 (en) | Consistent message ordering for semi-active and passive replication | |
| DuBourdieux | Implementation of Distributed Transactions. | |
| CN112527759B (en) | Log execution method and device, computer equipment and storage medium | |
| EP0834122B1 (en) | Synchronisation procedure in a routing node | |
| Saito et al. | Optimistic replication for internet data services | |
| Yap et al. | Fault tolerant remote procedure call | |
| US11030220B2 (en) | Global table management operations for multi-region replicated tables | |
| US7840671B2 (en) | Managing the size and accessibility of a name service | |
| KR100466140B1 (en) | Method, system and program products for managing processing groups of a distributed computing environment | |
| US20030018606A1 (en) | Revocation of tokens without communication between the token holders and the token server | |
| US7028219B2 (en) | Hybrid method for flushing transaction state in a fault-tolerant clustered database | |
| Adly et al. | A hierarchical asynchronous replication protocol for large scale systems | |
| US7984482B1 (en) | Global account lockout (GAL) and expiration using an ordered message service (OMS) | |
| Burger et al. | Performance of multiversion and distributed two-phase locking concurrency control mechanisms in distributed database | |
| Kumar et al. | An efficient algorithm for mutual exclusion using queue migration in computer networks | |
| Irún-Briz et al. | An improved optimistic and fault-tolerant replication protocol | |
| Arika et al. | A survey on efficient concurrency control algorithm in distributed database systems | |
| Ramírez et al. | Distributed control of updates in multiple copy databases: A time optimal algorithm | |
| Ballintijn et al. | Lightweight crash recovery in a wide-area location service | |
| Muñoz-Escoí et al. | Flexible management of consistency and availability of networked data replications |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| EEER | Examination request | ||
| MKLA | Lapsed |
Effective date: 20160520 |