CA2290289C - Cluster node distress signal - Google Patents
Cluster node distress signal Download PDFInfo
- Publication number
- CA2290289C CA2290289C CA002290289A CA2290289A CA2290289C CA 2290289 C CA2290289 C CA 2290289C CA 002290289 A CA002290289 A CA 002290289A CA 2290289 A CA2290289 A CA 2290289A CA 2290289 C CA2290289 C CA 2290289C
- Authority
- CA
- Canada
- Prior art keywords
- cluster
- distress
- node
- prebuilt
- message
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
- 230000009429 distress Effects 0.000 title claims abstract description 190
- 238000000034 method Methods 0.000 claims abstract description 66
- 230000007246 mechanism Effects 0.000 claims description 10
- 238000004590 computer program Methods 0.000 claims description 7
- 230000008569 process Effects 0.000 claims description 6
- 230000005540 biological transmission Effects 0.000 claims description 3
- 238000013024 troubleshooting Methods 0.000 abstract description 4
- 238000007726 management method Methods 0.000 description 14
- 238000012545 processing Methods 0.000 description 12
- 230000006854 communication Effects 0.000 description 11
- 238000004891 communication Methods 0.000 description 11
- 230000006870 function Effects 0.000 description 8
- 230000009471 action Effects 0.000 description 6
- 230000001360 synchronised effect Effects 0.000 description 6
- 230000006855 networking Effects 0.000 description 4
- 230000008901 benefit Effects 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 238000012546 transfer Methods 0.000 description 3
- 238000007792 addition Methods 0.000 description 2
- 238000004883 computer application Methods 0.000 description 2
- 230000003247 decreasing effect Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- 230000007175 bidirectional communication Effects 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 238000002360 preparation method Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000011664 signaling Effects 0.000 description 1
- 238000012384 transportation and delivery Methods 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
- H04L69/40—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass for recovering from a failure of a protocol instance or entity, e.g. service redundancy protocols, protocol state redundancy or protocol service redirection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/16—Error detection or correction of the data by redundancy in hardware
- G06F11/20—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
- G06F11/202—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where processing functionality is redundant
- G06F11/2023—Failover techniques
Abstract
The preferred embodiment of the present invention provides a cluster node distress system and method that improves the reliability of a cluster. The cluster node distress system provides a cluster node distress signal when a node within the cluster is about to fail. This allows the cluster to better to determine whether a non-communicating node has failed or has merely been partitioned from the cluster. The preferred cluster node distress system is embedded deeply into the operating system and provides a pre-built node distress signal that can be quickly sent to other nodes within the cluster when an imminent failure of any node is detected, improving the probability that the node distress signal will get out before the node totally fails. When the node distress signal is effectively sent to other nodes within the cluster, the cluster can accurately determine which node has failed and has not just been partitioned from the cluster. This allows the cluster to respond correctly, i.e., by assigning other nodes primary and backup responsibility, with less manual intervention and troubleshooting needed by administrators.
Description
CLUSTER NODE DISTRESS SIGNAL
BACKGROUND OF THE INVENTION
1. Technical Field This invention generally relates to clustering computers, and more specifically relates to distress signalling for cluster communications.
BACKGROUND OF THE INVENTION
1. Technical Field This invention generally relates to clustering computers, and more specifically relates to distress signalling for cluster communications.
2. Background Art Society depends upon computer systems for many types of information in this electronic age.
Based upon various combinations of hardware (e.g., semiconductors, circuit boards, etc.) and software (e.g., computer programs), computer systems vary widely in design.
Many computer systems today are designed to "network" with other computer systems. Through networking, a single computer system can access information stored on and processed by other computer systems.
Thus, networking results in greater numbers of computer systems having access to greater numbers of electronic resources.
Networking is made possible by physical "routes" between computer systems, and the use of agreed upon communications "protocols." What protocol is chosen depends upon factors including the number of networked computer systems, the distances separating the computer systems, and the purposes of information exchange between the computer systems. Communications protocols can be very simplistic if only a few computer systems are networked together at close proximity. However, these communications protocols become more sophisticated as greater numbers of computer systems are added, and as computer systems are separated by greater distances.
The sophistication of communications protocols also varies with the type of information exchange. For instance, some protocols emphasize accuracy in sending large amounts of information, while others emphasize the speed of information transfer. The communications requirements of the applications running on a computer system network determine what type of protocol is chosen. An example of a computer application requiring real-time, reliable information transfer is a "cluster" management application.
Clustering is the networking of computer systems for the purpose of providing continuous resource availability and for sharing workload. A cluster of computer systems appears as one computer system from a computer system user's perspective, but actually is a network of computer systems backing each other up. In the event of an overload or failure on one computer system in a cluster, cluster management applications automatically reassign processing responsibilities for the failing computer system to another computer system in the cluster. Thus, from a user's perspective there is no interruption in the availability of resources.
Typically, one node or computer system in the cluster is assigned primary responsibility for an application (e.g., database, server) and other nodes are assigned backup responsibility. When the primary node for an application fails, the back up nodes in the cluster take over responsibility for that application. This ensures the high availability of that application.
Clustering is made possible through cluster management application programs running on each node/computer system within a cluster. These applications relay cluster messages back and forth across the cluster network to control cluster activities. Cluster messaging is also used to distribute updates about which computer systems within the cluster have specific primary and back 1 S up responsibilities for processing an application.
To ensure the high availability of applications running on the cluster, the cluster needs to be able to keep track of the status of all the nodes on a cluster. To do this, each computer system in a cluster continuously monitors each of the other computer systems within the same cluster to ensure that each computer system is performing its assigned processing task. Thus, when a node on a cluster fails, the node's primary responsibilities can be assigned to the backup nodes.
Unfortunately, it is not always possible to identify that a node in the cluster has failed. For example, if the network connection between one node and the rest of the cluster fails, the cluster will no longer be able to identify if that node is operating properly. If a node is still operating but its network connection to other nodes in the cluster has failed, then the node is said to have been "partitioned" from the cluster. When a node unexpectedly stops communicating with the rest of the cluster, it cannot be easily determined whether the node has failed or instead the node has been merely partitioned from the rest of the cluster. If the cluster incorrectly assumes the node has failed, and assigns a backup node primary responsibility for the application, the cluster can have two nodes both operating as the primary node. This condition can result in data inconsistencies in a database as both nodes may respond to requests from the cluster. If on the other hand, the cluster incorrectly assumes the "partitioned" node is still performing its primary applications and has only been partitioned from the cluster, and the cluster does not assign primary responsibility to a back up node, then the partitioned node's applications will no longer be available to the clients of the cluster (because the presumed partitioned node may in fact have failed operating altogether.) Thus, in many cases the cluster is unable to correctly respond to a non-communicating node without manual intervention by administrators. This condition may waste valuable processing time of the administrator. Further an administrator may not be immediately available to manually intervene and troubleshoot the problem at hand.
As more resources become accessible across computer system networks, the demand for continuous access to such network resources will grow. The demand for clusters as a means to provide continuous availability to such network resources will grow correspondingly. Without improved methods for determining the status of cluster nodes, the continuous availability of these resources will not be fully realized.
DISCLOSURE OF INVENTION
According to the present invention, a cluster node distress system is provided that improves the operational reliability of a cluster. The cluster node distress system provides a cluster node distress signal when a node within the cluster is about to fail. This allows the cluster to better determine whether a non-communicating node has failed or has merely been partitioned from the cluster. The preferred cluster node distress system is embedded deeply into the operating system and provides a node distress signal that can be quickly sent to other nodes in the cluster when an imminent failure of a node is detected. This improves the probability that the node distress signal will get out before the node totally fails. When the node distress signal is effectively sent to the cluster, the cluster can accurately determine whether the node has failed or has been just partitioned from the cluster. This determination allows the cluster to respond correctly, i.e., by assigning other nodes primary and backup responsibilities, and requires less manual intervention and troubleshooting by administrators. Thus, the preferred embodiment provides improved cluster reliability and decreased reliance on administrators.
ROC9-1998-O l 96 3 The foregoing and other features and advantages of the invention will be apparent from the following more particular description as set forth in the preferred embodiments of the invention, and as illustrated in the accompanying drawings.
BRIEF DESCRIPTION OF DRAWINGS
The preferred embodiments of the present invention will hereinafter be described in conjunction with the appended drawings, where like designations denote like elements, and:
FIG. 1 is a block diagram of an apparatus according to a preferred embodiment of the present invention;
FIG. 2 is a schematic diagram illustrating a cluster node distress signaller in accordance with the preferred embodiment of the present invention; and FIG. 3 is a flow diagram of a cluster distress signal method in accordance with the preferred embodiment of the present invention.
BEST MODE FOR CARRYING OUT THE INVENTION
The present invention relates to cluster communications. For those individuals who are not generally familiar with the concepts of clustering, the Overview section below presents many of the basic concepts and terminology that will help to understand the preferred embodiment of the present invention. Individuals skilled in the art of clustering may skip the Overview and proceed directly to the Detailed Description of this specification.
1. Overview Clustering is the linking together of computer systems, or groups of computer systems, in a way that allows the computer systems to share work and act as back ups for each other. As such, a cluster allows a continuous delivery of processing services even if one or more of the computer systems (or nodes) in the cluster fails. From the computer user's perspective, the cluster of computer systems appears as one computer system. Clustering is transparent to the users of computer clusters, who need not be aware of whether they are using one computer system or multiple computer systems. Instead, what matters to the users of computer clusters is that they have access to the resources they need, such as applications, databases, printers, files, etc. By clustering computer systems together, continuous availability to necessary resources can be achieved.
There are numerous advantages to clustering computer systems together. First, and most important, clusters provide higher availability by allowing computer systems within a cluster to back each other up. Second, clustering increases scalability, by allowing additional computer systems to be added as needed to satisfy increased processing demands or to improve processing power. Third, workloads can be balanced between computer systems within a cluster.
The individual computer systems that make up a cluster are also called "nodes." Technically, the term node can refer to processors, communications controllers, or terminals. However for the purposes of a cluster, a node refers to one of the individual computer systems within a cluster.
Typically, each node in a cluster is assigned primary and backup responsibilities in support of the cluster. Assigned responsibilities may be for one or for multiple functions such as providing access to data, performing computer applications, or providing access to hardware resources, as in printers, scanners, or fax machines. The nodes communicate to ensure that all nodes are functioning, that is, that clustering software on each node is operating and is actively monitoring for conditions that would require a switching from primary to backup responsibilities or going offline altogether.
Nodes are assigned primary and backup responsibilities. The primary node for each application is the node that performs the task and interacts with the users of the cluster. When the primary node becomes unable to perform its assigned functions, the cluster management applications must act to ensure that the user still has access to their resources, even if the processing task assigned to the node is unable to perform. This involves switching one of the backup nodes for that resource to have primary responsibility. In this manner, the cluster user still has continuous access to needed resources, even when the original node initially responsible for providing those resources becomes unavailable.
Cluster management applications and a communications infrastructure between all the nodes enable a cluster to operate as a single computer system from a user's perspective. For instance, messages are sent to every node to inform them about the situation with respect to the other nodes within the cluster. Messages are sent to every node to keep them up-to-date as to which nodes have primary and backup responsibilities for particular applications. This prevents the nodes from ROC9-1998-O l 96 5 attempting to perform conflicting operations, such as multiple nodes trying to behave as the primary node for a particular application. If two nodes are allowed to operate such that they are both operating as primary nodes, problems such as data inconsistencies can occur.
Thus, messages are sent to every node so that all nodes will be in agreement as to the appropriate action to take when any node is unable to perform its assigned responsibilities. Every node in the cluster must receive these cluster messages in the correct order to ensure proper functioning of the cluster.
One type of basic cluster messages is called "heartbeats". Heartbeats are low level messages sent between nodes to allow the cluster to keep track of which nodes are presently performing properly. For example, each node will typically send a logically adjacent node a heartbeat signal at regular intervals. Each node within the cluster thus expects to receive a heartbeat signal from its logically adjacent (neighbour) node at these regular intervals. If a node does not receive the proper heartbeat signal for a extended period of time, that node knows there is a potential problem with its neighbour node. If this failure to receive a heartbeat continues, the cluster management system will attempt to take appropriate action with respect to the malfunctioning node.
1 S Another way the cluster monitors its nodes is through "message timers". In a exemplary clustering system, if a message sent to a node fails, that node it will automatically be retried for a set period of time. Additionally, the message could be attempted using other addresses available for that node. If after multiple tries, the message is still undelivered (or unacknowledged), the cluster management system will know there is a problem and attempt to take appropriate action with respect to that malfunctioning node.
Sometimes it is not always possible for the cluster management system to know what is the appropriate action to take. For example, if the network connection between a node and the rest of the cluster fails, the cluster will no longer be able to tell if that non-communicating node is operating properly. If the non-communicating node is still operating but its network connection to other nodes in the cluster has failed, then the node is said to have been ''partitioned"
from the cluster. When a node unexpectedly stops communicating with the rest of the cluster, it cannot be easily determined whether the non-communicating node has failed to operate or instead has been merely partitioned from the rest of the cluster. If the cluster incorrectly assumes the node has failed to operate when it has only been partitioned, and assigns a backup node the primary responsibility for the partitioned node's applications, then the cluster can be left with two nodes both operating as the primary node for the same responsibilities. Again, this can result in data inconsistencies as both nodes respond to requests from the cluster. If on the other hand, the cluster incorrectly assumes the node has partitioned when in fact the node has actually failed, and does not assign primary responsibility for the failed node's applications to a backup node, then those applications will no longer be available to the user. Thus, in many cases the cluster is unable to correctly respond to a non-communicating node, unless an administrator intervenes manually to re-assign node responsibilities 2. Detailed Description According to the present invention, a cluster node distress system is provided that improves the operational reliability of a cluster. The cluster node distress system provides a cluster node distress signal when a node within the cluster is about the fail. This allows the cluster to better determine whether a non-communicating node has failed to operate or has merely been partitioned from the cluster. The preferred cluster node distress system is embedded deeply into the operating system and provides a node distress signal that can be quickly sent to other nodes in the cluster when an imminent failure of any node is detected. This improves the probability that the node distress signal will get out before the node totally fails. When the node distress signal is sent to the cluster, the cluster can accurately determine whether the node has failed to operate or has been partitioned from the cluster. This determination allows the cluster to respond correctly, i.e., by assigning other nodes primary and backup responsibilities, with less manual intervention and troubleshooting needed by administrators. Thus, the preferred embodiment provides improved cluster reliability and decreased reliance on administrators.
Referring now to FIG. 1, a computer system 100 in accordance with a preferred embodiment of the present invention is an AS/400 mid-range computer system. However, those skilled in the art will appreciate that the methods and apparatus of the present invention apply equally to any computer system, regardless of whether the computer system is a complicated multi-user computing apparatus or a single user device such as a personal computer or workstation.
For example, those skilled in the art will recognize that these functions can also be provided to other systems, such as IBM's OS/2~, OS/390~ and RS/6000~, Microsoft's Windows NT~, Novell's NetWareTM, Linux and other variations ofUNIXTM. Computer system 100 illustrates in detail the preferred embodiment of how one node in a cluster would be configured, where other nodes in the cluster reside on network 195. Computer system 100 suitably comprises a processor 110, a main memory 120, a memory controller 130, an auxiliary storage interface 140, a terminal interface 150, and a network interface 190, all of which are interconnected via a system bus 160. Note that various modifications, additions, or deletions may be made to computer system 100 illustrated in FIG.
1 within the scope of the present invention, such as the addition of cache memory or other peripheral devices. FIG. 1 is presented to simply illustrate some of the salient features of computer system 100.
Processor 110 performs computation and control functions of computer system 100, and comprises a suitable central processing unit (CPU). Processor 110 may comprise a single integrated circuit, such as a microprocessor, or may comprise any suitable number of integrated circuit devices and/or circuit boards working in cooperation to accomplish the functions of a processor. Processor 110 suitably executes computer programs within main memory 120 as needed.
Auxiliary storage interface 140 allows computer system 100 to store and retrieve information from auxiliary storage devices, such as magnetic disk (e.g., hard disks or floppy diskettes) or optical storage devices (e.g., CD-ROM). One suitable storage device is a direct access storage device (DASD) 170. As shown in FIG. 1, DASD 170 may be a floppy disk drive which may read programs and data from a floppy disk 180. It is important to note that while the present invention has been (and will continue to be) described in the context of a fully functional computer system, those skilled in the art will appreciate that the mechanisms of the present invention are capable of being distributed as a program product in a variety of forms, and that the present invention applies equally regardless of the particular type of signal bearing media to actually carry out the distribution.
Examples of signal bearing media include: recordable type media such as floppy disks (e.g., disk 180) and CD ROMS, and transmission type media such as digital and analog communication links, including wireless communication links.
Memory controller 130, through use of a memory processor (not shown) separate from processor 110, is responsible for moving requested information from main memory 120 and/or through auxiliary storage interface 140 to processor 110. While for the purposes of explanation, memory controller 130 is shown as a separate entity, those skilled in the art understand that, in practice, portions of the function provided by memory controller 130 may actually reside in the circuitry associated with processor 110, main memory 120, and/or auxiliary storage interface 140.
Terminal interface 150 allows system administrators and computer programmers to communicate with computer system 100, normally through programmable workstations. Although the system 100 depicted in FIG. 1 contains only a single main processor 110 and a single system bus 160, it should be understood that the present invention applies equally to computer systems having multiple processors and multiple system buses. Similarly, although the system bus 160 of the preferred embodiment is a typical hardwired, multidrop bus, any connection means that supports bi-directional communication in a computer-related environment could be used.
Network interface 190 supports the transfer of information between computer system 100 and remote computer systems in network 195. In the preferred embodiment, one or more nodes on the network 195 are similarly set up to work with computer system 100 as a cluster. Network interface 190 suitably includes one or more network interface adapters 193, each network interface adapter 193 typically implemented as an expansion card that can easily be added to computer systems such as computer system 100. Examples of network interface adapter 193 include Peripheral Component Interconnect (PCI) expansion cards, Industry Standard Architecture (ISA) expansion cards, proprietary adapter cards, and any type of adapters for interfacing with a network known now or invented in the future. Those skilled in the art will recognize that the functionality of network interface 190 can be implemented directly as part of main memory and processor 110.
Network 195 represents any type of network known to those skilled in the art.
This includes Internet, intranets, local area networks (LAN's), wide area networks (WAN's) or any configuration of hardware and software set up to act as a network, currently known or developed in the future, by which computer systems communicate with each other. On network 195 will exist other nodes (i.e.
other computer systems) which are also within the cluster.
Main memory 120 suitably contains one or more application programs 121, cluster management applications 122, and operating system 123, which includes a cluster node distress signaller 124. All of these programs in memory 120 are used in its broadest sense, and includes any and all forms of computer programs, including source code, intermediate code, machine code, and any other representation of a computer program.
In the preferred embodiment, application programs 121 comprises any program for which clustering will be used to provide increased reliability and scalability. As such, application programs 121 would typically include all the programs for which computer system 100 has responsibility as either a primary or a backup node. Examples of such application programs include web servers, file servers, database servers, etc.
Cluster management application 122 provides the mechanism needed to create and manage the cluster. This would include the processing of administrative requests for the management of a computer cluster. For example, this would preferably include the mechanisms for creating a cluster, adding and removing nodes to a cluster, etc.
In the preferred embodiment, cluster node distress signaller 124 is integrated with the operating system 123 to provide the quickest and most efficient means of sending node distress signals when the imminent failure of a node is detected. However, cluster node distress signaller 124 does not necessarily have to be integrated with the operating system 123.
It should be understood that main memory 120 will not necessarily contain all parts of all mechanisms shown at all times. For example, portions of application program 121, cluster management application 122, and operating system 123 may be loaded into an instruction cache (not shown) for processor 110 to execute, while other files may well be stored on magnetic or optical disk storage devices (not shown). In addition, although the computer programs are all shown to reside in the same memory location, it is to be understood that main memory 120 may consist of disparate memory locations. The term "memory" as used herein refers to any storage location in the virtual or real memory space of computer system 100.
It should also be understood that computer system 100 is exemplary of each node in a cluster, and that thus each node in the cluster would have the ability to quickly send node distress signals to other nodes in the cluster in the event of a node failure. The cluster management applications 122 on each of the other nodes can then respond appropriately, i.e., by assigning appropriate primary responsibility to other nodes within the cluster.
Turning now to FIG.2, the preferred embodiment cluster node distress signaller 124 is illustrated in more detail. As described above, the cluster node distress signaller 124 provides the mechanisms needed to send a distress signal to other nodes in a cluster upon the imminent failure of a node. This allows the cluster management application 122 (on the other nodes of the cluster) to accurately determine whether that node has failed, and has not just been merely partitioned from the cluster.
In the preferred embodiment, the cluster node distress signaller 124 is integrated into the operating system 123 to allow the cluster to respond quickly to the imminent failure of the node.
In the preferred embodiment, the cluster node distress signaller 124 includes a send distress signal method, a prebuilt distress message, and a dedicated distress signal task for sending the message.
In the preferred embodiment, there are two implementations of the send distress signal method available. One sends the distress signal message synchronously, meaning that all other methods to be performed during the shutdown are put on hold until that distress signal message is confirmed as sent. The other method sends the distress signal asynchronously, meaning that the node can continue processing other tasks after starting the send distress signal method.
In the preferred embodiment, the asynchronous method is used when there is sufficient time for an orderly shutdown of the node. Using the asynchronous method allows the current execution thread to perform other tasks in preparation for shutdown while the distress message is being sent.
Allowing the node's current execution thread to continue to perform other tasks while preparing and sending the distress message has the advantage of resulting in a more orderly shutdown, but the disadvantage of a potentially longer delay before the distress message is actually sent. Conversely, the synchronous method is preferable when failure is imminent and the distress message must be sent immediately. The synchronous method places all other processing on the current execution thread on hold until the distress message is sent, resulting in quicker priority sending of the node distress message to the cluster.
For example, if the failure event is a power outage but there remains battery backup power, there will be relatively more time before failure of the node to allow the distress message to be sent asynchronously. This allows for more parallel actions to be performed, such as other shutdown procedures, providing for a more orderly shutdown.
As another example, if the failure event is a hardware failure or a termination of the IP stack then there may not be time to wait, and it is preferable to send the distress message synchronously.
This results in an immediate sending of the prebuilt distress message, with the current execution thread not moving on to other processes until that message is sent. This allows the distress message to be sent as quickly as possible, and in some cases prevents the node from shutting down before the distress message can be sent to the cluster.
In the preferred embodiment, synchronous distress signal messages are processed in the current execution task, allowing the message to be sent more quickly. In particular, the current task can immediately send the message, while the waiting dedicated distress task may have to be awakened before the message can be sent. But, because the current task is used to send the distress message without allowing the node to operate until the message is sent, other actions that may need to be performed will be on hold. Thus, using a synchronous method to send the distress signal message improves the chances that the message is sent before the failure event occurs, but at the expense of other processes.
Conversely, asynchronous distress signal messages are sent to a task queue for the dedicated distress signal task, and run in its context. Typically, this task will need to be awakened to perform the sending of the distress signal message, but because there will be nothing else on its queue, it will still be faster than using a non-dedicated task. Once the asynchronous call is made to the dedicated distress task, the current task can continue on with its other processes, without waiting for the distress message to be sent.
The dedicated distress signal task for sending the message preferably comprises a low level execution thread in the operating system that functions as a process. This can be implemented as an instantiated task object waiting to be executed when called. When the asynchronous distress signal method is called, it calls the task object message queue. The next processor available will then pick up the code defined by this method and run the code. Because there is a dedicated task for sending the node distress signal, there is no possibility of the method having to wait in the task object message queue. Instead, it will be run by the next available processor.
Typical operating systems have many different tasks available to run at any given time, but only one task can be executed by each processor at a time. When the processor has to wait for something, then the current task is set aside and it goes onto the next task.
The prebuilt distress message preferably includes a previously instantiated message object that is ready to be sent in the event of a failure event. The message preferably includes a header that indicates it is a node distress message, and the ID of the node that sent the distress message. The message could also include data as to the reason for its failure, if known, although this is not required.
Turning now to FIG. 3, a method 500 for sending a node distress signal in accordance with the preferred embodiment is illustrated. The first step 502 is when the cluster node experiences a failure event. The next step 504 is for the node's operating system (OS) to detect the failure and call the distress signal method on the cluster node distress signaller.
In the preferred embodiment, any type of failure event can be detected, although in reality not all types of failure events can be adequately detected in time for the correct response. For example, one type of failure event is the loss of power. Preferably, the node can detect the loss of power while maintaining operation long enough for the cluster node distress signal to be sent. As described above, because the cluster node distress signaller includes a prebuilt distress signal, the node distress signal can be sent much quicker than the previous mechanism would allow. This increases the likelihood that the failure event was detected in time for the message to be sent before the node completely fails. As another example of a failure event, a take down of the protocol stack used to communicate with other members of the node can be detected by the node system long enough before it goes down to call and send the distress message.
Of course, these are just three types of failure events that could be detected by the node with sufficient advanced warning to send the cluster node distress signal. Other failure events could include impending crashes in operating system and failure of one or more hardware components (e.g., drives, network adapters, etc.) Those skilled in the art will thus recognize that the preferred embodiments of the present invention can be applied to any type of failure event that can be detected far enough in advance to send the distress signal, and that the ability of the preferred embodiment to quickly send the pre-built distress signal greatly reduces the advance warning needed.
Preferably, the operating system recognizes the type of failure event that is occurring, such that it can call the appropriate send distress signal method (i.e., asynchronous or synchronous) depending upon the amount of time before failure occurs. Additionally, if the operating system does not recognize the type of failure as being one that can be responded to asynchronously, it preferably calls the synchronous method to best assure that the distress message will be sent before node failure occurs.
The next step 506 is for the distress signal method to send a prebuilt distress signal using a dedicated distress signal task to the cluster. The prebuilt distress signal preferably comprises an instantiated message object that includes the name of the node sending the message. This allows the distress signal to be quickly put onto the protocol stack without having to wait for the message to be built first.
In the preferred embodiment, step 506 can be performed either asynchronously or synchronously, depending on the type of failure event detected and the amount of time remaining to send the prebuilt distress signal.
The next step 508 is for the distress signal method to assure that the node discontinues all primary and backup responsibilities. Again, it is undesirable for multiple nodes to all attempt to perform primary functions for some applications. Thus, by sending the node distress signal, the failing node informs the cluster that the failing node is no longer available, and must be replaced by an appropriate back up node. If for some reason this failing node recovers, it must still take itself offline or perform other duties because another backup node will have likely begun to perform the failing nodes previous primary responsibilities.
Thus, the preferred embodiment of the present invention provides a cluster node distress system and method that improves the reliability of a cluster. The cluster node distress system provides a cluster node distress signal when a node on the cluster is about to fail. This allows the cluster to better to determine whether the non-communicating node has failed to operate or has merely been partitioned from the cluster. The preferred cluster node distress system is embedded deeply into the operating system and provides a pre-built node distress signal that can be quickly sent to the cluster when an imminent failure of a node is detected, improving the probability that the node distress signal will get out before the node totally fails. When the node distress signal is effectively sent to the cluster, the cluster can accurately determine that the node has failed and has not j ust been partitioned from the cluster. This allows the cluster to respond correctly, i.e., by assigning other backup nodes primary responsibility, and requires less manual intervention and troubleshooting by administrators.
While the invention has been particularly shown and described with reference to preferred embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention.
Based upon various combinations of hardware (e.g., semiconductors, circuit boards, etc.) and software (e.g., computer programs), computer systems vary widely in design.
Many computer systems today are designed to "network" with other computer systems. Through networking, a single computer system can access information stored on and processed by other computer systems.
Thus, networking results in greater numbers of computer systems having access to greater numbers of electronic resources.
Networking is made possible by physical "routes" between computer systems, and the use of agreed upon communications "protocols." What protocol is chosen depends upon factors including the number of networked computer systems, the distances separating the computer systems, and the purposes of information exchange between the computer systems. Communications protocols can be very simplistic if only a few computer systems are networked together at close proximity. However, these communications protocols become more sophisticated as greater numbers of computer systems are added, and as computer systems are separated by greater distances.
The sophistication of communications protocols also varies with the type of information exchange. For instance, some protocols emphasize accuracy in sending large amounts of information, while others emphasize the speed of information transfer. The communications requirements of the applications running on a computer system network determine what type of protocol is chosen. An example of a computer application requiring real-time, reliable information transfer is a "cluster" management application.
Clustering is the networking of computer systems for the purpose of providing continuous resource availability and for sharing workload. A cluster of computer systems appears as one computer system from a computer system user's perspective, but actually is a network of computer systems backing each other up. In the event of an overload or failure on one computer system in a cluster, cluster management applications automatically reassign processing responsibilities for the failing computer system to another computer system in the cluster. Thus, from a user's perspective there is no interruption in the availability of resources.
Typically, one node or computer system in the cluster is assigned primary responsibility for an application (e.g., database, server) and other nodes are assigned backup responsibility. When the primary node for an application fails, the back up nodes in the cluster take over responsibility for that application. This ensures the high availability of that application.
Clustering is made possible through cluster management application programs running on each node/computer system within a cluster. These applications relay cluster messages back and forth across the cluster network to control cluster activities. Cluster messaging is also used to distribute updates about which computer systems within the cluster have specific primary and back 1 S up responsibilities for processing an application.
To ensure the high availability of applications running on the cluster, the cluster needs to be able to keep track of the status of all the nodes on a cluster. To do this, each computer system in a cluster continuously monitors each of the other computer systems within the same cluster to ensure that each computer system is performing its assigned processing task. Thus, when a node on a cluster fails, the node's primary responsibilities can be assigned to the backup nodes.
Unfortunately, it is not always possible to identify that a node in the cluster has failed. For example, if the network connection between one node and the rest of the cluster fails, the cluster will no longer be able to identify if that node is operating properly. If a node is still operating but its network connection to other nodes in the cluster has failed, then the node is said to have been "partitioned" from the cluster. When a node unexpectedly stops communicating with the rest of the cluster, it cannot be easily determined whether the node has failed or instead the node has been merely partitioned from the rest of the cluster. If the cluster incorrectly assumes the node has failed, and assigns a backup node primary responsibility for the application, the cluster can have two nodes both operating as the primary node. This condition can result in data inconsistencies in a database as both nodes may respond to requests from the cluster. If on the other hand, the cluster incorrectly assumes the "partitioned" node is still performing its primary applications and has only been partitioned from the cluster, and the cluster does not assign primary responsibility to a back up node, then the partitioned node's applications will no longer be available to the clients of the cluster (because the presumed partitioned node may in fact have failed operating altogether.) Thus, in many cases the cluster is unable to correctly respond to a non-communicating node without manual intervention by administrators. This condition may waste valuable processing time of the administrator. Further an administrator may not be immediately available to manually intervene and troubleshoot the problem at hand.
As more resources become accessible across computer system networks, the demand for continuous access to such network resources will grow. The demand for clusters as a means to provide continuous availability to such network resources will grow correspondingly. Without improved methods for determining the status of cluster nodes, the continuous availability of these resources will not be fully realized.
DISCLOSURE OF INVENTION
According to the present invention, a cluster node distress system is provided that improves the operational reliability of a cluster. The cluster node distress system provides a cluster node distress signal when a node within the cluster is about to fail. This allows the cluster to better determine whether a non-communicating node has failed or has merely been partitioned from the cluster. The preferred cluster node distress system is embedded deeply into the operating system and provides a node distress signal that can be quickly sent to other nodes in the cluster when an imminent failure of a node is detected. This improves the probability that the node distress signal will get out before the node totally fails. When the node distress signal is effectively sent to the cluster, the cluster can accurately determine whether the node has failed or has been just partitioned from the cluster. This determination allows the cluster to respond correctly, i.e., by assigning other nodes primary and backup responsibilities, and requires less manual intervention and troubleshooting by administrators. Thus, the preferred embodiment provides improved cluster reliability and decreased reliance on administrators.
ROC9-1998-O l 96 3 The foregoing and other features and advantages of the invention will be apparent from the following more particular description as set forth in the preferred embodiments of the invention, and as illustrated in the accompanying drawings.
BRIEF DESCRIPTION OF DRAWINGS
The preferred embodiments of the present invention will hereinafter be described in conjunction with the appended drawings, where like designations denote like elements, and:
FIG. 1 is a block diagram of an apparatus according to a preferred embodiment of the present invention;
FIG. 2 is a schematic diagram illustrating a cluster node distress signaller in accordance with the preferred embodiment of the present invention; and FIG. 3 is a flow diagram of a cluster distress signal method in accordance with the preferred embodiment of the present invention.
BEST MODE FOR CARRYING OUT THE INVENTION
The present invention relates to cluster communications. For those individuals who are not generally familiar with the concepts of clustering, the Overview section below presents many of the basic concepts and terminology that will help to understand the preferred embodiment of the present invention. Individuals skilled in the art of clustering may skip the Overview and proceed directly to the Detailed Description of this specification.
1. Overview Clustering is the linking together of computer systems, or groups of computer systems, in a way that allows the computer systems to share work and act as back ups for each other. As such, a cluster allows a continuous delivery of processing services even if one or more of the computer systems (or nodes) in the cluster fails. From the computer user's perspective, the cluster of computer systems appears as one computer system. Clustering is transparent to the users of computer clusters, who need not be aware of whether they are using one computer system or multiple computer systems. Instead, what matters to the users of computer clusters is that they have access to the resources they need, such as applications, databases, printers, files, etc. By clustering computer systems together, continuous availability to necessary resources can be achieved.
There are numerous advantages to clustering computer systems together. First, and most important, clusters provide higher availability by allowing computer systems within a cluster to back each other up. Second, clustering increases scalability, by allowing additional computer systems to be added as needed to satisfy increased processing demands or to improve processing power. Third, workloads can be balanced between computer systems within a cluster.
The individual computer systems that make up a cluster are also called "nodes." Technically, the term node can refer to processors, communications controllers, or terminals. However for the purposes of a cluster, a node refers to one of the individual computer systems within a cluster.
Typically, each node in a cluster is assigned primary and backup responsibilities in support of the cluster. Assigned responsibilities may be for one or for multiple functions such as providing access to data, performing computer applications, or providing access to hardware resources, as in printers, scanners, or fax machines. The nodes communicate to ensure that all nodes are functioning, that is, that clustering software on each node is operating and is actively monitoring for conditions that would require a switching from primary to backup responsibilities or going offline altogether.
Nodes are assigned primary and backup responsibilities. The primary node for each application is the node that performs the task and interacts with the users of the cluster. When the primary node becomes unable to perform its assigned functions, the cluster management applications must act to ensure that the user still has access to their resources, even if the processing task assigned to the node is unable to perform. This involves switching one of the backup nodes for that resource to have primary responsibility. In this manner, the cluster user still has continuous access to needed resources, even when the original node initially responsible for providing those resources becomes unavailable.
Cluster management applications and a communications infrastructure between all the nodes enable a cluster to operate as a single computer system from a user's perspective. For instance, messages are sent to every node to inform them about the situation with respect to the other nodes within the cluster. Messages are sent to every node to keep them up-to-date as to which nodes have primary and backup responsibilities for particular applications. This prevents the nodes from ROC9-1998-O l 96 5 attempting to perform conflicting operations, such as multiple nodes trying to behave as the primary node for a particular application. If two nodes are allowed to operate such that they are both operating as primary nodes, problems such as data inconsistencies can occur.
Thus, messages are sent to every node so that all nodes will be in agreement as to the appropriate action to take when any node is unable to perform its assigned responsibilities. Every node in the cluster must receive these cluster messages in the correct order to ensure proper functioning of the cluster.
One type of basic cluster messages is called "heartbeats". Heartbeats are low level messages sent between nodes to allow the cluster to keep track of which nodes are presently performing properly. For example, each node will typically send a logically adjacent node a heartbeat signal at regular intervals. Each node within the cluster thus expects to receive a heartbeat signal from its logically adjacent (neighbour) node at these regular intervals. If a node does not receive the proper heartbeat signal for a extended period of time, that node knows there is a potential problem with its neighbour node. If this failure to receive a heartbeat continues, the cluster management system will attempt to take appropriate action with respect to the malfunctioning node.
1 S Another way the cluster monitors its nodes is through "message timers". In a exemplary clustering system, if a message sent to a node fails, that node it will automatically be retried for a set period of time. Additionally, the message could be attempted using other addresses available for that node. If after multiple tries, the message is still undelivered (or unacknowledged), the cluster management system will know there is a problem and attempt to take appropriate action with respect to that malfunctioning node.
Sometimes it is not always possible for the cluster management system to know what is the appropriate action to take. For example, if the network connection between a node and the rest of the cluster fails, the cluster will no longer be able to tell if that non-communicating node is operating properly. If the non-communicating node is still operating but its network connection to other nodes in the cluster has failed, then the node is said to have been ''partitioned"
from the cluster. When a node unexpectedly stops communicating with the rest of the cluster, it cannot be easily determined whether the non-communicating node has failed to operate or instead has been merely partitioned from the rest of the cluster. If the cluster incorrectly assumes the node has failed to operate when it has only been partitioned, and assigns a backup node the primary responsibility for the partitioned node's applications, then the cluster can be left with two nodes both operating as the primary node for the same responsibilities. Again, this can result in data inconsistencies as both nodes respond to requests from the cluster. If on the other hand, the cluster incorrectly assumes the node has partitioned when in fact the node has actually failed, and does not assign primary responsibility for the failed node's applications to a backup node, then those applications will no longer be available to the user. Thus, in many cases the cluster is unable to correctly respond to a non-communicating node, unless an administrator intervenes manually to re-assign node responsibilities 2. Detailed Description According to the present invention, a cluster node distress system is provided that improves the operational reliability of a cluster. The cluster node distress system provides a cluster node distress signal when a node within the cluster is about the fail. This allows the cluster to better determine whether a non-communicating node has failed to operate or has merely been partitioned from the cluster. The preferred cluster node distress system is embedded deeply into the operating system and provides a node distress signal that can be quickly sent to other nodes in the cluster when an imminent failure of any node is detected. This improves the probability that the node distress signal will get out before the node totally fails. When the node distress signal is sent to the cluster, the cluster can accurately determine whether the node has failed to operate or has been partitioned from the cluster. This determination allows the cluster to respond correctly, i.e., by assigning other nodes primary and backup responsibilities, with less manual intervention and troubleshooting needed by administrators. Thus, the preferred embodiment provides improved cluster reliability and decreased reliance on administrators.
Referring now to FIG. 1, a computer system 100 in accordance with a preferred embodiment of the present invention is an AS/400 mid-range computer system. However, those skilled in the art will appreciate that the methods and apparatus of the present invention apply equally to any computer system, regardless of whether the computer system is a complicated multi-user computing apparatus or a single user device such as a personal computer or workstation.
For example, those skilled in the art will recognize that these functions can also be provided to other systems, such as IBM's OS/2~, OS/390~ and RS/6000~, Microsoft's Windows NT~, Novell's NetWareTM, Linux and other variations ofUNIXTM. Computer system 100 illustrates in detail the preferred embodiment of how one node in a cluster would be configured, where other nodes in the cluster reside on network 195. Computer system 100 suitably comprises a processor 110, a main memory 120, a memory controller 130, an auxiliary storage interface 140, a terminal interface 150, and a network interface 190, all of which are interconnected via a system bus 160. Note that various modifications, additions, or deletions may be made to computer system 100 illustrated in FIG.
1 within the scope of the present invention, such as the addition of cache memory or other peripheral devices. FIG. 1 is presented to simply illustrate some of the salient features of computer system 100.
Processor 110 performs computation and control functions of computer system 100, and comprises a suitable central processing unit (CPU). Processor 110 may comprise a single integrated circuit, such as a microprocessor, or may comprise any suitable number of integrated circuit devices and/or circuit boards working in cooperation to accomplish the functions of a processor. Processor 110 suitably executes computer programs within main memory 120 as needed.
Auxiliary storage interface 140 allows computer system 100 to store and retrieve information from auxiliary storage devices, such as magnetic disk (e.g., hard disks or floppy diskettes) or optical storage devices (e.g., CD-ROM). One suitable storage device is a direct access storage device (DASD) 170. As shown in FIG. 1, DASD 170 may be a floppy disk drive which may read programs and data from a floppy disk 180. It is important to note that while the present invention has been (and will continue to be) described in the context of a fully functional computer system, those skilled in the art will appreciate that the mechanisms of the present invention are capable of being distributed as a program product in a variety of forms, and that the present invention applies equally regardless of the particular type of signal bearing media to actually carry out the distribution.
Examples of signal bearing media include: recordable type media such as floppy disks (e.g., disk 180) and CD ROMS, and transmission type media such as digital and analog communication links, including wireless communication links.
Memory controller 130, through use of a memory processor (not shown) separate from processor 110, is responsible for moving requested information from main memory 120 and/or through auxiliary storage interface 140 to processor 110. While for the purposes of explanation, memory controller 130 is shown as a separate entity, those skilled in the art understand that, in practice, portions of the function provided by memory controller 130 may actually reside in the circuitry associated with processor 110, main memory 120, and/or auxiliary storage interface 140.
Terminal interface 150 allows system administrators and computer programmers to communicate with computer system 100, normally through programmable workstations. Although the system 100 depicted in FIG. 1 contains only a single main processor 110 and a single system bus 160, it should be understood that the present invention applies equally to computer systems having multiple processors and multiple system buses. Similarly, although the system bus 160 of the preferred embodiment is a typical hardwired, multidrop bus, any connection means that supports bi-directional communication in a computer-related environment could be used.
Network interface 190 supports the transfer of information between computer system 100 and remote computer systems in network 195. In the preferred embodiment, one or more nodes on the network 195 are similarly set up to work with computer system 100 as a cluster. Network interface 190 suitably includes one or more network interface adapters 193, each network interface adapter 193 typically implemented as an expansion card that can easily be added to computer systems such as computer system 100. Examples of network interface adapter 193 include Peripheral Component Interconnect (PCI) expansion cards, Industry Standard Architecture (ISA) expansion cards, proprietary adapter cards, and any type of adapters for interfacing with a network known now or invented in the future. Those skilled in the art will recognize that the functionality of network interface 190 can be implemented directly as part of main memory and processor 110.
Network 195 represents any type of network known to those skilled in the art.
This includes Internet, intranets, local area networks (LAN's), wide area networks (WAN's) or any configuration of hardware and software set up to act as a network, currently known or developed in the future, by which computer systems communicate with each other. On network 195 will exist other nodes (i.e.
other computer systems) which are also within the cluster.
Main memory 120 suitably contains one or more application programs 121, cluster management applications 122, and operating system 123, which includes a cluster node distress signaller 124. All of these programs in memory 120 are used in its broadest sense, and includes any and all forms of computer programs, including source code, intermediate code, machine code, and any other representation of a computer program.
In the preferred embodiment, application programs 121 comprises any program for which clustering will be used to provide increased reliability and scalability. As such, application programs 121 would typically include all the programs for which computer system 100 has responsibility as either a primary or a backup node. Examples of such application programs include web servers, file servers, database servers, etc.
Cluster management application 122 provides the mechanism needed to create and manage the cluster. This would include the processing of administrative requests for the management of a computer cluster. For example, this would preferably include the mechanisms for creating a cluster, adding and removing nodes to a cluster, etc.
In the preferred embodiment, cluster node distress signaller 124 is integrated with the operating system 123 to provide the quickest and most efficient means of sending node distress signals when the imminent failure of a node is detected. However, cluster node distress signaller 124 does not necessarily have to be integrated with the operating system 123.
It should be understood that main memory 120 will not necessarily contain all parts of all mechanisms shown at all times. For example, portions of application program 121, cluster management application 122, and operating system 123 may be loaded into an instruction cache (not shown) for processor 110 to execute, while other files may well be stored on magnetic or optical disk storage devices (not shown). In addition, although the computer programs are all shown to reside in the same memory location, it is to be understood that main memory 120 may consist of disparate memory locations. The term "memory" as used herein refers to any storage location in the virtual or real memory space of computer system 100.
It should also be understood that computer system 100 is exemplary of each node in a cluster, and that thus each node in the cluster would have the ability to quickly send node distress signals to other nodes in the cluster in the event of a node failure. The cluster management applications 122 on each of the other nodes can then respond appropriately, i.e., by assigning appropriate primary responsibility to other nodes within the cluster.
Turning now to FIG.2, the preferred embodiment cluster node distress signaller 124 is illustrated in more detail. As described above, the cluster node distress signaller 124 provides the mechanisms needed to send a distress signal to other nodes in a cluster upon the imminent failure of a node. This allows the cluster management application 122 (on the other nodes of the cluster) to accurately determine whether that node has failed, and has not just been merely partitioned from the cluster.
In the preferred embodiment, the cluster node distress signaller 124 is integrated into the operating system 123 to allow the cluster to respond quickly to the imminent failure of the node.
In the preferred embodiment, the cluster node distress signaller 124 includes a send distress signal method, a prebuilt distress message, and a dedicated distress signal task for sending the message.
In the preferred embodiment, there are two implementations of the send distress signal method available. One sends the distress signal message synchronously, meaning that all other methods to be performed during the shutdown are put on hold until that distress signal message is confirmed as sent. The other method sends the distress signal asynchronously, meaning that the node can continue processing other tasks after starting the send distress signal method.
In the preferred embodiment, the asynchronous method is used when there is sufficient time for an orderly shutdown of the node. Using the asynchronous method allows the current execution thread to perform other tasks in preparation for shutdown while the distress message is being sent.
Allowing the node's current execution thread to continue to perform other tasks while preparing and sending the distress message has the advantage of resulting in a more orderly shutdown, but the disadvantage of a potentially longer delay before the distress message is actually sent. Conversely, the synchronous method is preferable when failure is imminent and the distress message must be sent immediately. The synchronous method places all other processing on the current execution thread on hold until the distress message is sent, resulting in quicker priority sending of the node distress message to the cluster.
For example, if the failure event is a power outage but there remains battery backup power, there will be relatively more time before failure of the node to allow the distress message to be sent asynchronously. This allows for more parallel actions to be performed, such as other shutdown procedures, providing for a more orderly shutdown.
As another example, if the failure event is a hardware failure or a termination of the IP stack then there may not be time to wait, and it is preferable to send the distress message synchronously.
This results in an immediate sending of the prebuilt distress message, with the current execution thread not moving on to other processes until that message is sent. This allows the distress message to be sent as quickly as possible, and in some cases prevents the node from shutting down before the distress message can be sent to the cluster.
In the preferred embodiment, synchronous distress signal messages are processed in the current execution task, allowing the message to be sent more quickly. In particular, the current task can immediately send the message, while the waiting dedicated distress task may have to be awakened before the message can be sent. But, because the current task is used to send the distress message without allowing the node to operate until the message is sent, other actions that may need to be performed will be on hold. Thus, using a synchronous method to send the distress signal message improves the chances that the message is sent before the failure event occurs, but at the expense of other processes.
Conversely, asynchronous distress signal messages are sent to a task queue for the dedicated distress signal task, and run in its context. Typically, this task will need to be awakened to perform the sending of the distress signal message, but because there will be nothing else on its queue, it will still be faster than using a non-dedicated task. Once the asynchronous call is made to the dedicated distress task, the current task can continue on with its other processes, without waiting for the distress message to be sent.
The dedicated distress signal task for sending the message preferably comprises a low level execution thread in the operating system that functions as a process. This can be implemented as an instantiated task object waiting to be executed when called. When the asynchronous distress signal method is called, it calls the task object message queue. The next processor available will then pick up the code defined by this method and run the code. Because there is a dedicated task for sending the node distress signal, there is no possibility of the method having to wait in the task object message queue. Instead, it will be run by the next available processor.
Typical operating systems have many different tasks available to run at any given time, but only one task can be executed by each processor at a time. When the processor has to wait for something, then the current task is set aside and it goes onto the next task.
The prebuilt distress message preferably includes a previously instantiated message object that is ready to be sent in the event of a failure event. The message preferably includes a header that indicates it is a node distress message, and the ID of the node that sent the distress message. The message could also include data as to the reason for its failure, if known, although this is not required.
Turning now to FIG. 3, a method 500 for sending a node distress signal in accordance with the preferred embodiment is illustrated. The first step 502 is when the cluster node experiences a failure event. The next step 504 is for the node's operating system (OS) to detect the failure and call the distress signal method on the cluster node distress signaller.
In the preferred embodiment, any type of failure event can be detected, although in reality not all types of failure events can be adequately detected in time for the correct response. For example, one type of failure event is the loss of power. Preferably, the node can detect the loss of power while maintaining operation long enough for the cluster node distress signal to be sent. As described above, because the cluster node distress signaller includes a prebuilt distress signal, the node distress signal can be sent much quicker than the previous mechanism would allow. This increases the likelihood that the failure event was detected in time for the message to be sent before the node completely fails. As another example of a failure event, a take down of the protocol stack used to communicate with other members of the node can be detected by the node system long enough before it goes down to call and send the distress message.
Of course, these are just three types of failure events that could be detected by the node with sufficient advanced warning to send the cluster node distress signal. Other failure events could include impending crashes in operating system and failure of one or more hardware components (e.g., drives, network adapters, etc.) Those skilled in the art will thus recognize that the preferred embodiments of the present invention can be applied to any type of failure event that can be detected far enough in advance to send the distress signal, and that the ability of the preferred embodiment to quickly send the pre-built distress signal greatly reduces the advance warning needed.
Preferably, the operating system recognizes the type of failure event that is occurring, such that it can call the appropriate send distress signal method (i.e., asynchronous or synchronous) depending upon the amount of time before failure occurs. Additionally, if the operating system does not recognize the type of failure as being one that can be responded to asynchronously, it preferably calls the synchronous method to best assure that the distress message will be sent before node failure occurs.
The next step 506 is for the distress signal method to send a prebuilt distress signal using a dedicated distress signal task to the cluster. The prebuilt distress signal preferably comprises an instantiated message object that includes the name of the node sending the message. This allows the distress signal to be quickly put onto the protocol stack without having to wait for the message to be built first.
In the preferred embodiment, step 506 can be performed either asynchronously or synchronously, depending on the type of failure event detected and the amount of time remaining to send the prebuilt distress signal.
The next step 508 is for the distress signal method to assure that the node discontinues all primary and backup responsibilities. Again, it is undesirable for multiple nodes to all attempt to perform primary functions for some applications. Thus, by sending the node distress signal, the failing node informs the cluster that the failing node is no longer available, and must be replaced by an appropriate back up node. If for some reason this failing node recovers, it must still take itself offline or perform other duties because another backup node will have likely begun to perform the failing nodes previous primary responsibilities.
Thus, the preferred embodiment of the present invention provides a cluster node distress system and method that improves the reliability of a cluster. The cluster node distress system provides a cluster node distress signal when a node on the cluster is about to fail. This allows the cluster to better to determine whether the non-communicating node has failed to operate or has merely been partitioned from the cluster. The preferred cluster node distress system is embedded deeply into the operating system and provides a pre-built node distress signal that can be quickly sent to the cluster when an imminent failure of a node is detected, improving the probability that the node distress signal will get out before the node totally fails. When the node distress signal is effectively sent to the cluster, the cluster can accurately determine that the node has failed and has not j ust been partitioned from the cluster. This allows the cluster to respond correctly, i.e., by assigning other backup nodes primary responsibility, and requires less manual intervention and troubleshooting by administrators.
While the invention has been particularly shown and described with reference to preferred embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention.
Claims (38)
1. A node in a cluster of nodes, the node comprising:
at least one processor;
a memory coupled to at least one processor;
a cluster node distress signaler residing in said memory, the cluster node distress signaler including a prebuilt distress message indicating failure of the node on a cluster, the cluster node distress signaler sending the prebuilt distress message to the other nodes in the cluster when a failure event is detected.
at least one processor;
a memory coupled to at least one processor;
a cluster node distress signaler residing in said memory, the cluster node distress signaler including a prebuilt distress message indicating failure of the node on a cluster, the cluster node distress signaler sending the prebuilt distress message to the other nodes in the cluster when a failure event is detected.
2. The apparatus of wherein the cluster node distress signaler includes a dedicated distress signal execution task waiting to send the prebuilt distress message when a failure event is detected.
3. The apparatus of wherein the cluster distress signaler includes a method for sending the prebuilt distress message asynchronously and a method for sending the prebuilt distress message synchronously.
4. The apparatus of wherein the method for sending the prebuilt distress message asynchronously uses a current execution task to send to the prebuilt distress message and wherein the method for sending the prebuilt distress message synchronously uses a waiting dedicated distress signal execution task.
5. The apparatus of wherein said cluster distress node signaler uses the method for sending the prebuilt distress message asynchronously to allow for more orderly shutdown procedures when sufficient time before the failure event exists, and uses the method for sending the prebuilt distress message synchronously otherwise.
6. The apparatus of wherein the cluster distress node signaler includes a mechanism for assuring the node removes itself from the cluster after the cluster distress signal has been sent to the other nodes in the cluster.
7. The apparatus of wherein the cluster distress node signaler comprises an integrated part of an operating system residing in the memory.
8. The apparatus of wherein the prebuilt distress message comprises a previously instantiated message object.
9. A cluster node apparatus comprising:
at least one processor;
a memory coupled to at least one processor;
a cluster node distress signaler residing in said memory, the cluster node distress signaler including a prebuilt distress message indicating failure of the cluster node apparatus, wherein the cluster node distress signaler includes a method for sending the prebuilt distress message asynchronously that uses a current execution task and a method for sending the prebuilt distress message synchronously method uses a waiting dedicated distress signal execution task, and wherein the cluster node distress signaler sends the prebuilt distress message to the other nodes in the cluster when a failure event is detected.
at least one processor;
a memory coupled to at least one processor;
a cluster node distress signaler residing in said memory, the cluster node distress signaler including a prebuilt distress message indicating failure of the cluster node apparatus, wherein the cluster node distress signaler includes a method for sending the prebuilt distress message asynchronously that uses a current execution task and a method for sending the prebuilt distress message synchronously method uses a waiting dedicated distress signal execution task, and wherein the cluster node distress signaler sends the prebuilt distress message to the other nodes in the cluster when a failure event is detected.
10. The apparatus of wherein said cluster distress node signaler uses the method for sending the prebuilt distress message asynchronously to allow for more orderly shutdown procedures when sufficient time before the failure event exists, and user the method for sending the prebuilt distress message synchronously otherwise.
11. The apparatus of wherein the cluster distress node signaler includes a mechanism for assuring the cluster node apparatus removes itself from the cluster after the cluster distress signal has been sent to the other nodes in the cluster.
12. The apparatus of wherein the cluster distress node signaler is part of an operating system residing in memory.
13. The apparatus of wherein the prebuilt distress message comprises a previously instantiated message object.
14. A method comprising the steps of:
providing a prebuilt distress message indicating a node on a cluster is experiencing a failure event; and sending the prebuilt distress message when an impending failure event is detected within the node.
providing a prebuilt distress message indicating a node on a cluster is experiencing a failure event; and sending the prebuilt distress message when an impending failure event is detected within the node.
15. The method of further comprising the step of providing a dedicated distress signal task to process and send the prebuilt distress message.
16. The method of wherein the step of sending they prebuilt distress message comprises sending the prebuilt distress message asynchronously to allow for more orderly shutdown procedures when sufficient time before the impending failure event exists and sending the prebuilt distress message synchronously otherwise.
17. The method of further comprising the step of assuring the node removes itself from the cluster after the cluster distress signal has been sent to the other nodes in the cluster.
18. The method of further comprising the step of determining the type of impending failure event.
19. A method comprising the steps of:
providing a prebuilt distress message indicating a node on a cluster is experiencing a failure event;
providing a dedicated distress signal task to process and send the prebuilt distress message;
detecting an impending failure event within the node;
asynchronously sending the prebuilt distress message to other nodes in the cluster when sufficient time before the detected impending failure event exists and otherwise sending the prebuilt distress message to the other nodes synchronously; and assuring the node removes itself from the cluster after the cluster distress signal has been sent to the other nodes in the cluster.
providing a prebuilt distress message indicating a node on a cluster is experiencing a failure event;
providing a dedicated distress signal task to process and send the prebuilt distress message;
detecting an impending failure event within the node;
asynchronously sending the prebuilt distress message to other nodes in the cluster when sufficient time before the detected impending failure event exists and otherwise sending the prebuilt distress message to the other nodes synchronously; and assuring the node removes itself from the cluster after the cluster distress signal has been sent to the other nodes in the cluster.
20. The method of wherein the step of asynchronously sending the prebuilt distress message uses the dedicated distress signal task and wherein the step of synchronously sending the prebuilt distress message uses a current execution task.
21. The method of wherein the prebuilt distress message comprises a previously instantiated message object.
22. A computer program product comprising a computer readable media storing computer readable code which provides for:
a cluster node distress signaler including a prebuilt distress message indicating failure of a node on a cluster, the cluster node distress signaler sending the prebuilt distress message from the node to the other nodes in the cluster when a failure event is detected.
a cluster node distress signaler including a prebuilt distress message indicating failure of a node on a cluster, the cluster node distress signaler sending the prebuilt distress message from the node to the other nodes in the cluster when a failure event is detected.
23. The program product of wherein said signal bearing media comprises transmission media.
24. The program product of wherein said signal bearing media comprises recordable media.
25. The program product of wherein the cluster node distress signaler includes a dedicated distress signal execution task waiting to send the prebuilt distress message when a failure event is detected.
26. The program product of wherein the cluster distress signaler includes a method for sending the prebuilt distress message asynchronously and a method for sending the prebuilt distress message synchronously.
27. The program product of wherein the method for sending the prebuilt distress message asynchronously uses a current execution task to send to the prebuilt distress message and wherein the method for sending the prebuilt distress message synchronously uses a waiting dedicated distress signal execution task.
28. The program product of wherein said cluster distress node signaler uses the method for sending the prebuilt distress message asynchronously to allow for more orderly shutdown procedures when sufficient time before the failure event exists, and uses the method for sending the prebuilt distress message synchronously otherwise.
29. The program product of wherein the cluster distress node signaler includes a mechanism for assuring the node removes itself from the cluster after the cluster distress signal has been sent to the other nodes in the cluster.
30. The program product of wherein the cluster distress node signaler comprises an integrated part of an operating system residing in the memory.
31. The program product of wherein the prebuilt distress message comprises a previously instantiated message object.
32. A computer program product comprising a computer readable media storing computer readable code which provides for:
a cluster node distress signaler including a prebuilt distress message indicating failure of a cluster node apparatus, wherein the cluster rode distress signaler includes a method for sending the prebuilt distress message asynchronously that uses a current execution task and a method for sending the prebuilt distress message synchronously method uses a waiting dedicated distress signal execution task, and wherein the cluster node distress signaler sends the prebuilt distress message from the cluster node apparatus to the other nodes in the cluster when a failure event is detected.
a cluster node distress signaler including a prebuilt distress message indicating failure of a cluster node apparatus, wherein the cluster rode distress signaler includes a method for sending the prebuilt distress message asynchronously that uses a current execution task and a method for sending the prebuilt distress message synchronously method uses a waiting dedicated distress signal execution task, and wherein the cluster node distress signaler sends the prebuilt distress message from the cluster node apparatus to the other nodes in the cluster when a failure event is detected.
33. The program product of wherein said signal bearing media comprises transmission media.
34. The program product of wherein said signal bearing media comprises recordable media.
35. The program product of wherein said cluster distress node signaler uses the method for sending the prebuilt distress message asynchronously to allow for more orderly shutdown procedures when sufficient time before the failure event exists, and uses the method for sending the prebuilt distress message synchronously otherwise.
36. The program product of wherein the cluster distress node signaler includes a mechanism for assuring the cluster node apparatus removes itself from the cluster after the cluster distress signal has been sent to the other nodes in the cluster.
37. The program product of wherein the cluster distress node signaler is part of an operating system residing on the signal bearing media.
38. The program product of wherein the prebuiit distress message comprises a previously instantiated message object.
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| US09/281,026 US6442713B1 (en) | 1999-03-30 | 1999-03-30 | Cluster node distress signal |
| US09/281,026 | 1999-03-30 |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CA2290289A1 CA2290289A1 (en) | 2000-09-30 |
| CA2290289C true CA2290289C (en) | 2005-07-12 |
Family
ID=23075659
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CA002290289A Expired - Fee Related CA2290289C (en) | 1999-03-30 | 1999-11-22 | Cluster node distress signal |
Country Status (6)
| Country | Link |
|---|---|
| US (1) | US6442713B1 (en) |
| JP (1) | JP2000293497A (en) |
| KR (1) | KR100358663B1 (en) |
| CA (1) | CA2290289C (en) |
| SG (1) | SG90111A1 (en) |
| TW (1) | TW466399B (en) |
Families Citing this family (35)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6944183B1 (en) | 1999-06-10 | 2005-09-13 | Alcatel | Object model for network policy management |
| US7032022B1 (en) | 1999-06-10 | 2006-04-18 | Alcatel | Statistics aggregation for policy-based network |
| US6789213B2 (en) * | 2000-01-10 | 2004-09-07 | Sun Microsystems, Inc. | Controlled take over of services by remaining nodes of clustered computing system |
| US6594786B1 (en) * | 2000-01-31 | 2003-07-15 | Hewlett-Packard Development Company, Lp | Fault tolerant high availability meter |
| US6691244B1 (en) * | 2000-03-14 | 2004-02-10 | Sun Microsystems, Inc. | System and method for comprehensive availability management in a high-availability computer system |
| US7606898B1 (en) * | 2000-10-24 | 2009-10-20 | Microsoft Corporation | System and method for distributed management of shared computers |
| US6952766B2 (en) * | 2001-03-15 | 2005-10-04 | International Business Machines Corporation | Automated node restart in clustered computer system |
| US6918051B2 (en) * | 2001-04-06 | 2005-07-12 | International Business Machines Corporation | Node shutdown in clustered computer system |
| JP4491167B2 (en) * | 2001-04-27 | 2010-06-30 | 富士通株式会社 | Backup system for management device in communication system |
| US6880100B2 (en) * | 2001-07-18 | 2005-04-12 | Smartmatic Corp. | Peer-to-peer fault detection |
| US6925582B2 (en) | 2001-08-01 | 2005-08-02 | International Business Machines Corporation | Forwarding of diagnostic messages in a group |
| US7409427B2 (en) * | 2003-02-25 | 2008-08-05 | Bea Systems, Inc. | Systems and methods for lightweight conversations |
| US7689676B2 (en) | 2003-03-06 | 2010-03-30 | Microsoft Corporation | Model-based policy application |
| US7890543B2 (en) | 2003-03-06 | 2011-02-15 | Microsoft Corporation | Architecture for distributed computing system and automated design, deployment, and management of distributed applications |
| US8122106B2 (en) * | 2003-03-06 | 2012-02-21 | Microsoft Corporation | Integrating design, deployment, and management phases for systems |
| US7657781B1 (en) * | 2003-07-25 | 2010-02-02 | Cisco Technology, Inc. | System and method for providing redundant data load sharing in a distributed network |
| US7302607B2 (en) * | 2003-08-29 | 2007-11-27 | International Business Machines Corporation | Two node virtual shared disk cluster recovery |
| US7228462B2 (en) * | 2004-01-23 | 2007-06-05 | Hewlett-Packard Development Company, L.P. | Cluster node status detection and communication |
| US7778422B2 (en) * | 2004-02-27 | 2010-08-17 | Microsoft Corporation | Security associations for devices |
| US20050246529A1 (en) * | 2004-04-30 | 2005-11-03 | Microsoft Corporation | Isolated persistent identity storage for authentication of computing devies |
| DE102004032450B4 (en) * | 2004-06-29 | 2008-01-17 | Otten, Gert, Prof. Dr.med. | Surgical device for clamping organic tissue, in particular blood vessels |
| US7802144B2 (en) * | 2005-04-15 | 2010-09-21 | Microsoft Corporation | Model-based system monitoring |
| US7797147B2 (en) * | 2005-04-15 | 2010-09-14 | Microsoft Corporation | Model-based system monitoring |
| US8489728B2 (en) * | 2005-04-15 | 2013-07-16 | Microsoft Corporation | Model-based system monitoring |
| US20060235664A1 (en) * | 2005-04-15 | 2006-10-19 | Microsoft Corporation | Model-based capacity planning |
| US8549513B2 (en) | 2005-06-29 | 2013-10-01 | Microsoft Corporation | Model-based virtual system provisioning |
| US7941309B2 (en) | 2005-11-02 | 2011-05-10 | Microsoft Corporation | Modeling IT operations/policies |
| US8082340B2 (en) * | 2006-01-30 | 2011-12-20 | Cisco Technology, Inc. | Technique for distinguishing between link and node failure using bidirectional forwarding detection (BFD) |
| US7801997B2 (en) | 2006-03-30 | 2010-09-21 | International Business Machines Corporation | Asynchronous interconnect protocol for a clustered DBMS |
| US8312135B2 (en) * | 2007-02-02 | 2012-11-13 | Microsoft Corporation | Computing system infrastructure to administer distress messages |
| US7937610B2 (en) * | 2007-04-27 | 2011-05-03 | International Business Machines Corporation | Fast node failure detection via disk based last gasp mechanism |
| CN103297396B (en) * | 2012-02-28 | 2016-05-18 | 国际商业机器公司 | The apparatus and method that in cluster system, managing failures shifts |
| CN105574127A (en) * | 2015-12-12 | 2016-05-11 | 天津南大通用数据技术股份有限公司 | Quasi real-time disaster recovery method of distributed database system |
| CN113822395A (en) * | 2021-08-19 | 2021-12-21 | 中铁二十四局集团上海电务电化有限公司 | Submarine tunnel electric power thing networking wisdom management system |
| CN114095392B (en) * | 2021-12-02 | 2022-04-08 | 深圳市光网视科技有限公司 | Communication power supply monitoring method and system based on Internet of things |
Family Cites Families (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US4486826A (en) * | 1981-10-01 | 1984-12-04 | Stratus Computer, Inc. | Computer peripheral control apparatus |
| US5117352A (en) | 1989-10-20 | 1992-05-26 | Digital Equipment Corporation | Mechanism for fail-over notification |
| US5371852A (en) * | 1992-10-14 | 1994-12-06 | International Business Machines Corporation | Method and apparatus for making a cluster of computers appear as a single host on a network |
| US5590277A (en) * | 1994-06-22 | 1996-12-31 | Lucent Technologies Inc. | Progressive retry method and apparatus for software failure recovery in multi-process message-passing applications |
| US5440726A (en) * | 1994-06-22 | 1995-08-08 | At&T Corp. | Progressive retry method and apparatus having reusable software modules for software failure recovery in multi-process message-passing applications |
| US5627962A (en) * | 1994-12-30 | 1997-05-06 | Compaq Computer Corporation | Circuit for reassigning the power-on processor in a multiprocessing system |
| US5996075A (en) * | 1995-11-02 | 1999-11-30 | Sun Microsystems, Inc. | Method and apparatus for reliable disk fencing in a multicomputer system |
| US5805785A (en) * | 1996-02-27 | 1998-09-08 | International Business Machines Corporation | Method for monitoring and recovery of subsystems in a distributed/clustered system |
| GB2319369B (en) * | 1996-06-20 | 2000-10-25 | Fujitsu Ltd | Work inheriting system |
| KR19980058552A (en) * | 1996-12-30 | 1998-10-07 | 김영환 | Clustering device between two systems |
| US6151688A (en) * | 1997-02-21 | 2000-11-21 | Novell, Inc. | Resource management in a clustered computer system |
| US6108699A (en) * | 1997-06-27 | 2000-08-22 | Sun Microsystems, Inc. | System and method for modifying membership in a clustered distributed computer system and updating system configuration |
| US6003075A (en) * | 1997-07-07 | 1999-12-14 | International Business Machines Corporation | Enqueuing a configuration change in a network cluster and restore a prior configuration in a back up storage in reverse sequence ordered |
| US6192483B1 (en) * | 1997-10-21 | 2001-02-20 | Sun Microsystems, Inc. | Data integrity and availability in a distributed computer system |
| US6122735A (en) * | 1998-05-21 | 2000-09-19 | Intel Corp. | Fault resilient boot circuit using ring counter |
-
1999
- 1999-03-30 US US09/281,026 patent/US6442713B1/en not_active Expired - Lifetime
- 1999-11-22 CA CA002290289A patent/CA2290289C/en not_active Expired - Fee Related
- 1999-12-23 TW TW088122708A patent/TW466399B/en not_active IP Right Cessation
-
2000
- 2000-03-13 SG SG200001455A patent/SG90111A1/en unknown
- 2000-03-16 JP JP2000073269A patent/JP2000293497A/en active Pending
- 2000-03-22 KR KR1020000014476A patent/KR100358663B1/en not_active IP Right Cessation
Also Published As
| Publication number | Publication date |
|---|---|
| TW466399B (en) | 2001-12-01 |
| SG90111A1 (en) | 2002-07-23 |
| JP2000293497A (en) | 2000-10-20 |
| CA2290289A1 (en) | 2000-09-30 |
| KR20010006847A (en) | 2001-01-26 |
| KR100358663B1 (en) | 2002-10-30 |
| US6442713B1 (en) | 2002-08-27 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CA2290289C (en) | Cluster node distress signal | |
| US6477663B1 (en) | Method and apparatus for providing process pair protection for complex applications | |
| US8375363B2 (en) | Mechanism to change firmware in a high availability single processor system | |
| US6986076B1 (en) | Proactive method for ensuring availability in a clustered system | |
| US6839752B1 (en) | Group data sharing during membership change in clustered computer system | |
| US7076689B2 (en) | Use of unique XID range among multiple control processors | |
| US8185776B1 (en) | System and method for monitoring an application or service group within a cluster as a resource of another cluster | |
| US7401265B2 (en) | Scalable method of continuous monitoring the remotely accessible resources against the node failures for very large clusters | |
| KR100297906B1 (en) | Dynamic changes in configuration | |
| US7188237B2 (en) | Reboot manager usable to change firmware in a high availability single processor system | |
| JP3737695B2 (en) | System and method for transparent time-based selective software rejuvenation | |
| US20040153624A1 (en) | High availability synchronization architecture | |
| US20050108593A1 (en) | Cluster failover from physical node to virtual node | |
| US7065673B2 (en) | Staged startup after failover or reboot | |
| WO2012016175A1 (en) | Providing application high availability in highly-available virtual machine environments | |
| WO2002021276A1 (en) | A system and method for managing clusters containing multiple nodes | |
| US8943191B2 (en) | Detection of an unresponsive application in a high availability system | |
| JP2008052407A (en) | Cluster system | |
| JP2001043105A (en) | High-availability computer system and data backup method of the system | |
| US7003645B2 (en) | Use of a storage medium as a communications network for liveness determination in a high-availability cluster | |
| CN113765697B (en) | Method and system for managing logs of a data processing system and computer readable medium | |
| US5961650A (en) | Scheme to perform event rollup | |
| CN115145782A (en) | Server switching method, mooseFS system and storage medium | |
| Corsava et al. | Self-healing intelligent infrastructure for computational clusters | |
| CN111694634A (en) | Monitoring method and monitoring device for virtual machine |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| EEER | Examination request | ||
| MKLA | Lapsed | ||
| MKLA | Lapsed |
Effective date: 20101122 |