WO2000045553A2

WO2000045553A2 - A method to monitor and control server applications using low cost covert channels

Info

Publication number: WO2000045553A2
Application number: PCT/US2000/001598
Authority: WO
Inventors: Thomas Wong; Panagiotis Tsirigotis; Swee Lim; Sanjay Radia
Original assignee: Sun Microsystems, Inc.
Priority date: 1999-01-29
Filing date: 2000-01-21
Publication date: 2000-08-03
Also published as: DE60023449D1; JP2002536734A; US6412001B1; EP1145496A3; EP1145496B1; WO2000045553A3; EP1145496A2; DE60023449T2; AU3348300A

Abstract

A covert channel is established between a network service and one or more service monitors in a service group. The covert channel minimizes overhead by providing an indication of the status of the service through use of operating system utilities rather than conventional remote procedure calls (RPCs) or posting methods. The covert channel relies on one or more communication files established and updated by the service, at least one which has an attribute value in proportion to the workload of the service. By monitoring these attributes, the service monitor is able to determine the status of the service, including its workload and availability, without incurring costly operational overhead.

Description

A METHOD TO MONITOR AND CONTROL

SERVER APPLICATIONS USING LOW COST

COVERT CHANNELS

BACKGROUND

Field of the Invention

The invention relates to network service groups, and more particularly, to a method and apparatus for monitoring the status of services performed by such groups.

Description of Related Art

In a client-server networking environment, a network service is typically provided by an application running on a server machine that processes requests sent from many client machines via a network. A primary challenge is to provide a scalable and reliable platform for a network service to process an increasing volume of client requests. U.S. Patent No. 5,938,732, filed on December 9, 1996, entitled "Load Balancing and Failover of Network Services" and issued to Swee Boon Lim, Ashish Singhai and Sanjay Radia on August 17, 1999, describes a system having a scalable and reliable architecture that uses a group of server machines, each running the same application, to cooperatively provide a network service. To a client machine, the group of server machines, hereafter referred to as a service group, appears as a single server machine.

In such applications, one server machine may provide multiple network services and may belong to multiple service groups. Each server machine that is a member of a service group has a service monitor that monitors the workload and determines the availability status of the service on the server machine. Service monitors of members of a service group communicate with each other through the network. One of the service monitors is designated as the service group leader and periodically collects the workload and availability status of each member of the service group. The service group leader uses the information for load balance and recovery of service crashes.

There are three methods that a service monitor can use to obtain workload and availability status of a service. These are the direct method of remote procedure calls (RPCs), the direct method of posting, and the indirect method of using operating system services. Under the first, remote procedure call (RPC) method, the service monitor implements the client side of the RPC, and the service being monitored by the service monitor implements the server side of the RPC. The service monitor periodically makes a remote procedure call to the service in order to obtain workload information. If the service monitor does not receive an RPC reply from the service and a retry fails after a certain interval, the service monitor may assume that the service is dead.

While the RPC approach is straightforward, programming RPCs and setting up the RPC client and server are complex and expensive procedures, thereby making other options, such as the posting method, attractive. Under the posting method, a service periodically and separately publishes its workload and availability status (for example, at the current time) using operating system services such as shared memory or IP multi-cast or broadcast. A service monitor retrieves the published information from the service that it periodically monitors, possibly at a different interval than that in which publication by the service occurred, and determines the status of the service therefrom. The service monitor can thus detect that the service it is monitoring is not available if the time published by the service that represents service availability does not change after a certain interval. Again, however, a drawback of the posting method is the complex programming technique needed to program a service and its service monitor.

Under the indirect method of using operating system services, a service monitor can take advantage of utilities provided by the operating system and, for example, determine whether a computing process exists, or obtain the overall CPU (Central Processing Unit) utilization of the system, without directly communicating with the service it is monitoring. This is particularly useful since service monitors and applications running on a server machine are typically implemented as background computing processes in a server machine and thus lend themselves to such interaction. One operating system that provides such utilities is Solaris™ of Sun Microsystems, Inc.

As an example, a service monitor may rely on the CPU utilization of a server machine as a representation of the current workload of a service running on the server machine. There may be a direct relationship between the CPU utilization of a system and the workload of a service running in the system. For example, if the CPU utilization of a server machine is high, there may not be enough CPU cycles left over for the service in the server machine to process client requests, thereby indicating a high workload. Similarly, a service monitor may periodically check the existence of the computing process that implements the service. If the computing process of the service does not exist, the service is not available.

The indirect method is efficient because the cost of obtaining workload or availability status is merely the cost of using the system calls provided by the host operating system. In addition, the service monitor can obtain the workload and service availability information any time without waiting for the service to respond.

A drawback of the indirect method is that it cannot obtain workload information specific to a service. Also, the indirect method cannot really be sure that a service is available: it cannot distinguish a hung computing process that cannot process client requests from a normal computing process that can process client requests.

Thus, it is desirable for a service monitor to obtain workload information and availability status directly from the service that it monitors because such an approach would yield more accurate results. Since direct methods conventionally involve making changes to the services so that workload information specific to the services could be communicated to the service monitors, it is desirable that this information be communicated without using complex programming techniques such as shared memory, remote procedure calls or network programming. Finally, it is desirable to reduce the cost of obtaining this information in order to maximize service throughput.

SUMMARY OF THE INVENTION The present invention overcomes the deficiencies of the prior art by providing a covert channel to allow communication between a service monitor and a service that it monitors without incurring excessive overhead for monitoring and updating or passing messages indicating certain information. In one embodiment, the covert channel is a communication file the size of which corresponds to the workload of the service being monitored, such that the service monitor can determine the workload of the service by merely examining a size attribute of the communication file. Illustratively, the communication file is constantly updated by the service in order to provide a "heartbeat" to the service monitor to indicate that the service is available.

In another embodiment of the invention, the "heartbeat" is provided by a second communication file associated with that service. A separate communication file is particularly desirable in systems that do not provide a time stamp or last modification date of a file, in which case the service regularly or continuously modifies the size of the second communication file to indicate that the service is available.

A service or process running on a server in a multi-server environment thus periodically updates information about a communication file to indicate the status and availability of the service or process. The file is typically a "holey" file, that is, one that occupies no file system memory. The running process updates the size of the file to indicate, for example, the workload of the running process and the date-last-modified to indicate the availability of the running process. Any other process and/or service monitor, even those on other servers, need only examine the file attributes to determine the running process' status and availability. Thus, a covert channel is established between the running process and a monitoring process that eliminates all normal message processing overhead.

BRIEF DESCRIPTION OF THE DRAWINGS Many advantages of the present invention will be apparent to those skilled in the art with a reading of this specification in conjunction with the attached drawings, wherein like reference numerals are applied to like elements and wherein:

FIG. 1 is schematic diagram showing operation of a covert channel; FIG. 2 is a schematic diagram showing a service monitoring system in accordance with one embodiment of the invention; and

FIG. 3 is a schematic diagram showing a service monitoring system in accordance with an alternative embodiment of the invention.

DETAILED DESCRIPTION A covert channel is a communication channel that allows a computing process to transfer information by exploiting a mechanism not intended to be used for communication purposes. If one computing process can change the state of a characteristic that another process can sense, then a covert channel exists between the two processes. A covert storage channel uses shared system variables as the means of information transfer.

FIG. 1 illustrates a covert channel C established between Processes A and B designated as 30 and 32 respectively. The covert channel C arises through Process A's interaction with device or process 34 and Process B's separate interaction or access to that same device or process. Any manipulation of device or process 34 by Process A can be used to effectively communicate information about Process A to Process B, and vice versa if the channel is bi-directional. For more information on covert channels, see "Information Security, an Integrated Collection of Essays," edited by Marshall D. Abrams, Sushil Jajodia, Harold J. Podell, IEEE Computer Society Press, 1995, p. 117. A list of covert channel examples can be found in the paper "Multilevel Security with Fewer Fetters," M.D. Mcllroy and J.A. Reeds, Proceeding UNIX Security Workshop (pp. 24-31 ). Historically, the objective of a covert channel has been to break the multi-level security of a multi-level security enabled computing system. A multi-level secure system contains information with different sensitivities and simultaneously permits access by users with different security clearances and needs to know, but prevents users from obtaining access to information for which they lack authorization. (See "Information Security, an Integrated Collection of Essays," edited by Marshall D. Abrams, Sushil Jajodia, Harold J. Podell, IEEE Computer Society Press, 1995, pp. 330-349)

The communication efficiency of a covert channel usually is not of crucial importance, and each information transfer between the cooperating processes via the covert channel may incur expensive overhead.

In accordance with an embodiment of the present invention, use of the general purpose but complex and high overhead interprocess communication services provided by the operating system for publishing workload and availability status of a service is avoided. Instead, one or two simple covert channels are relied upon for conveying information regarding workload and availability status of a service. More specifically, the covert channel used in this invention is the size of a predetermined file. For lowest communication overhead, if a system supports holey files and a memory-based file system, the covert channel used is the size of a holey file in the memory-based file system. Relying on the covert channel, a service monitor thus acquires information indicative of the status of a service and uses this information to, inter alia, apprise other monitors of this status. The file that is agreed upon by a service and its service monitor to be used as the covert channel is hereinafter referred to as a communication file. As can be seen from FIG. 2 illustrating an embodiment of the invention, a service 36 has associated therewith a communication file 38. The workload of service 36 is represented as an integer number X corresponding to the percent of full capacity at which the service 36 is utilized. The service 36 publishes its workload periodically by setting the size of the communication file 38 to correspond to the workload number X. The service monitor 40 of service 38 periodically and independently reads the size of the communication file 38 to obtain the workload number X.

Communication file 38 can also be used to indicate the availability status of service 36 if the operating system of the server machine (not shown) provides a utility to obtain the last modification date of a file. Illustratively, the last modification date of a file changes if the contents or an attribute of the file changes. In such a system, service 36 continuously or regularly updates the workload number, even if the workload number does not change, thereby allowing its service monitor 40 to rely on the last modification date of the communication file 38 to infer the availability status of the service 36. If the last modification date does not change after a certain interval of time, the service monitor 40 may assume that the service 36 it is monitoring is not available.

Alternatively, a service and a service monitor can designate another, separate, communication file specifically for communicating service availability status. This embodiment of the invention is illustrated in FIG. 3, with service 42 being monitored by service monitor 44 via communication files 46 and 48. Communication file 46 operates as above, providing information about the percentage of utilization of the service 42.

Communication file 48 is dedicated to providing information about the availability status of service 42. This is accomplished by regularly altering the size of the communication file 48. In one implementation of this embodiment, service 42 periodically increments the size of the communication file 48 by one, thereby providing the equivalent of a "heartbeat" to indicate that the service 42 is "alive" or available. If the size of the communication file 48 does not increase after a certain interval, as sensed by service monitor 44, the service monitor may assume that the service 42 is not available.

In most operating system environments, updating the size of a communication file may require allocating disk storage for the file and may incur a disk I O (input/output) operation each time the file size is changed. However, these costs can be eliminated for certain systems that support a memory-based file system and holey files. Illustratively, a hole is a region in a file that does not occupy storage space and is treated as if all the data stored in the region are bytes having a value of zero. A holey file is a file that contains at least one hole.

A memory-based file system is a file system that uses random access memory (RAM) to store the contents of data files and the directory hierarchy of the file system. One advantage of a memory-based file system is speed, because accessing files in a memory-based file system has the same speed as accessing RAM. A drawback of a memory-based file system is that the contents of a file of a memory-based file system do not persist across system reboot. Solaris™ operating system of Sun Microsystems, Inc. supports a memory-based file system called tmpfs. The ufs file system in the Solaris™ operating system supports holey files.

If a holey file in a memory-based file system is designated as the communication file 38 by service 36 and its service monitor 40, updating the size of the communication file 38 does not incur a costly disk I/O (input/output) operation. Furthermore, if the communication file 38 does not contain any data (that is, it is a holey file just a hole), then no RAM allocation is needed for the file 38. Thus, accessing the communication file 38 has the same speed as accessing RAM, and the cost of setting or retrieving the size of the holey communication file 38 in a memory-based file system is about the same cost as a simple system call.

Utilization of file system services in the form of a covert channel in accordance with the invention is much simpler than writing RPC or network programs. Furthermore, for systems such as Sun Microsystems, Inc.'s Solaris™ operating system that support memory-based file systems and holey files, using a covert channel as described above to communicate workload information and service availability has very low communication overhead and is extremely efficient. It should be noted, however, that other file attributes may also be used as a covert channel for publishing workload and availability of a service such as service 36. However, among the available file attributes that can be used as a covert channel, file size is the simplest to use.

The above are exemplary modes of carrying out the invention and are not intended to be limiting. It will be apparent to those skilled in the art that modifications thereto can be made without departure from the spirit and scope of the invention as set forth in the following claims.

Claims

What Is Claimed Is:

1. A method for publishing the status of a service having a variable workload and being performed in a computer system having one or more computers, the method comprising the steps of. determining the workload of the service at a particular time; and establishing a first communication file of a size proportional to the determined workload.

2. The method of claim 1, further comprising the step of periodically updating the size of the first communication file.

3. The method of claim 2, wherein the step of periodically updating comprises periodically re-determining the workload of the service and re-establishing the size of the first communication file in accordance with the re-determination.

4. The method of claim 1 , further comprising the steps of: establishing a second communication file; and periodically changing the size of the second communication file to thereby indicate that the service is active.

5. The method of claim 1, wherein the steps of determining and establishing are performed by the service.

6. A method for communicating service status information to one or more service monitors of a service group having one or more computers, the method comprising the steps of: determining the workload of the service; establishing a first communication file; controlling an attribute of the first communication file in proportion with the workload of the service; and using the controlled attribute to provide an indication of the status of the service.

7. The method of claim 6, wherein the step of controlling comprises designating a size for the first communication file that is proportional to the workload of the service.

8. The method of claim 7, further comprising the step of periodically updating the size of the first communication file.

9. The method of claim 8, wherein the step of periodically updating comprises periodically re-determining the workload of the service and reestablishing the size of the first communication file in accordance with the re-determination.

10. The method of claim 6, further comprising the steps of: establishing a second communication file; and periodically changing an attribute of the second communication file.

11. The method of claim 10, wherein the attribute of the second communication file is the size of the second communication file.

12. The method of claim 6, wherein the steps of determining, establishing and controlling are performed by the service.

13. The method of claim 6, wherein the indication is acquired by a first service monitor, the first service monitor further performing the step of apprising at least a second service monitor of the status of the service.

14. A computer-readable medium containing a program that publishes the status of a service having a variable workload and being performed in a computer system having one or more computers, the program executing a procedure comprising the steps of: determining the workload of the service at a particular time; and establishing a first communication file of a size proportional to the determined workload.

15. The computer-readable medium of claim 14, wherein the procedure further comprises the step of periodically updating the size of the first communication file.

16. The computer-readable medium of claim 15, wherein the step of periodically updating comprises periodically re-determining the workload of the service and re-establishing the size of the first communication file in accordance with the re- determination.

17. The computer-readable medium of claim 14, wherein the procedure further comprises the steps of: establishing a second communication file; and periodically changing the size of the second communication file to thereby indicate that the service is active.

18. The computer-readable medium of claim 14, wherein the steps of determining and establishing are performed by the service.

19. A computer-readable medium containing a program that communicates service status information to one or more service monitors of a service group having one or more computers, the program executing a procedure comprising the steps of: determining the workload of the service; establishing a first communication file; controlling an attribute of the first communication file in proportion to the workload of the service; and using the controlled attribute to provide an indication of the status of the service.

20. The computer-readable medium of claim 19, wherein the step of controlling comprises designating a size for the first communication file that is proportional to the workload of the service.

21. The computer-readable medium of claim 20, the procedure further comprising the step of periodically updating the size of the first communication file.

22. The computer-readable medium of claim 21, wherein the step of periodically updating comprises periodically re-determining the workload of the service and re-establishing the size of the first communication file in accordance with the re- determination.

23. The computer-readable medium of claim 19, the procedure further comprising the steps of. establishing a second communication file; and periodically changing an attribute of the second communication file.

24. The computer-readable medium of claim 23, wherein the attribute of the second communication file is the size of the second communication file.

25. The computer-readable medium of claim 19, wherein the steps of determining, establishing and controlling are performed by the service.

26. The computer-readable medium of claim 19, wherein the indication is acquired by a first service monitor, the first service monitor further performing the step of apprising at least a second service monitor of the status of the service.