WO1995027254A1 - Graphically controlled method and system for data integration - Google Patents

Abstract

A method and system for accessing, processing and/or storing data in a variety of locations and formats. A user defines an action graph in a graphical editor to select data sources, filters and data destinations and the graph is executed by the system and method to obtain the desired result. A generic call interface and apllication sockets allow a variety of applications and data repositories to be employed, and the object-oriented basis of the system facilitates modifications and enhancements.

Description

  
 



   Graphically Controlled Method and System for Data Integration
 BACKGROUND OF THE INVENTION
FIELD OF THE INVENTION
 The present invention relates to a system for integrating data between two or more diverse programs and/or data storage systems. More specifically, the present invention relates to a integration system wherein a user may access and translate data between different systems and programs by defining an action graph within a graphical user interface.



  DESCRIPTION OF THE PRIOR ART
 In many environments it is common to have data which relates to different aspects of the environment stored in different formats and locations.



  For example, in business environments it is not uncommon that production planning data may be stored in one location and/or format while demand forecasting data may be stored in another location and/or format.



   Often it is required to use data stored in one format and/or location with a program which may be at another location and/or which may require data in a different format. In the past, this has required that a programmer create explicit macros, filters and/or programs to access the data and translate it into the required format. For example, in computer systems running the UNIX operating system, data may be accessed and translated through complex shell scripts which explicitly access the data storage systems and which invoke various UNIX programs such as SED (Stream Editor) or
AWK (a string parsing program) to translate the required data.



   While a knowledgable programmer can create powerful scripts and/or programs to perform the access and translation of data, problems eidst with this technique in that a less knowledgable (or non-programming) user cannot easily accomplish such a task. Further, any change in the storage or in the format of a data source or destination typically'breaks'the scripts/programs requiring a skilled programmer to modify the scripts/programs  accordingly. When several inter-related sources and destinations are involved, it is often the case that one or more programmers may be employed full time in producing new scripts and programs and correcting and amending existing scripts and programs as the sources and destinations evolve with time.



   SUMMARY OF THE INVENTION
 It is an object of the present invention to provide a novel method and system to access data from one or more data sources and to allow appropriate translation and supply of that data to one or more data destinations.



   According to one aspect of the present invention, there is provided a method defining a series of operations to access and process data comprising the steps of :
 (i) selecting a data connection from a plurality of data connections presented to a user;
 (ii) selecting a data table from a plurality of data tables presented to said user which are related to said selected data connection;
 (iii) selecting at least one data processing action to operate on said selected data table from a plurality of data processing actions presented to said user;
 (iv) selecting a destination for the data produced by said at least one data processing action;

   and
 (v) executing each of said at least one data processing actions.
 According to another aspect of the present invention, there is provided a method of accessing and processing data comprising the steps of :
 (i) defining at least one data repository;
 (ii) defining at least one data table for said at least one defined data repository;
 (iii) with a graphical user interface, selecting at least one of said data connections;  
 (iv) with said graphical user interface, selecting said at least one data table from said selected data connection;
 (v) with said graphical user interface, selecting at least one of a plurality of data processing actions and relating each of said at least one data processing actions to appropriate ones of said at least one data table;

  
 (vi) with said graphical user interface, defining a destination for data produced by each of said at least one data processing actions;
 (vii) executing each of said at least one data processing actions.



   According to yet another aspect of the present invention, there is provided a system for accessing and processing data comprising: means to define at least one data connection to a data repository; means to define at least one data table for each of said at least one data connections; graphical user interface means to select at least one predefined data processing action and to relate said selected data processing action to one or more of said at least one data tables through the definition of an action graph.



   According to yet another aspect of the present invention, there is provided a system for accessing and processing data comprising: data access means to access at least one data repository; data processing action means to perform at least one data processing action on data from said at least one data repository; graphical control means to allow a user to select said at least one data repository and said at least one data processing action graphically.



   According to yet another aspect of the present invention, there is provided a method to allow a user to define graphically an action graph representing a series of operations to access and process data comprising the steps of :
 (i) graphically presenting to a user a plurality of predefined data connections from which the user selects at least one data connection;
 (ii) graphically presenting to a user a plurality of predefined data tables which are related to said at least one selected data connection from which the user selects at least one data table;  
 (iii) graphically presenting to a user a plurality of predefined data processing actions to operate on said at least one selected data table from which the user selects at least on data processing action;

  
 (iv) graphically presenting to a user at least one destination for the data produced by said at least one data processing action from which the user selects a destination for said data;
 (v) graphically connecting said at least one data table, said at least one data processing action and said at least one destination for said data to define a data flow; and
 (vi) graphically presenting a user with the option to execute each of said at least one data processing actions in said action graph.



   BRIEF DESCRIPTION OF THE DRAWINGS
 An embodiment of the present invention will be described, by way of example only, with reference to the accompanying drawings, in which:
 Figure 1 shows a representation of the structure of the present invention in block diagram form;
 Figure 2 shows an example of an action graph in accordance with the present invention;
 Figure 3 shows a representation of a Data Access module of the present invention in block diagram form;
 Figure 4 shows a Storage class hierarchy employed in the preferred embodiment of the present invention;
 Figure 5 shows a view of the user interface of a Graphical Control module of the present invention;
 Figure 6 shows a Data Connection dialog box in the user interface of Figure 5;
 Figure 7 shows a Data Connection Creation dialog box in the user interface of Figure 5;

    
 Figure 8 shows a Data Object dialog box in the user interface of
Figure 5;
 Figure 9 shows a Select Action Graph dialog box in the user interface of Figure 5;
 Figure 10 shows an Action Graph Editor in the user interface of
Figure 5;
 Figure 11 shows a Join Filter Definition dialog box in the user interface of Figure 5;
 Figure 12 shows an Aggregation Filter dialog box in the user interface of Figure 5;
 Figure 13 shows a Selection Filter dialog box in the user interface of Figure 5;
 Figure 14 shows a modified version of the action graph of Figure 2;
 Figure 15 shows another modified version of the action graph of
Figure 2; and
 Figure 16 shows a modified version of the action graph of Figure 15.



   DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT
 Figure 1 shows, in block diagram form, a representation of the system of the present invention which is indicated generally at 20. As shown in the Figure, system 20 interacts with various applications 24 and various data repositories 28. System 20 comprises a Filter/Translator module 32, a Data
Access module 36 and a Graphical Control module 40.



   Graphical Control module 40, described in more detail below, presents a user with a graphical tool to create an action graph which defines: the selection of, and connection to, appropriate data repositories; any Site and/or semantic translators required; and the destination application or intermediate applications. When an action graph is defined, the resulting set of instructions is transparently sent to Filter/Translator module 32, Data Access  module 36, the appropriate data repositories 28 and/or the appropriate applications 24 to accomplish the operations defined by the user in the action graph.



   Figure 2 shows an example of a simple graph designed in
Graphical Control module 40 and implemented in system 20. As described in more detail below, in the example the user has identified two data repositories (within databases), two filters and a destination data repository. Specifically, the user has selected source data repositories labelled"Bill of Materials"and "Production Forecast", join and aggregation filters labelled"Product Materials" and"Materials Calculate"respectively, and a destination data repository labelled"Raw Materials Ordering". As described below, each of the labels for the action graph components is specified by the user to increase the user friendliness and comprehensibility of the action graph.



   This example action graph defines a series of operations to determine the amount and type of raw materials which need to be ordered to produce one or more products. The"Bill of Materials"database contains information defining how much and the type of raw materials required to make each product and the"Production Forecast"database contains the information defining how much of each finished product is expected to be required in the next month. The join filter,"Product Materials", joins related data from the two databases and the aggregate filter,"Materials Calculate", performs the calculations necessary to determine the required quantities of raw materials.



  The resultant data is then forwarded to the"Raw Materials Ordering"database.



  This latter database may then be used to produce raw material orders in the conventional manner.



   Once an action graph has been created, it can be invoked in several different manners including, invoking it from Graphical Control module 40 or setting a'trigger'to invoke it at a regular, pre-defined time (such as every Friday morning at 10: 00AM) or whenever a pre-defined event occurs (such as an update to the data in the Production Forecast database).  



   Figure 3 shows a representation of Data Access module 36 in block diagram format. In order to allow Data Access module 36 to operate with a wide variety of applications, the uppermost layer of Data Access module 36 comprises a series of application sockets 60, one socket for each application which is intended to interact with system 20. Each socket 60 provides a means for an application to exchange instructions, data and/or data objects with a
Generic Call Interface 64 which is the next lower level of Data Access module 36. If an application is specifically developed to provide interaction with 20, its socket 60 merely passes the appropriately formatted instructions and/or data to and from Generic Call Interface 64.

   When system 20 is to interact with an application which does not explicitly support interaction with system 20, socket 60 may provide a'wrapper'around the application, either through shell scripting or through a control language interface (such as TEL), or socket 60 may merely allow data to be transferred directly into and out of the application's data space.



   The next level beneath Generic Call Interface 64 comprises a
Connectivity Information module 68 and a Data Binding Information module 72. Connectivity Information module 68 provides the necessary connectivity information to access various data repositories such as a storage type (i. e.
Oracle, Sybase, etc.), a database name (i. e.-production forecast), a username and password and a representative label that identifies the significance of the connection. For example, if a connection is defined to access the abovementioned Production Forecasting database to extract data for determining a raw materials order, the label might be"Forecast Raw Materials".



   Data Binding Information module 72 provides the mapping of logical data structures and/or data objects to their physical representation in the databases. Specifically, table and column names are specified for relational databases, file and positioning information is specified for ASCII-style files, etc.  



   At the bottom level of Data Access module 36 are one or more database clients 76. Database clients 76 are not particularly limited and may be RDBMS (Relational Database Management Systems), ASCII Files (either fixed length or token delimited), ODBC (Open Database Connectivity) or
OODBMS (Object Oriented Database Management Systems) implemented in data repositories 28. In the case wherein a user may want to access an otherwise unsupported database client (DB2 for example), Connectivity
Information module 68 and Data Binding Information module 72 may communicate with a third party translation/access product 80 (referred to as
Middleware) which provides the desired interoperability. It is contemplated that, in this manner, Data Access module 36 will be readily adapted to a wide range or database clients.



   In the presently preferred embodiment, Data Access module 36 is implemented in C + + using Object Oriented design principals. Figure 4 shows the class hierarchy for the Storage class 100 used to implement Data
Access module 36. The class hierarchy shown provides a common interface to all supported data repositories including methods for connection initialization, tuple management, table management, and transaction control.



  Storage class 100 is the parent class of all the classes within the hierarchy and is essentially a virtual class which establishes Generic Call Interface 64 through the methods defined therein. The present definition of Storage class 100 is given in Appendix 1 attached hereto.



   A subclass of Storage class 100 is defined for each connection type which is defined in Graphical Control Module 40, as described below.



  The ASCII File class 104 is a subclass of Storage Class 100 and, through polymorphism, provides generic methods to access ASCII Files. The Fixed
Format class 108 is, in turn, a subclass of ASCII File class 104 and provides specific methods to access fixed length ASCII files. Similarly, Token
Delimited class 112 is a subclass of ASCII File class 104 and provides specific methods to access token delimited ASCII files.  



   ANSI SQL class 116 is a subclass of Storage class 100 and provides generic access methods which produce ANSI standard SQL statements. Sybase class 120 is a subclass of ANSI SQL class 116 and provides access methods which interface with the SQL interface library suppliez by Sybase. Similarly, Oracle class 124 is a subclass of ANSI SQL class 116 and provides access methods which interface with the SQL interface library provided by Oracle.



   Many vendors of database engines provide a superset of ANSI
SQL which provides additional functionality and/or optimizations. It is contemplated that the present invention provides a particular advantage in that the sub-classes implemented for a specific database engine can employ any additional functionality and/or optimization provided by the database vendor.



  In this manner, the system performance is not limited to a'lowest common denominator'of database functionality. Further, if a vendor provides an upgrade with additional functionality it is only necessary to re-implement the appropriate sub-class to employ that additional functionality.



   Middleware class 128 is also a subclass of ANSI SQL class 116 which will provide access methods to interface to SQL libraries in a third party middleware product such as the EDA/SQL product sold by Information
Builders Inc which provides a DB2 SQL interface. A Middleware class 128 will be implemented for each third party middleware product which is to be supported by system 20. Finally, ODBC class 132 is also a subclass of ANSI
SQL class 116 and provides operators to access a database which is compliant with Microsoft's ODBC standard.



   As will be apparent to those of skill in the art, the class hierarchy shown in Figure 4 has been designed such that, to add the capability to access an additional database product, the creation of an appropriate subclass need only be undertaken. For example, when an object oriented database (OODBMS) standard emerges, only an OODBMS subclass 136 need be  designed to provide access to this potentially important new type of data repository.



   Filter/Translator module 32 provides several types of filters from which the user can select and/or define the necessary filtering and translation operations required for an action graph. Specifically, the system includes: selection filters allowing the combining of data sets and/or extraction of subsets meeting a specified criteria; simple computational filters allowing calculation of various computed values from a data set; aggregation filters which identify data within a set that have the same'key'to allowing grouping and/or computations; matrix manipulation filters allowing matrix operations on a data set; expression filters to provide parsing and string manipulation of various expressions; and various custom filters which may be programmed by users or third party developers to provide additional functionality as required.



   Filters can be combined and/or arranged to perform complex semantic translations. For example, a semantic translator may operate on data representing the total monthly demand for hand soap in the eastern region to translate it into the requirements for each specific size of hand soap bar to be made in a particular factory in a week. Such a semantic translation may involve three filter steps (translations): from the hand soap family to the individual bars of different sizes; from the eastern region to the specific factory; and from monthly forecast demand to weekly production requirements.



   In the preferred embodiment of system 20, Filter/Translator module 32 is also implemented in C + + using Object Oriented design principals. The present definition of the Filter class is given in Appendix 2 attached hereto.



   As with Storage Class 100, the Filter class described above is a virtual class used when instantiating specific filter subclasses as required.



  Essentially, the Filter class defines a generic interface with which the specific filter algorithms are implemented. The generic interface allows a user to select appropriate pre-existing filter types, define various filter criteria (expressions)  and link the filters with other filters and/or applications and/or data repositories as desired.



   In the preferred embodiment, filters are implemented in three categories, namely: light weight filters (LWF) which perform simple operations such as multiplying two number fields or extracting a sub-string from a string; middle weight filters (MWF) which perform more complex operations such as simple data group manipulations including calculating averages, minimums, etc.; and heavy weight functions (HWF) which perform complex data group manipulations such as Days of Supply, standard deviation, etc.



   As mentioned above, with reference to Figure 2, system 20 includes aggregation and join filters as well as other operations, including: selection filters; composite filters; rolling aggregation filters; aggregation/join filters; bucketizers; and others. Each of the filters is implemented as a subclass of the Filter class or as a subclass of a subclass of the Filter class. A brief description of some of the more common filters and operations is presented below.



   Aggregation filters group data from various sources based upon some defined characterizing properties. An aggregation filter may, for example, compute the total production for each product family from the production figures for the individual products.



   Join filters take data from two or more different sources, either in the same database or different databases, and combines the data according to some characterizing property. A join filter may, for example, combine a commodity name used in one database with the commodity description used in another database with the resulting data set being used in a report, etc. A join filter generates three output data sets, namely: the joined data; exception data for the first input data set which cannot be joined with data in the second data set; and exception data for the second input data set which cannot be combined with the first data set. Any combination of these outputs may be used as desired.  



   Selection filters only select elements from a data set that meet predefined selection criteria. A selection filter may, for example, select only overtime entries from time for work crews at various plant sites.



   Rolling Aggregation filters are similar to the above-described aggre, niters but the grouping and calculation is performed over a specified time period. A rolling aggregation filter may, for example, calculate a four month rolling average of production at a plant.



   Aggregation/Join filters combine functionality from each of the aggregation and join types to provide a filter which can take data based upon one breakdown of the data domain and present that data in terms of an alternate breakdown of the data domain. An aggregation/join filter may, for example, convert data based upon product families into comparable results for individual products within the families.



   As mentioned above, users are also able to combine two or more filters into a composite filter to provide additional functionality. The selection and/or definition of specific filters is accomplished through Graphical Control module 40, discussed below.



   The bucketizer filter allows a user to combine events, and dates when those events occur, into'time buckets'. Essentially, the bucketizer allows a user to group discrete events within time groups. For example, a user may combine data sets for various products into a bucket which represents the demand for all of the products within a week.



   Action graphs are defined by users within Graphical Control module 40. Figure 5 shows the screen 200 of a session of Graphical Control module 40 and the icon selection box 204. The icon selection box includes a
Connections icon 208, a Data Table icon 210 and an Action icon 212.



   To specify a data connection, a user selects Connections icon 208 and the data connections dialog box 220 opens, as shown in Figure 6. The user then selects a predefined, or defines a new, connection which defines the logical characteristics for each storage entry the user may wish to access from  system 20. As shown in the Figure, each connection has a representative name such as"Plant Data"and the type of connection such as"Oracle"is indicated to the user.



   If the user wishes to create a, the NEW button is selected and the
Data Connection Creation dialog box 224 opens, as shown ils Figure 7. This is where the user defines a representative name for the connection such as "Forecast Connection"and defines the connection to a particular data repository with a representative name such as Sybase, ASCII Fixed file, etc.
 a connection has been selected or defined, the user selects
Data Table icon 210 to define the specific tables within a connection which are of interest. When Data table icon 210 is selected, the Data Table dialog box 228, shown in Figure 8, is presented to the user. Data Table dialog box 228 allows the user to select either a predefined table definition, such as"Bill Of
Materials", edit an existing table definition, or to define a new table definition as desired.

   If the user wishes to define a new table definition, a Data Table definition dialog box (not shown but similar to New Connection dialog box 224) is presented to the user which includes a list of available connections and their respective tables.



   To define or modify an action graph, the user selects Action icon 212 to open the Action dialog box 240, as shown in Figure 9. As with the connection and data table dialog boxes, at this point the user may select either a predefined action graph or may create a new action graph. In either case, the
Action editor 244 opens, as shown in Figure 10, and either the selected action graph is displayed or a blank screen is displayed for the user to define a new action graph. In the Figure, the action graph displayed has been previously defined and is the same as that shown in Figure 2.



   To design or modify an action graph, the user can select from a toolbar 246 of icons. Toolbar 246 includes: Data Object icon 250, Selection
Filter icon 254 ; Aggregation Filter icon 258; Join Filter icon 262; Bucketizer icon 251; Input icon 252; Output icon 253; Data Flow icon 255 and Data  
Editor icon 264. Toolbar 246 may also include icons to connect to other systems/tools such as Schedulex icon 265, which is used to establish a connection to the Schedulex program sold by the assignee of the present invention; Linx icon 266, which is used to establish a connection to the Linx program sold by the assignee of the present invention; and a Script icon 267 which represents a script, such as a UNIX shell or TCL script.



   Action editor 244 also includes: Run One icon 268 which is selected to execute a single operation in an action graph; Run icon 269 which is selected to run all of the operations in an action graph; and Select icon 270 which is used to enter select mode to select or de-select icons in an action graph. In addition, Controlled Start icon 271 allows a user to execute an action graph up to a selected operation (icon).



   When the user selects data-table icon 250, a dialog box (not shown) appears listing all the available data tables from which the user may select a data table of interest. When the table is selected, the user places it in the action editor workspace 274 where it is represented with an appropriate icon and the data table name is shown adjacent it. The user may select multiple data tables as desired.



   To provide any necessary filtering or semantic translation, the user then selects one or more appropriate operations from Selection Filter icon 254, Aggregation Filter icon 258, Join filter icon 262, Input icon 252, Output icon 253 or Bucketizer icon 251. The user places the selected operation icon in the action editor workspace 274 and repeats the process for as many instances of the operation of that type as are required. The user may then select a different operation icon to select and place other operations as required.



   Once the data tables and appropriate operations are selected, the user uses Select icon 270 to define the flow of data within the action graph.



  Data flow in the present invention is left to right and top to bottom.



  Essentially, the user draws an appropriate series of links between icons representing the various data tables and filters which have been placed in the  action editor workspace 274. As shown in the Figures, these links comprise straight lines which extend between attachment points on the icons in an action graph.



   Generally, there are two types of links in an action graph: data links which represent the flow of data between various operations and/or locations in the action graph; and dependency links which represent a dependency between operations and/or locations and which may also define a flow of data. The type of link is established by the attachment point to which the link connects. A link to an attachment point on the left side of an icon means that the operation or location represented by the icon receives data from the icon at the other end of the link, i. e.-a data input. A link to an attachment point on the right side of an icon means that the operation or location represented by the icon outputs data to the icon at the other end of the link, i. e.-a data output.

   A link attached to an attachment point on the top of an icon means that the operation or location represented by the icon is dependent upon the completion of the operation or location at the other end of the link and will execute once that other operation or location is complete. A link to an attachment point on the bottom of an icon means that the operation or location at the other end of the link is dependent on this operation or location represented by this icon.

 

   Different icons have different numbers and locations of attachment points. For example, join filter icons 254 have two attachment points on their left side and three on their right. This is both visually and functionally representative of the operation of join filters wherein-two data inputs are combined into one principle joined data output, represented by the top right hand attachment point, and two exception data outputs represented by the middle and bottom attachment points.



   To complete the action graph design process, the user selects each of the operation icons placed in the action editor workspace, in turn, for final  definition. As an operation is selected, an appropriate definition dialog box appears.



   Figure 11 shows the Join Filter dialog box for defining the
Product Material filter. This is where the user enters the representative name of the filter (i. e.-"Product Materials"), if they have not already done so, and defines the Selection Criteria (i. e.-the'Key') for each database upon which the selection is to be based. The fields in each of the Production Forecast and
Bill of Materials data tables are displayed to the user when defining the
Selection Criteria to make the appropriate selection of the respective Key fields.



   Figure 12 shows the Aggregation Filter dialog box for defining the Materials Calculate filter. Again, the user enters the representative name of the filter (i. e.-"Materials Calculate"), if they have not previously done so, and defines the aggregation criteria. Figure 13 shows an example of a
Selection Filter dialog box.



   In the present invention, the definition of an operation is an intelligent process in that the definition dialog box only offers the appropriate choices to the user. For example, as mentioned above, the Join Filter dialog box lists all of the fields in each data table linked to the filter when defining selection criteria and/or join criteria. For a Selection Filter, the user is presented with a list of available fields and a selection of boolean operators.



  Similarly, the Aggregation Filter dialog box prompts the user for the aggregation criteria and the fields to which the formula is to be applied. A variety of predefined (either system supplied or user created) aggregation functions, such as minimums, maximums, standard deviation, etc. are available to the user for incorporation in the aggregation formula. The user can also select from a variety of predefined aggregation formulas to which they need only assign the appropriate fields.



   To define where the results of the action graph are to be stored, in the example of Figures 10 and 11, the user selects the icon representing the
Raw Materials Ordering data table and a data table dialog box (not shown) is  presented to the user. As with the other dialog boxes in Action Editor 244, the user merely selects the appropriate choices from those presented within the dialog box. Each field which has been aggregated in the aggregation filter is presented to the user who matches it to one or more fields from the"Raw
Materials Ordering"data table which are also presented to the user.



   When an action graph is completed, the user may either save the action graph for future and/or repeated use or may execute the graph to perform the action immediately. If the graph is to be executed immediately, one of Run Single icon 268, Run icon 269 or Controlled Start icon 271 is selected by the user. Prior to saving the action graph or executing it, the action editor parses the graph to ensure that all necessary elements have been correctly defined. In the event that the parse fails, the user is prompted to correct the error by, for example, completing the definition of the Key for the join filter which was omitted. To aid in debugging, or to perform what-if analysis, the action graph may be executed one icon at a time by selecting Run Single icon 268. In this manner, a user may examine the results of each step in the action graph, prior to the next step being executed.

   If single-stepping through the action graph is not required, the user may select Run icon 269 to have the complete action graph executed.



   The present invention also allow a user to define and save a subgraph which may then be incorporated into other graphs as desired. In this manner, action graphs for common operations need only be created once. A sub-graph is defined in a manner similar to action graphs except inputs to and outputs from the sub-graph are defined via Input icon 252 nd Output icon 253 respectively. To incorporate a defined sub-graph into action graph, Data
Flow icon 255 is selected and placed in the action graph. Definition of the
Data Flow icons then proceeds via one or more appropriate dialog boxes.



   It is contemplated that one of the particular advantages of the present invention is the ease with which actions can be amended, either due to changes in the user's requirements or due to changes in one of the data  repositories. For example, if a database is moved from a Sybase database engine to an Ingres database engine, the user need only edit the appropriate connection and data table information as described above. Once the connection and data table information has been amended, every graph which included the former cl section or data table will operate correctly, without the need for explicit individual amendment.



   Another contemplated advantage of the present invention is that, to a large extent, action graphs are self documenting. Once defined and labelled appropriately, it is contemplated that a user will be able to view an action graph and quickly and intuitively be able to comprehend the data operations which occur upon execution of the action graph.



   Another contemplated advantage of the present invention is that, if the user's requirements have changed, the user may simply edit the action graph accordingly. For example, assume that the user's requirements for raw materials ordering information have changed so that the user only wishes to be provided with a list of raw materials which must be ordered two months prior to delivery (i. e.-a 2 month delivery lag time). In this scenario, assuming that the Bill of Materials data table includes data representing the delivery time requirements for the raw materials, the user merely edits the action graph of
Figure 10 to add a selection filter entitled,"Two-Month Delivery Lag", after the join and the aggregate filters, as shown in Figure 15. The selection filter is defined by the user to select joined and aggregated data which has a delivery lag time greater or equal to 2 months.



   If the Bill of Materials data table does not include raw materials delivery time requirements, the user could select (or define) the appropriate data table representing delivery time requirements and add it to the graph of figure 14 via a second join filter located prior to the Two Month Delivery Lag selection filter. In either event, the user may save the edited graph either in place of the previous graph or in addition to the previous graph. In the latter  case, the user may in future select between the two graphs to suit his actual needs.



   Figure 15 shows the action graph of Figure which has been modified to demonstrate a link dependency. In the example, a Shell Script icon has been linked to the top attachment point of the Materials Calculate aggregation filter. When the action graph is modified in this manner, the aggregation filter will not execute until completion of the shell script represented by the Shell Script icon.



   Another contemplated advantage of the present invention is the provision of data visualization tools to the user. Specifically, the user can select a data table or filter from an action graph and then select Data Editor icon 246 from Toolbar 246 to call up one or more data visualization tools. For example, a user may select the Bill of Materials data table and then select Data
Editor icon 246 to open a data editor to view and/or modify data in the table.



  Also, Data Editor icon 246 may be included in an action graph as an operation and/or location as shown in Figure 16. In this manner, the results of an operation may be displayed to, and/or edited by the user, prior to the next operation, if any, proceeding. Essentially, system 20 treats a visualization tool as a destination data table or application. Each tool has an appropriate predefined equivalent of a data connection and data table and/or an Application
Socket 60. In this manner, the visualization tools employed are not particularly limited and may comprise the a spreadsheet-like data editor, a graphic display (charts and graphs) system or any other suitable or special purpose tool the user requires.



   The present invention provides a novel method and system whereby a user can define actions graphs which are executed by the system to access, filter, modify and store data from a variety of locations and in a variety of formats. Complex operations including semantic translations may be defined and maintained by the user with relative ease. Further, a change to a component of an action graph only necessitates the amendment/reconfiguration  of that component. Once the amendment/reconfiguration is performed for the component, all action graphs which employ that component will function properly, without any need for the amendment of individual action graphs.  



   APPENDIX 1 //================================================================ //class: Storage //Copyright (c) 1994-1995, Numetrix Ltd. All rights reserved.



  // fifndef StorageH   #define Stsx. age H    /*=cdoc[Storage]=================================== beginclassdesc
The Storage class is an abstract class which provides a common API for data repositories. In conjunction with the Tuple class, it provides a single store view of a data repository for the application programmer.
 endslassdesc endclassdesc
Sincludemboolean. h >    #include < rw/catring.   h >    #include < rw/tpordvec.   h >    #include < rw/tvordvec.   h >    #include < dRef.   h >    #include < tupleRef.   h >  class OrderBy; class TupleCon; class TupleSet; class TableMaster; class
StorageObj; class ObjectDef; enum eCommitType {
 eIMPLICIT,
 eEXPLICIT } ;

     #ifdef   NMX TESTPRIVATE   #define   private public define protected public   #endif    class Storage {friend class Tuple; friend class Guard;//for initialization friend class Transaction; public:
 enum eDefStat
 {
 eEXISTS,
 eDNE,
 eDIFFERENT
 //Constructor.



   //
 Storage:: Storage ();
 //
 //These are required by the RW containers.



   //
 boolean operator== (const Storage  & storage);
 boolean operator <  (const Storage  & storage);
 virtual boolean initialize (const RWCString  & user,
 const RWCString  & passwd,  
 const RWCString  & connect) = 0;
 virtual-Storage ();
 //
 //Commitment control.



   //
 virtual boolean beginTransaction ();
 virtual boolean abortTransaction ();
 virtual boolean commitTransaction ();
 virtual boolean setCommitControl (eCommitType ct);//N/A
 //
 //Table Management.



   //
 virtual boolean createTable (tupleRef < TableMaster >   & tmIn);
 virtual boolean createIndex (tupleRef < TableMaster >   & tmIn, int keyNo);
 virtual boolean deleteTable (tupleRef < TableMaster >   & tmIn) ;
 virtual boolean buildTable (tupleRef < TableMaster >   & tmln);
 virtual eDefStat existsTable (tupleRef < TableMaster >   & tmIn);
 virtual tupleRef < TableMaster >  createDef (const RWCString  & tableName);
 virtual boolean clearTable (tupleRef < TableMaster >   & /*tmIn*/)
 {return FALSE;}
 boolean deleteAllRows (const RWCString  & tableName);
 //
 //ObjectDef Management.



   //
 void addDefinition (dRef  & defIn);
 dRef removeDefinition (const RWCString  & defName) ;
 dRef findDefinition (const RWCString  & defName,
 const char *typeId);
 virtual dRef buildDef (const RWCString  & tablename);
 virtual dRef buildDef (tupleRef < TableMaster >  tmIn);
 //
 //Tuple Management.



   //
 boolean select (TupleCon  & tcOut);
 virtual boolean select (dRef  & objDef, const RWCString *pred,
 const OrderBy *order, TupleCon  & tc);
 virtual boolean deleteSelected (dRef  & objDef,
 const RWCString *predicate);
 virtual tRef fetch (tRef  & tupleIn);
 virtual boolean qryListTables (RWTValOrderedVector < RWCString >   & tableList);
 boolean setPersistent (TupleCon  & tcIn);
 boolean setPersistent (tRef  & trIn);
 void setDefinition (const tupleRef < StorageObj >   & defIn)
 {mDefinition = defIn;}
 tupleRef < StorageObj >  qryDefinition () const {return mDefinition ;}
 //
 //Garbage collection.



   //
 boolean collectGarbage ();
 boolean qryActiveTx () {return mActiveTx;}
 virtual void setDataFormats () const = 0;
 virtual boolean setPhysicalOrder (dRef  & /*objDef*/, OrderBy * /*order*/)
 {return FALSE;} protected:
 void setDataTypeValue (TransTuple *tupleIn,
 int colIndex,  
 void *pData,
 boolean nullFlag);
 boolean setDataTypeValue (TransTuple *tupleln,
 const int colIndex,
 const RWCString sData,
 const RWCString sFormat);
 boolean updateStorage ();
 boolean updatestorageState ();
 void changeKeyValues (TransTuple * tuple,
 RWTValOrderedVector < DataType* >  & keyValues,
 RWTValOrderedVector < int >   & keyIndexes) ;
 virtual boolean updateKey (Tuple *tuple,
 RWTValOrderedVector < DataType* >  & keyValues,
 RWTValOrderedVector < int >  & keyIndexes);
 void addTuple (Tuple *tuple);
 Tuple* findTuple (Tuple *tuple);
 Tuple* removeTuple (Tuple *tuple);

  
 static Tuple* cloneTuple (Tuple *tuple, boolean addToCache = TRUE);
 //Returns the character used as a separator in table names.
 virtual const char* qrySeparatorChar () const {return"." ;}
 //Data Management Methods
 //
 virtual boolean executeSelect (Tuple */*tupleIn*/,
 const RWCString */*pred*/,
 const OrderBy */*order*/);
 virtual boolean executeDeleteSelected (dRef  & objDef,
 const RWCString *predicate,
 TupleSet *tempTS) = 0;
 virtual boolean getNextRow () {return FALSE;}
 virtual Tuple* loadTheRow (Tuple */*tupleIn*/) {return NULL;}
 virtual Tuple* loadNextRow (Tuple */*tupleln*/) (return NULL;}
 virtual boolean erase (Tuple *tupleIn) = 0;
 virtual boolean exists (Tuple *tupleIn) = 0;
 virtual boolean update (Tuple *tupleIn) = 0;
 virtual boolean insert (Tuple *tupleln) = 0;
 virtual boolean executeBegin () = 0;
 virtual boolean executeCommit () = 0;

  
 virtual boolean executeAbort () = 0; private:
 //
 //Transaction lll
 //
 Transaction *mTransaction;
 boolean mSIM;
 boolean mActiveTx;
 RWTPtrOrderedVector < dRef > mObjDefinitions;
 tupleRef < Storageobj > mDefinition; } ;   #endif//Storage H     
 APPENDIX 2 //File: Filter. h// ////This file contains the declaration of the filter base class.



  // //Copyright (c) 1994, Numetrix Ltd. All rights reserved.



  // //¯¯¯¯¯¯¯¯¯¯¯¯¯   #ifndef   FILTER H   #define   FILTER H   #include    < Ee. Jression. h > /* for using Expression class */   #include    < FuncFilter. h > /* for using user-defined Fitler functions */   #include    < rw/cstring. h >    #include    < rw/tpordvec. h >    #include  < lwf.   h >    #include zmwf.   h >    #include    < hwf. h >    #include    < TupleSet. h >    #include    < array. h >    #include    < rw/bitvec. h >    #include CNamed Expression.   h > /* use for Variablefield resolve*/ //Associates a TupleSet with a character string name that defines its role.



  /*=cdoc[TupleSetRole]================================
 beginsynopsis
 struct TupleSetRole
 endsynopsis
 beginclassdesc
 This class is a record containing the data required to bind
 a specific tupleset with given logical name
 endclaesdescj
 endclassdesc struct TupleSetRole {
 TupleSetRole (): TSRdata (NULL), TupleCurser (NULL), ActiveFields (255,
FALSE) {}
 RWCString TSRname;//TupleSet Role
 TupleSet **TSRdata;//Data
 const tRef** TupleCurser;
 RWBitVec ActiveFields ;
 //These are necessary for the RWPtrOrederVector
 //equality test operator
 int operator== (const TupleSetRole  &  tupleSetRole) const{
 if (TSRname == tupleSetRole. TSRname)
 return 1;
 else
 return 0;
 } } ; //Associates a role with a pointer to a list of LWFs.



  /*=cdoc ExpressionListRole ==============================
 beginsynopsis
 struct ExpressionListRole
 endsynopsis
 beginclassdesc
 This class is a record containing the data required to add
 a field on to the existing output tuple. This structure consists
 of two parts a field name and a List of associated Light Weight
Functor   

 endclassdesc
 =======================================*/     struct ExpressionListRole {
 ExpressionListRole (): LWFList (NULL) {}
 RWCString ELname;
 NMXTPtrArray < LWF >  * LWFList;
 //These are necessary for the RWPtrOrederVector
 //equality test operator
 int operator== (const ExpressionListRole  &  expressionListRole) const {
 if (ELname == expressionListRole.

   ELname)
 return 1;
 else
 return 0;
 } } ; enum FilterType {JOIN, AGGR, SELECTION, UNION}; (beginsynopsis
 [beginsynopsis
 Filter (Storage * inStorage)
 endsynopsis
 beginclassdesc
 This class will be interface between the Filters and the Flow controller.



   It is an abstract base class for filters such join and aggregation.



   It facilitates the management of all logical names and expression used
 within the filters
 endclaSBdesc)
 ---------class Filter {friend class CalculateMvc; public:
 F i 1 t e r (S t o r a g e * i n S t o r a g e) : mStorage (inStorage), MWF¯Flag(FALSE), HWF¯Flag(FALSE), result (NULL), HWF¯LookAhead¯Flag (FALSE), NamedExprFlag (FALSE)    {
 -   
 reinit () ;
 }
 virtual-Filter ();
 LWF * FilterParse (RWCString  &  expr) ;
 //Set a TupleSetRole to a TupleSet
 boolean setTupleSet (RWCString  &  tsrolename, TupleSet *newdata);
 //Get the TupleSet roles
 boolean reinit ();//Re-initialize filter
 RWTPtrOrderedVector < RWCString > getTupleSetRoles () ;
 //Get the LWF for the named expression
 LWF* resolveAtom (RWCString  &  atomname);
 RWTPtrOrderedVector < RWCString >  getEx. essionRoles ();
 virtual TupleSet * apply (const RWCString  & ) = O;

  //evaluate the filter
 virtual TupleSet * value () =0;
 Storage * qryStorage() { return mStorage;}  
 void printTuple (tRef *tupleIn);
 //TEMPORARY UNTIL PARSER CAN PASS BACK OBJECTS
 static RWTValOrderedVector < LWF* >  FilterStack;
 static Filter * currFilter;//
 static boolean initExprFlag;
 //Access method for TupleSetRolesList
 TupleSetRole * qryTupleSetRolesLis' RWCString  &  rolename) ;
 NMXTPtrArray < HWF >  HWFList;
 NMXTPtrArray < MWF >  MWFList;
 NMXTPtrArray < LWF >  LWFList;
 //used for MTFilters two minimizie data redundancy
 boolean MWFFlag;
 boolean HWF Flag;
 boolean NamedExprFlag;
 boolean HWF LookAhead Flag;
 //Append an expression to the Expression List
 boolean addExpr (RWCString  &  elrolename, RWCString  & expression);
 boolean removeExpr (RWCString  &  elrolename);

  
 boolean addNamedExpr (RWCString  &  elrolename, RWCString  &  expression, boolean Visable = TRUE);
 boolean removeNamedExpr (RWCString  &  fieldname) ;
 //List of named expressions
 RWTPtrOrderedVector < Named Expression >  outputExpressionList;
 boolean addDependency (const RWCString &  name);
 //used to store expresson which have not been solved
 RWTPtrOrderedVector < NamedExpression >  tempoutputExpressionList;
 RWCString currentActiveRole;
 FilterType filtertype;
 boolean resolveSymbols (RWCString  & );
 protected:
 RWTPtrOrderedVector < RWCString >  TupleSetRoleNameList;
 RWTPtrOrderedVector < RWCString >  ExprListRoleNameList;
 //Add a TupleSetRole to a TupleSet
 //Append an ExpressionList Role to an Expression
 boolean addExpressionRole (const RWCString  &  elrolename,
NMXTPtrArray  < LWF >  * expressionlwfList);

  
 boolean addTupleSet (const RWCString  &  tsrolename, TupleSet ** data , const tRef** Curser);
 TupleSet *result;//Output TupleSet
 private:
 void FilterDeleteExpr (void);
 void FilterInitExpr (void);
 //Lists of TupleSets of Data pointer pointers and their roles
 RWTValOrderedVector < RWCString >  dependentsname;
 RWTPtrOrderedVector < TupleSetRole >  TupleSetRolesList;
 static Expression * FilterExpr;
 //List of Expression List roles
 RWTPtrOrderedVector < ExpressionListRole >  ExpressionRolesList;  
 Storage * mStorage; } ; &num;endif
  

Claims

What is claimed is: 1. A method defining a series of operations to access and process data comprising the steps of : (i) selecting a data connection from a plurality of data connections presented to a user; (ii) selecting a data table from a plurality of data tables presented to said user which are related to said selected data connection; (iii) selecting at least one data processing action to operate on said selected data table from a plurality of data processing actions presented to said user; (iv) selecting a destination for the data produced by said at least one data processing action; and (v) executing each of said at least one data processing actions.

2. The method of claim 1 further comprising the step of, prior to step (i), a user defining at least one of data connection.

3. The method of claim 2 further comprising the step of, prior to step (ii), a user defining at least one data table related to one of said at least one data connections.

4. The method of claim 1 wherein step (iii) comprises selecting said predefined data processing action and relating it to said selected data table and to said selected data destination through a graphical user interface.

5. The method of claim 4 wherein different data processing actions are identified to said user by different respective representations within said graphical user interface.

6. The method of claim 1 wherein said destination for said data is another data processing action.

7. The method of claim 5 wherein said different data processing actions comprise a data selection action, a data joint action and a data aggregation action.

8. The method of claim 1 further comprising the step of storing the selections made by a user for execution at another time.

9. The method of claim 5 further comprising the step of storing the selections made by a user for execution at another time and wherein the stored representations are presented to the user in a graphical format prior to execution.

10. A method of accessing and processing data comprising the steps of : (i) defining at least one data repository; (ii) defining at least one data table for said at least one defined data repository; (iii) with a graphical user interface, selecting at least one of said data connections; (iv) with said graphical user interface, selecting said at least one data table from said selected data connection; (v) with said graphical user interface, selecting at least one of a plurality of data processing actions and relating each of said at least one data processing actions to appropriate ones of said at least one data table; (vi) with said graphical user interface, defining a destination for data produced by each of said at least one data processing actions; (vii) executing each of said at least one data processing actions. 11.

The method of claim 10 wherein said destination for data produced by said data processing action is another data processing action.

12. The method of claim 11 wherein said destination for data produced by said data processing action is a computer program 13. A system for accessing and processing data comprising: means to define at least one data connection to a data repository; means to define at least one data table for each of said at least one data connections; graphical user interface means to select at least one predefined data processing action and to relate said selected data processing action to one or more of said at least one data tables through the definition of an action graph.

14. A system for accessing and processing data comprising: data access means to access at least one data repository; data processing action means to perform at least one data processing action on data from said at least one data repository; graphical control means to allow a user to select said at least one data repository and said at least one data processing action graphically.

15. A system according to claim 14 comprising at least first and second data repositories, said first data repository having a first predefined access method and said second data depository having a different predefined access method.

16. A system according to claim 14 wherein said graphical control means allows said user to select a plurality of data processing actions and to define a data flow path from said data repository to one of said plurality of data processing actions and from said one of said plurality of data processing actions to a different one of said plurality of data processing actions.

17. A system according to claim 14 wherein said data access means comprises: means for defining connectivity information to a plurality of data repositories; means for defining data binding information for each of said plurality of data repositories; generic interface means to define an interface to said means for defining connectivity information and to said means for defining data binding information; and at least one application program socket to relate an application program to said generic interface means.

18. A method to allow a user to define graphically an action graph representing a series of operations to access and process data comprising the steps of : (i) graphically presenting to a user a plurality of predefined data connections from which the user selects at least one data connection; (ii) graphically presenting to a user a plurality of predefined data tables which are related to said at least one selected data connection from which the user selects at least one data table; (iii) graphically presenting to a user a plurality of predefined data processing actions to operate on said at least one selected data table from which the user selects at least on data processing action; (iv) graphically presenting to a user at least one destination for the data produced by said at least one data processing action from which the user selects a destination for said data;

(v) graphically connecting said at least one data table, said at least one data processing action and said at least one destination for said data to define a data flow; and (vi) graphically presenting a user with the option to execute each of said at least one data processing actions in said action graph.

19. The method of claim 18 where step (v) further includes the step of graphically selecting the precedence of said data flow in said action graph.