US20030120670A1 - Method and system for parsing navigation information - Google Patents

Method and system for parsing navigation information Download PDF

Info

Publication number
US20030120670A1
US20030120670A1 US10/320,171 US32017102A US2003120670A1 US 20030120670 A1 US20030120670 A1 US 20030120670A1 US 32017102 A US32017102 A US 32017102A US 2003120670 A1 US2003120670 A1 US 2003120670A1
Authority
US
United States
Prior art keywords
information
data
dimension
entry
routine
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Abandoned
Application number
US10/320,171
Inventor
Krishnamohan Nareddy
Radha Uppala
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Revenue Science Inc
Original Assignee
Revenue Science Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Family has litigation
First worldwide family litigation filed litigation Critical https://patents.darts-ip.com/?family=24458912&utm_source=google_patent&utm_medium=platform_link&utm_campaign=public_patent_search&patent=US20030120670(A1) "Global patent litigation dataset” by Darts-ip is licensed under a Creative Commons Attribution 4.0 International License.
Application filed by Revenue Science Inc filed Critical Revenue Science Inc
Priority to US10/320,171 priority Critical patent/US20030120670A1/en
Assigned to REVENUE SCIENCE, INC. reassignment REVENUE SCIENCE, INC. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: DIGIMINE, INC.
Publication of US20030120670A1 publication Critical patent/US20030120670A1/en
Assigned to COMERICA BANK reassignment COMERICA BANK SECURITY AGREEMENT Assignors: REVENUE SCIENCE, INC.
Assigned to AUDIENCESCIENCE, INC. FKA REVENUE SCIENCE, INC. FKA DIGIMINE, INC. FKA MINEPOINT.COM, INC. reassignment AUDIENCESCIENCE, INC. FKA REVENUE SCIENCE, INC. FKA DIGIMINE, INC. FKA MINEPOINT.COM, INC. RELEASE BY SECURED PARTY (SEE DOCUMENT FOR DETAILS). Assignors: COMERICA BANK
Abandoned legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06QINFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q30/00Commerce
    • G06Q30/02Marketing; Price estimation or determination; Fundraising
    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10TECHNICAL SUBJECTS COVERED BY FORMER USPC
    • Y10STECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10S707/00Data processing: database and file management or data structures
    • Y10S707/99931Database or file accessing
    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10TECHNICAL SUBJECTS COVERED BY FORMER USPC
    • Y10STECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10S707/00Data processing: database and file management or data structures
    • Y10S707/99931Database or file accessing
    • Y10S707/99933Query processing, i.e. searching
    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10TECHNICAL SUBJECTS COVERED BY FORMER USPC
    • Y10STECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10S707/00Data processing: database and file management or data structures
    • Y10S707/99931Database or file accessing
    • Y10S707/99933Query processing, i.e. searching
    • Y10S707/99936Pattern matching access
    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10TECHNICAL SUBJECTS COVERED BY FORMER USPC
    • Y10STECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10S707/00Data processing: database and file management or data structures
    • Y10S707/99931Database or file accessing
    • Y10S707/99937Sorting
    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10TECHNICAL SUBJECTS COVERED BY FORMER USPC
    • Y10STECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10S707/00Data processing: database and file management or data structures
    • Y10S707/99951File or database maintenance
    • Y10S707/99952Coherency, e.g. same view to multiple users
    • Y10S707/99954Version management

Definitions

  • the described technology relates to analyzing of data relating to events generated by a computer program.
  • HTTP Hypertext Transfer Protocol
  • each HTTP request message contains a universal resource identifier (a “URI”), which specifies to which network resource the request is to be applied.
  • a URI is either a Uniform Resource Locator (“URL”) or Uniform Resource Name (“URN”), or any other formatted string that identifies a network resource.
  • URLs as an example of URIs, are discussed in detail in T. Berners-Lee, et al., Uniform Resource Locators ( URI ), RFC 1738, CERN, Xerox PARC, Univ. of Minn., December 1994.
  • FIG. 1 illustrates how a browser application enables users to navigate among nodes on the web network by requesting and receiving web pages.
  • a web page is any type of document that abides by the HTML format. That is, the document includes an “ ⁇ HTML>” statement.
  • a web page is also referred to as an HTML document.
  • the HTML format is a document mark-up language, defined by the Hypertext Markup Language (“HTML”) specification. HTML defines tags for specifying how to interpret the text and images stored in an HTML document. For example, there are HTML tags for defining paragraph formats and for emboldening and underlining text.
  • the HTML format defines tags for adding images to documents and for formatting and aligning text with respect to images. HTML tags appear between angle brackets, for example, ⁇ HTML>. Further details of HTML are discussed in T. Berners-Lee and D. Connolly, Hypertext Markup Language— 2.0, RFC 1866, MIT/W3C, November 1995.
  • a web browser application 101 is shown executing on a client computer 102 , which communicates with a server computer 103 by sending and receiving HTTP packets (messages). HTTP messages may also be generated by other types of computer programs, such as spiders and crawlers.
  • the web browser “navigates” to new locations on the network to browse (display) what is available at these locations. In particular, when the web browser “navigates” to a new location, it requests a new document from the new location (e.g., the server computer) by sending an HTTP-request message 104 using any well-known underlying communications wire protocol.
  • the HTTP-request message follows the specific layout discussed above, which includes a header 105 and a URI field 106 , which specifies the network location to which to apply the request.
  • URI URI
  • the server computer specified by URI receives the HTTP-request message, it interprets the message packet and sends a return message packet to the source location that originated the message in the form of an HTTP-response message 107 . It also stores a copy of the request and basic information about the requesting computer in a log file.
  • the HTTP-response message contains the requested HTML document 109 .
  • HTTP can also be used to transfer other media types, such as the Extensible Markup Language (“XML”) and graphics interchange format (“GIF”) formats.
  • XML Extensible Markup Language
  • GIF graphics interchange format
  • the World Wide Web is especially conducive to conducting electronic commerce (“e-commerce”).
  • E-commerce generally refers to commercial transactions that are at least partially conducted using the World Wide Web.
  • numerous web sites are available through which a user using a web browser can purchase items, such as books, groceries, and software.
  • a user of these web sites can browse through an electronic catalog of available items to select the items to be purchased.
  • To purchase the items a user typically adds the items to an electronic shopping cart and then electronically pays for the items that are in the shopping cart.
  • the purchased items can then be delivered to the user via conventional distribution channels (e.g., an overnight courier) or via electronic delivery when, for example, software is being purchased.
  • Many web sites are also informational in nature, rather than commercial in nature. For example, many standards organizations and governmental organizations have web sites with a primary purpose of distributing information. Also, some web sites (e.g., a search engine) provide information and derive revenue from advertisements that are displayed.
  • Web sites typically collect extensive information on how its users use the site's web pages. This information may include a complete history of each HTTP request received by and each HTTP response sent by the web site. The web site may store this information in a navigation file, also referred to as a log file or click stream file. By analyzing this navigation information, a web site operator may be able to identify trends in the access of the web pages and modify the web site to make it easier to use and more useful.
  • FIG. 1 illustrates how a browser application enables users to navigate among nodes on the web network by requesting and receiving web pages.
  • FIG. 2A is a block diagram illustrating components of the data warehouse system in one embodiment.
  • FIG. 2B is a block diagram illustrating details of the components of the data warehouse system in one embodiment.
  • FIG. 3 is a block diagram illustrating the sub-components of the data processor component in one embodiment.
  • FIG. 4 is a block diagram illustrating some of the tables of the local data warehouse and the main data warehouse in one embodiment.
  • FIG. 5 is a flow diagram illustrating the parse log data routine that implements the parser in one embodiment.
  • FIG. 6 is a flow diagram of the filter log entry routine in one embodiment.
  • FIG. 7 is a flow diagram illustrating the normalize log entry routine.
  • FIG. 8 is a flow diagram of the generate dimensions routine in one embodiment.
  • FIG. 9 is a flow diagram of the identify logical site routine in one embodiment.
  • FIG. 10 is a flow diagram of the identify user routine in one embodiment.
  • FIG. 11 is a flow diagram of the identify page type routine in one embodiment.
  • FIG. 12 is a flow diagram illustrating the identify events routine in one embodiment.
  • FIG. 13 is a flow diagram illustrating the identify sessions routine in one embodiment.
  • FIG. 14 is a flow diagram of the generate aggregate statistics routine in one embodiment.
  • FIG. 15 is a flow diagram of the import log data routine implementing the importer in one embodiment.
  • FIG. 16 is a flow diagram of the load dimension table routine and one embodiment.
  • FIG. 17 is a flow diagram of the load fact table routine in one embodiment.
  • FIG. 18 is a flow diagram illustrating the identify user aliases routine in one embodiment.
  • a method and system for providing customers with access to and analysis of event data (e.g., navigation data collected at customer web sites) is provided.
  • the event data may be stored in log files and supplemented with data from other sources, such as product databases and customer invoices.
  • a data warehouse system collects customer data from the customer web sites and stores the data at a data warehouse server.
  • the customer data may include application event data (e.g., click stream log files), user attribute data of users of the customer web site (e.g., name, age, and gender), product data (e.g., catalog of products offered for sale by the customer), shopping cart data (i.e., identification of the products currently in a user's shopping cart), and so on.
  • the data warehouse server interacts with the customer servers to collect to the customer data on a periodic basis.
  • the data warehouse server may provide instructions to the customer servers identifying the customer data that is to be uploaded to the data warehouse server. These instructions may include the names of the files that contains the customer data and the name of the web servers on which the files reside. These instructions may also indicate the time the day when the customer data is to be uploaded to the data warehouse server.
  • the data warehouse server receives customer data, it converts the customer data into a format that is more conducive to processing by decision support system applications by which customers can analyze their data. For example, the data warehouse server may analyze low-level navigation events (e.g., each HTTP request that is received by the customer web site) to identify high-level events (e.g., a user session).
  • the data warehouse server then stores the converted data into a data warehouse.
  • the data warehouse server functions as an application service provider that provides various decision support system applications to the customers.
  • the data warehouse server provides decision support system applications to analyze and graphically display the results of the analysis for a customer.
  • the decision support system applications may be accessed through a web browser.
  • the customer servers are connected to the data warehouse server via the Internet and the data warehouse server provides data warehousing services to multiple customers.
  • the data warehouse system may provide a data processor component that converts the log files into a format that is more conducive to processing by the decision support system applications.
  • the converted data is stored in a data warehouse that includes fact and dimension tables.
  • Each fact table contains entries corresponding to a type of fact derived from the log files.
  • a web page access fact table may contain an entry for each web page access identified in the log files.
  • Each entry may reference attributes of the web page access, such as the identity of the web page and identity of the accessing user.
  • the values for each attribute are stored in a dimension table for that attribute.
  • a user dimension table may include an entry for each user and the entries of the web access fact table may include a user field that contains an index (or some other reference) to the entry of the user dimension table for the accessing user.
  • the user dimension table may contain the names of the users and other user-specific information.
  • the user dimension table may itself also be a fact table that includes references to dimension tables for the attributes of users.
  • the data warehouse may also include fact tables and dimension tables that represent high-level facts and attributes derived from the low-level facts and attributes of the log files. For example, high-level facts and attributes may not be derivable from only the data in a single log entry.
  • the higher level category (e.g., shoes or shirts) of a web page may be identified using a mapping of web page URIs to categories. These categories may be stored in a category dimension table. Also, certain facts, such as the collection of log entries that comprise a single user web access session or visit, may only be derivable by analyzing a series of log entries.
  • the data processor component may have a parser component and a loader component.
  • the parser of the data processor parses and analyzes a log file and stores the resulting data in a local data warehouse that contains information for only that log file.
  • the local data warehouse may be similar in structure (e.g., similar fact and dimension is tables) to the main data warehouse used by decision support system applications.
  • the local data warehouse may be adapted to allow efficient processing by the parser.
  • the local data warehouse may be stored in primary storage (e.g., main memory) for speed of access, rather than in secondary storage (e.g., disks).
  • the parser may use parser configuration data that defines, on a customer-by-customer basis, the high-level data to be derived from the log entries.
  • the parser configuration data may specify the mapping of URIs to web page categories.
  • the loader of the data processor transfers the data from the local data warehouse to the main data warehouse.
  • the loader may create separate partitions for the main data warehouse. These separate partitions may hold the customer data for a certain time period (e.g., a month's worth of data).
  • the loader adds entries to the main fact tables (i.e., fact tables of the main data warehouse) for each fact in a local fact table (i.e., fact table of the local data warehouse).
  • the loader also adds new entries to the main dimension tables to represent attribute values of the local dimension tables that are not already in the main dimension tables.
  • the loader also maps the local indices (or other references) of the local dimension tables to the main indices used by the main dimension tables.
  • FIG. 2A is a block diagram illustrating components of the data warehouse system in one embodiment.
  • the data warehouse system includes customer components that execute on the customer servers and data warehouse components that execute on the data warehouse server.
  • the customer servers 210 and the data warehouse server 260 are interconnected via the Internet 250 .
  • Customer components executing on a customer server includes a data collection component 220 and a data viewer 230 .
  • the data viewer may reside on a client computer of the customer, rather than a server.
  • the data collection component collects the customer data from the storage devices 240 of the customer servers.
  • the data viewer provides access for viewing of data generated by the decision support system applications of the data warehouse server.
  • the data viewer may be a web browsed
  • the data warehouse server includes a data receiver component 270 , the data processor component 280 , the data warehouse 290 , and decision support system applications 291 .
  • the data receiver component receives customer data sent by the data collection components executing at the various customer web sites.
  • the data processor component processes the customer data and stores it in the data warehouse.
  • the decision support system application provides the customer with tools for analyzing and reviewing the customer data that is stored in the main data warehouse. Analysis performed on and reports generated from are described in U.S. patent application Ser. No. ______ (Attorney Ref. No. 34821-8010US), entitled “Identifying and Reporting on Combinations of Events in Usage Data,” and U.S. patent application Ser. No.
  • each customer has its own set of dimension and fact tables so that the information of multiple customers are not intermingled.
  • FIG. 2B is a block diagram illustrating details of the components of the data warehouse system in one embodiment.
  • the data collection component 220 includes a monitor sub-component 221 and a pitcher sub-component 222 .
  • the data collection component is described in more detail in U.S. patent application Ser. No. ______ (Attorney Ref. No. 34821-800/US), entitled “Method and System for Monitoring Resource via the Web,” which is being filed concurrently and which is hereby incorporated by reference.
  • the pitcher is responsible for retrieving instructions from the data warehouse server, collecting the customer data in accordance with the retrieved instructions, and uploading the customer data to the data warehouse server.
  • the monitor is responsible for monitoring the operation of the pitcher and detecting when the pitcher may have problems in collecting and uploading the customer data.
  • the monitor detects that a problem may occur, it notifies the data warehouse server so that corrective action may be taken in advance of the collecting and uploading of the customer data.
  • the pitcher may use certain log on information (e.g., user ID and password) to access a customer web server that contains customer data to be uploaded.
  • the monitor may use that log on information to verify that the log on information will permit access to the customer data. Access may be denied if, for example, a customer administrator inadvertentry deleted from the customer web server the user ID used by the pitcher.
  • the monitor provides advance notification of a problem, the problem might be corrected before the pitcher attempts to access the customer data.
  • the monitor also periodically checks the pitcher to ensure that the pitcher is executing and, if executing, executing correctly.
  • the data receiver component of the data warehouse server includes a status receiver sub-component 271 , a catcher sub-component 272 , an FTP server 273 , a status database 274 , and a collected data database 275 .
  • the status receiver receives status reports from the customer servers and stores the status information in the status database.
  • the catcher receives and processes the customer data that is uploaded from the customer web sites and stores the data in the collected data database.
  • the data processor component includes a parser sub-component 281 and a loader sub-component 282 . The parser analyzes the low-level events of the customer data and identifies high-level events and converts the customer data into a format that facilitates processing by the decision support system applications.
  • the loader is responsible for storing the identified high-level events in the data warehouse 290 .
  • a customer may decide not to have the data collection component executing on its computer systems.
  • the customer server may include an FTP client 245 that is responsible for periodically transferring the customer data to the FTP server 273 of the data warehouse server.
  • the data receiver may process this customer data at the data warehouse server in the same way as the pitcher processes the data at the customer servers. The processed data is then stored in the collected data database.
  • FIG. 3 is a block diagram illustrating the sub-components of the data processor component in one embodiment.
  • the data processor component 300 includes a parser 310 , data storage area 370 , and a loader 330 .
  • the data processor component inputs parser configuration data 340 and a log file 350 and updates the main data warehouse 360 .
  • the parser configuration data may include a mapping of actual web sites to logical sites and a mapping of a combination of Uniform Resource Identifiers (“URls”) and query strings of the log entries to page definitions (e.g., categories) and event definitions.
  • URLs Uniform Resource Identifiers
  • the parser processes the entries of the log file to generate facts and dimensions to eventually be stored in the main data warehouse.
  • the parser identifies events in accordance with the parser configuration data.
  • the parser includes a filter log entry component 311 , a normalize log entry component 312 , a generate dimensions component 313 , an identify sessions component 314 , and a generate aggregate statistics component 315 .
  • the filter log entry component identifies which log entries should not be included in the main data warehouse. For example, a log entry that has an invalid format should not be included.
  • the normalize log entry component normalizes the data in a log entry. For example, the component may convert all times to Greenwich Mean Time (“GMT”).
  • GTT Greenwich Mean Time
  • the generate dimensions component identifies the various dimensions related to a log entry. For example, a dimension may be the Uniform Resource Identifier of the entry or the logical site identifier.
  • the identify sessions component processes the parsed log file data stored in the local data warehouse to identify user sessions.
  • a user session generally refers to the concept of a series of web page accesses that may be related in some way, such as by temporal proximity.
  • the generate aggregate statistics component aggregates data for the log file being processed as each log entry is processed or after the log file is parsed.
  • the data storage area 320 includes a local data warehouse 321 .
  • the local data warehouse is stored non-persistently (or temporarily) in main memory of the computer system.
  • the local data warehouse may contain fact tables and dimension tables that correspond generally to the tables of the main data warehouse 360 .
  • the loader retrieves the information from the local data warehouse and stores the information in the main data warehouse.
  • the loader includes a create partitions component 331 , a load dimension table component 332 , and a load fact table component 333 .
  • the create partitions components creates new partitions for the main data warehouse.
  • a partition may correspond to a collection of information within a certain time range.
  • the main data warehouse may have a partition for each month, which contains all the data for that month.
  • the load dimension table component and the load fact table component are responsible for loading the main data warehouse with the dimensions and facts that are stored in the local data warehouse.
  • the log file is a web server log file of a customer.
  • the log file may be in the “Extended Log File Foimat” as described in the document “http://www.w3.org/TR/WD-logfile-960323” provided by the World Wide Web Consortium, which is hereby incorporated by reference.
  • the log file contains lines that are either directives or entries.
  • An entry corresponds to a single HTTP transaction (e.g., HTTP request and an HTTP response) and consists of a sequence of fields (e.g., integer, fixed, URI, date, time, and string).
  • the meaning of the fields in an entry is specified by a field directive specified in the log file.
  • a field directive may specify that a log entry contains the fields date, time, client IP address, server IP address, and success code. Each entry in the log file would contain these five fields.
  • the parser configuration data defines logical sites, page definitions, and event definitions.
  • a logical site is a collection of one or more IP addresses and ports that should be treated as a single web site. For example, a web site may actually have five web servers with different IP addresses that handle HTTP requests for the same domain. These five IP addresses may be mapped to the same logical site to be treated as a single web site.
  • FIG. 4 is a block diagram illustrating some of the tables of the local data warehouse and the main data warehouse in one embodiment.
  • These data warehouses are databases that include fact tables and dimension tables.
  • a fact table contains an entry for each instance of fact (e.g., web page access).
  • a dimension table contains an entry for each possible attribute value of an attribute (e.g., user).
  • the entries of a fact table contain dimension fields that refer to the entries into the dimension tables for their attribute values.
  • a table may be both a fact table and a dimension table.
  • a user dimension table with an entry for each unique user may also be a fact table that refers to attributes of the users that are stored in other dimension tables.
  • the data warehouses contain a log entry table 401 , a user table 402 , a logical site table 403 , a URI table 404 , a referrer URI table 405 , a page type table 406 , event type tables 407 , a query string table 408 , and a referrer query string table 409 .
  • the log entry table is a fact table that contains an entry for each log entry that is not filtered out by the parser.
  • the other tables are dimension tables for the log entry table.
  • the user table contains an entry for each unique user identified by the parser.
  • the logical site table contains an entry for each logical site as defined in the parser configuration data.
  • the URI table contains an entry for each unique URI of an entry in the log entry table.
  • the referred URI table contains an entry for each referrer URI of the log entry table.
  • the page type table contains an entry for each page type identified by the parser as defined in the parser configuration data.
  • the data warehouse contains an event table for each type of event defined in the parser configuration data. Each event table contains an entry for each event value of that event type specified in an entry of the log entry table.
  • the query string table contains an entry for each unique query string identified in an entry of the log entry table.
  • the referrer query string contains an entry for each unique referrer query string identified in an entry of the log entry table.
  • Table 1 is an example portion of a log file.
  • the “#fields” directive specifies the meaning of the fields in the log entries. Each field in a log entry is separated by a space and an empty field is represented by a hyphen.
  • the #fields directive in this example indicates that each entry includes the date and time when the transaction was completed (i.e., “date” and “time”), the client IP address (i.e., “c-ip”), and so on.
  • the first log entry has a data and time of “2000-06-01 07:00:04” and a client IP address of “165.21.83.61.”
  • Table 2 is an example portion of parser configuration data.
  • ⁇ NewsItemID># ⁇ Uri ⁇ indicates that a page type of “news item” is specified for logical site 1 by a URI pattern of “/homepage_include/industrynews_detail.asp.”
  • the definition also indicates that the event value is “ ⁇ NewsItemID># ⁇ Uri ⁇ ,” where the URI of the log entry is substituted for “ ⁇ Uri ⁇ and the value of NewsItemID in the query string is substituted for “ ⁇ NewsItemID>.”
  • the event type definitions map a site identifier, URI pattern, and query string pattern to an event type and value.
  • the definitions also specify the name of the event type and the name of the dimension table for that event type.
  • FIGS. 5 - 14 are flow diagrams of components of the parser in one embodiment.
  • FIG. 5 is a flow diagram illustrating the parse log data routine that implements the main routine of parser in one embodiment.
  • the routine processes each entry in the log file based on the parser configuration data.
  • the routine filters out certain log entries, normalizes the attribute values of the log entries, and generates entries in the dimension tables for the attributes of the log entries.
  • the parser identifies user sessions and generates various statistics.
  • the routine loops selecting and processing each log entry.
  • the routine selects the next log entry of the log file starting with the first log entry.
  • the routine may also pre-process the header information of the log file to identify the fields of the log entries.
  • decision block 1502 if all the log entries have already been selected, then the routine continues at block 509 , else the routine continues at block 503 .
  • the routine extracts the values for the fields of the selected log entry.
  • the routine invokes the filter log entry routine, which returns an indication as to whether the selected log entry should be filtered out.
  • decision block 505 if the filter log entry routine indicates that the selected log entry should be filtered out, then the routine skips to block 508 , else the routine continues at block 506 .
  • the routine invokes the normalize log entry routine to normalize the values of the fields of the selected log entry.
  • the routine invokes the generate dimensions routine to update the dimension tables based on the selected log entry and to add an entry into the log entry fact table.
  • the routine updates the statistics for the log file. For example, the routine may track the number of log entries that have been filtered out. The routine then loops to block 501 to select the next log entry.
  • the routine outputs the log file statistics.
  • the routine invokes the identify sessions routine that scans the log entry table to identify the user sessions and updates a session dimension table.
  • the routine invokes the generate aggregate statistics routine to generate various statistics and then completes.
  • FIG. 6 is a flow diagram of the filter log entry routine in one embodiment.
  • the filter log entry routine is passed a log entry and determines whether the log entry should be filtered out.
  • the routine determines whether the filter out conditions have been satisfied.
  • the routine determines whether the log entry has a field count problem. A field count problem arises when the number of fields in the log entry does not correspond to the number of expected fields for that log entry. The number and types of fields may be defined in a “fields” directive line of the log file.
  • decision block 602 the routine determines whether the log entry is outside of a specified time range. The routine compares the time field of the log entry to the time range.
  • the time range may be specified so that only those log entries within that time range are processed.
  • the routine determines whether the IP address of the log entry should be ignored. For example, a log entry may be ignored if the entry originated from a server whose function is to ping the customer's web server at periodic intervals.
  • the routine determines whether the log entry corresponds to a comment (e.g., a “g#remarks” directive).
  • the routine determines whether the success code associated with the log entry indicates that log entry should be ignored. For example, if the success code indicates a failure, then the log entry may be ignored.
  • the routine determines whether the log entry is requesting a resource whose extension indicates that the log entry should be ignored. For example, the routine may ignore log entries requesting graphic files, such as those in the “.gif” format.
  • the routine determines whether the values within the fields of the log entry are corrupt. For example, a value in the date field that indicates a date of February 30th is corrupt.
  • the various filtering, conditions may be specified in a configuration file. For example, the time range, IP addresses, and so on may be specified in the configuration file. These configuration files may be specified on a customer-by-customer basis.
  • FIG. 7 is a flow diagram illustrating the normalize log entry routine.
  • the routine normalizes the values of the fields in the passed log entry.
  • the routine converts the time of the log entry into a standard time such as Greenwich Mean Time.
  • the routine collects the time based on the variation between the times of the customer web servers. For example, the time of one web server may be five minutes ahead of the time of another web server. This correction may be based on current time information collected from computer systems that generated the events and then correlated to base current time information.
  • the routine normalizes, the values of the fields of the log entry. This normalization may include processing search strings to place them in a canonical form.
  • a search string of “back pack” may have a canonical form of “backpack.”
  • Other normalization of search strings may include stemming of words (e.g., changing “clothes” and “clothing” to “cloth”), synonym matching, and first and last word grouping. The first word grouping for the search strings of “winter clothing” and “winter shoes” results in the string of “winter.”
  • FIG. 8 is a flow diagram of the generate dimensions routine in one embodiment.
  • This routine identifies a value for each dimension associated with the passed log entry and ensures that the dimension tables contains entries corresponding to those values.
  • each entry in a dimension table includes the attribute value (e.g., user identifier) and a hash value. The hash value may be used by the loader when transferring information to the main data warehouse.
  • each entry has a local identifier, which may be an index into the local dimension table. The loader maps these local identifiers to their corresponding main identifiers that are used in the main data warehouse.
  • the routine invokes a routine that identifies the logical site associated with the log entry and ensures that an entry for the logical site is in the logical site dimension table.
  • the routine invokes a routine that identifies the user associated with the log entry and ensures that an entry for the user is in the user dimension table.
  • the routine invokes a routine that identifies the URI associated with log entry and ensures that an entry for that URI is in the URI dimension table.
  • the routine invokes a routine that identifies the page type based on the parser configuration data and ensures that an entry for that page type is in the page type dimension table.
  • the routine invokes a routine that identifies the various events associated with the log entry based on the parser configuration data and ensures that an entry for each event type is in the corresponding event table.
  • the routine identifies other dimensions (e.g., referrer URI) as appropriate.
  • the routine adds an entry to the log entry table that is linked to each of the identified dimensions using the local identifiers.
  • the routine updates the statistics information based on the log entry and then returns.
  • FIG. 9 is a flow diagram of the identify logical site routine in one embodiment.
  • This routine compares the site information of the passed log entry with the logical site definitions in the parser configuration data.
  • the routine selects the next logical site definition from the parser configuration data.
  • decision block 902 if all the logical site definitions have already been selected, then the routine continues the block 905 , else the routine continues at block 903 .
  • decision block 903 if the URI of the log entry matches the selected logical site definition, then the routine continues at block 904 , else the routine loops to block 901 to select the next logical site definition.
  • the routine updates the logical site dimension table to ensure that it contains an entry for the logical site defined by the selected logical site definition. The routine then returns.
  • the routine updates the logical site dimension table to ensure that it contains a default logical site definition and then returns. The log entries that do not map to a logical site definition are mapped to a default logical site.
  • FIG. 10 is a flow diagram of the identify user routine in one embodiment.
  • This routine may use various techniques to identify the user associated with the passed log entry.
  • the selection of the technique is configured based on the customer web site. For example, one customer may specify to use a cookie to identify users. In absence of a user identifier in the cookie, the industry norm is to identify users based on their IP addresses.
  • This routine illustrates a technique in which a combination of cookies and IP addresses are used to identify a user.
  • the routine extracts the user identifier from the cookie associated with the log entry.
  • the format of a cookie may be specified on a customer-by-customer basis.
  • the routine continues at block 1006 , else the routine continues at block 1003 .
  • the extraction may not be successful if, for example, the log entry did not include a cookie.
  • the routine extracts the IP address from the log entry.
  • decision block 1004 if the IP address is determined to be unique, then routine continues at block 1006 , else the routine continues at block 1005 .
  • Certain IP addresses may not be unique. For example, an Internet service provider may use one IP address for many of its users. The Internet service provider performs the mapping of the one IP address to the various users.
  • the routine extracts the browser identifier from the log entry. The combination of IP address and browser identifier may uniquely identify a user.
  • the routine updates the user dimension table to ensure that it has an entry for this user and then returns.
  • FIG. 11 is a flow diagram of the identify page type routine in one embodiment.
  • This routine uses the page type definitions of the parser configuration data to identify the page type associated with the log entry.
  • the routine selects the next page type definition from the parser configuration data.
  • decision block 1101 if all the page type definitions have already been selected, then no matching page type has been found and the routine returns, else the routine continues at block 1103 .
  • decision block 1103 if the log entry matches the selected page type definition, then the routine continues at block 1104 , else the routine loops to block 1101 to select the next page type definition.
  • the routine updates the page type dimension table to ensure that it contains an entry for the page type represented by the selected page type definition. The routine then returns.
  • FIG. 12 is a flow diagram illustrating the identify events routine in one embodiment.
  • This routine determines whether the log entry corresponds to any of the events specified in the parser configuration data.
  • the routine selects the next type of event from the parser configuration data.
  • decision block 1202 if all the event types have already been selected, then the routine returns, else the routine continues at block 1203 .
  • the routine selects the next event definition of the selected event type.
  • decision block 1204 if all the event definitions of the selected event type have already been selected, then the log entry does not correspond to this type of event and the routine loops to block 1201 to select the next type of event, else the routine continues at block 1205 .
  • the routine continues at block 1206 , else the routine loops to block 1203 to select the next event definition of the selected event type.
  • the routine updates the dimension table for the selected type of the event to ensure that it contains an entry for the selected event definition.
  • the routine then loops to block 1201 to select the next type of event. In this way, the routine matches no more than one event definition for a given event type. For example, if there are two event definitions for the event type “Keyword Search,” then if the first one processed matches, then the second one is ignored.
  • FIG. 13 is a flow diagram illustrating the identify sessions routine in one embodiment.
  • This routine scans the log entry table of the local data warehouse to identify user sessions.
  • a user session may be delimited by a certain period of inactivity (e.g., thirty minutes).
  • the criteria for identifying a session may be configurable on a customer-by-customer basis.
  • the routine selects the next user from the user dimension table.
  • decision block 1302 if all the users have already been selected, then the routine returns, else the routine continues at block 1303 .
  • the routine selects the next log entry for the selected user in time order.
  • decision block 1304 if all log entries for the selected user have already been selected, then the routine loops to block 1301 to select the next user, else the routine continues at block 1305 .
  • decision block 1305 if the selected log entry indicates that a new session is starting (e.g., its time is more than 30 minutes greater than that of the last log entry processed), then the routine continues at block 1306 , else the routine loops to block 1303 to select the next log entry for the selected user.
  • the routine updates a session fact table to add an indication of the new session. The routine then loops to block 1303 to select the next log entry for the selected user. The routine may also update the log entries to reference their sessions.
  • FIG. 14 is a flow diagram of the generate aggregate statistics routine in one embodiment.
  • This routine generate statistics based on analysis of the fact and dimension tables used by the parser.
  • the routine selects the next fact table of intent.
  • decision block 1402 if all the fact tables have already been selected, then the routine returns, else the routine continues at block 1403 .
  • the routine selects the next entry of the selected fact table.
  • decision block 1404 if all the entries of the selected fact table have already been selected, then the routine loops to block 1401 to select the next fact table, else the routine continues at block 1405 .
  • the routine aggregates various statistics about the selected fact table. The routine then loops to block 1404 to select the next entry of the fact table.
  • FIGS. 15 - 17 are flow diagrams illustrating components of the loader in one embodiment.
  • FIG. 15 is a flow diagram of the load log data routine implementing the main routine of the loader in one embodiment. This routine controls the moving of the data from the local data warehouse (created and used by the parser) into the main data warehouse.
  • the routine invokes the create partitions routine to create partitions for the main data warehouse as appropriate.
  • the routine loops loading the dimension tables into the main data warehouse.
  • the routine selects the next dimension table.
  • decision block 1503 if all the dimension tables have already been selected, then the routine continues at block 1505 , else the routine continues at block 1504 .
  • the routine invokes the load dimension table routine for the selected dimension table.
  • the routine then loops to block 1502 to select the next dimension table.
  • the routine loops adding the entries to the fact tables of the main data warehouse.
  • the routine selects the next fact table in order.
  • the order in which the fact tables are to be loaded may be specified by configuration information.
  • the fact tables may be loaded in order based on their various dependencies. For example, a log entry fact table may be dependent on a user dimension table that is itself a fact table.
  • decision block 1506 if all the fact tables have already been loaded, then the routine returns, else the routine continues at block 1507 .
  • the routine invokes the load fact table routine for the selected fact table. The routine then loops to block 1505 to select the next fact table.
  • FIG. 16 is a flow diagram of the load dimension table routine in one embodiment.
  • This routine maps the local identifiers used in the local data warehouse to the main identifiers used in the main data warehouse.
  • the routine selects the next entry from the dimension table.
  • decision block 1602 if all the entries of the dimension table have already been selected, then the routine returns, else the routine continues at block 1603 .
  • the routine retrieves an entry from the dimension table of the main data warehouse corresponding to the selected entry.
  • decision block 1604 if the entry is retrieved, then the routine continues at block 1606 , else the dimension table does not contain an entry and the routine continues at block 1605 .
  • the routine adds an entry to the dimension table of the main data warehouse corresponding to the selected entry from the dimension table of the local data warehouse.
  • the routine creates a mapping of the local identifier (e.g., index into the local dimension table) of the selected entry to the main identifier (e.g., index into the main dimension table) for that selected entry.
  • the routine then loops to block 1601 to select the next entry of the dimension table.
  • FIG. 17 is a flow diagram of the load fact table routine in one embodiment.
  • This routine adds the facts of the local data warehouse to the main data warehouse.
  • the routine maps the local identifiers for the dimensions used in the local warehouse to the main identifiers of dimensions used in the main data warehouse.
  • the routine selects the next entry in the fact table.
  • decision block 1702 if all the entries of the fact table have already been selected, then the routine returns, else the routine continues at block 1703 .
  • the routine selects the next dimension for the selected entry.
  • decision block 1704 if all the dimensions for the selected entry have already been selected, then the routine continues at block 1706 , else the routine continues at block 1705 .
  • the routine retrieves the main identifier for the selected dimension and then loops to block 1703 to select the next dimension.
  • the routine stores an entry in the fact table of the main data warehouse. The routine then loops to block 1701 to select the next entry in the fact table.
  • FIG. 18 is a flow diagram illustrating the identify user aliases routine in one embodiment.
  • This routine tracks the different user identifiers as a user switches from one web site to another.
  • the routine maps the user identifiers used by a referrer web site to the user identifiers used by the referred-to web site. In this way, the same user can be tracked even though different web sites use different identifiers for that user.
  • This routine may be invoked as part of the parsing of the log files.
  • decision block 1801 if the log entry indicates a referrer web site, then the routine continues at block 1802 , else the routine returns.
  • the routine identifies the user identifier for the referrer web site.
  • the routine creates a mapping between the referrer user identifier and the referred-to user identifier. The routine then returns.

Abstract

A method and system for providing customers with access to and analysis of the navigation data collected at customer web sites is provided. In one embodiment, a data warehouse system collects customer data from the customer web sites and stores the data at a data warehouse server. The customer data may include navigation data (e.g., click stream log files), user attribute data of users of the customer web site (e.g., name, age, and gender), product data (e.g., catalog of products offered for sale by the customer), shopping cart data (i.e., identification of the products currently in a user's shopping cart), and so on. When the data warehouse server receives customer data, it converts the customer data into a format that is more conducive to processing by decision support system applications by which customers can analyze their data. For example, the data warehouse server may analyze low-level navigation events (e.g., each HTTP request that is received by the customer web site) to identify high-level events (e.g., a user session). The data warehouse server then stores the converted data into a data warehouse.

Description

    TECHNICAL FIELD
  • The described technology relates to analyzing of data relating to events generated by a computer program. [0001]
  • BACKGROUND
  • Today's computer networking environments, such as the Internet, offer mechanisms for delivering documents between heterogeneous computer systems. One such network, the World Wide Web network, which comprises a subset of Internet sites, supports a standard protocol for requesting and receiving documents known as web pages. This protocol is known as the Hypertext Transfer Protocol, or “HTTP.” HTTP defines a message passing protocol for sending and receiving packets of information between diverse applications. Details of HTTP can be found in various documents including T. Berners-Lee et al., [0002] Hypertext Transfer Protocol—HTTP 1.0, Request for Comments (RFC) 1945, MIT/LCS, May 1996. Each HTTP message follows a specific layout, which includes among other information, a header which contains information specific to the request or response. Further, each HTTP request message contains a universal resource identifier (a “URI”), which specifies to which network resource the request is to be applied. A URI is either a Uniform Resource Locator (“URL”) or Uniform Resource Name (“URN”), or any other formatted string that identifies a network resource. The URI contained in a request message, in effect, identifies the destination machine for a message. URLs, as an example of URIs, are discussed in detail in T. Berners-Lee, et al., Uniform Resource Locators (URI), RFC 1738, CERN, Xerox PARC, Univ. of Minn., December 1994.
  • FIG. 1 illustrates how a browser application enables users to navigate among nodes on the web network by requesting and receiving web pages. For the purposes of this discussion, a web page is any type of document that abides by the HTML format. That is, the document includes an “<HTML>” statement. Thus, a web page is also referred to as an HTML document. The HTML format is a document mark-up language, defined by the Hypertext Markup Language (“HTML”) specification. HTML defines tags for specifying how to interpret the text and images stored in an HTML document. For example, there are HTML tags for defining paragraph formats and for emboldening and underlining text. In addition, the HTML format defines tags for adding images to documents and for formatting and aligning text with respect to images. HTML tags appear between angle brackets, for example, <HTML>. Further details of HTML are discussed in T. Berners-Lee and D. Connolly, [0003] Hypertext Markup Language—2.0, RFC 1866, MIT/W3C, November 1995.
  • In FIG. 1, a [0004] web browser application 101 is shown executing on a client computer 102, which communicates with a server computer 103 by sending and receiving HTTP packets (messages). HTTP messages may also be generated by other types of computer programs, such as spiders and crawlers. The web browser “navigates” to new locations on the network to browse (display) what is available at these locations. In particular, when the web browser “navigates” to a new location, it requests a new document from the new location (e.g., the server computer) by sending an HTTP-request message 104 using any well-known underlying communications wire protocol. The HTTP-request message follows the specific layout discussed above, which includes a header 105 and a URI field 106, which specifies the network location to which to apply the request. When the server computer specified by URI receives the HTTP-request message, it interprets the message packet and sends a return message packet to the source location that originated the message in the form of an HTTP-response message 107. It also stores a copy of the request and basic information about the requesting computer in a log file. In addition to the standard features of an HTTP message, such as the header 108, the HTTP-response message contains the requested HTML document 109. When the HTTP-response message reaches the client computer, the web browser application extracts the HTML document from the message, and parses and interprets (executes) the HTML code in the document and displays the document on a display screen of the client computer as specified by the HTML tags. HTTP can also be used to transfer other media types, such as the Extensible Markup Language (“XML”) and graphics interchange format (“GIF”) formats.
  • The World Wide Web is especially conducive to conducting electronic commerce (“e-commerce”). E-commerce generally refers to commercial transactions that are at least partially conducted using the World Wide Web. For example, numerous web sites are available through which a user using a web browser can purchase items, such as books, groceries, and software. A user of these web sites can browse through an electronic catalog of available items to select the items to be purchased. To purchase the items, a user typically adds the items to an electronic shopping cart and then electronically pays for the items that are in the shopping cart. The purchased items can then be delivered to the user via conventional distribution channels (e.g., an overnight courier) or via electronic delivery when, for example, software is being purchased. Many web sites are also informational in nature, rather than commercial in nature. For example, many standards organizations and governmental organizations have web sites with a primary purpose of distributing information. Also, some web sites (e.g., a search engine) provide information and derive revenue from advertisements that are displayed. [0005]
  • The success of any web-based business depends in large pail on the number of users who visit the business's web site and that number depends in large part on the usefulness and ease-of-use of the web site. Web sites typically collect extensive information on how its users use the site's web pages. This information may include a complete history of each HTTP request received by and each HTTP response sent by the web site. The web site may store this information in a navigation file, also referred to as a log file or click stream file. By analyzing this navigation information, a web site operator may be able to identify trends in the access of the web pages and modify the web site to make it easier to use and more useful. Because the information is presented as a series of events that are not soiled in a useful way, many software tools are available to assist in this analysis. A web site operator would typically purchase such a tool and install it on one of the computers of the web site. There are several drawbacks with the use of such an approach of analyzing navigation information. First, the analysis often is given a low priority because the programmers are typically busy with the high priority task of maintaining the web site. Second, the tools that are available provide little more than standard reports relating to low-level navigation through a web site. Such reports are not very useful in helping a web site operator to visualize and discover high-level access trends. Recognition of these high-level access trends can help a web site operator to design the web site. Third, web sites are typically resource intensive, that is they use a lot of computing resources and may not have available resources to effectively analyze the navigation information. [0006]
  • It would also be useful to analyze the execution of computer programs, other than web server programs. In particular, many types of computer programs generate events that are logged by the computer programs themselves or by other programs that receive the events. If a computer program does not generate explicit events, another program may be able to monitor the execution and generate events on behalf of that computer program. Regardless of how event data is collected, it may be important to analyze that data. For example, the developer of an operating system may want to track and analyze how the operating system is used so that the developer can focus resources on problems that are detected, optimize services that are frequently accessed, and so on. The operating system may generate a log file that contains entries for various types of events (e.g., invocation of a certain system call).[0007]
  • BRIEF DESCRIPTION OF THE DRAWINGS
  • FIG. 1 illustrates how a browser application enables users to navigate among nodes on the web network by requesting and receiving web pages. [0008]
  • FIG. 2A is a block diagram illustrating components of the data warehouse system in one embodiment. [0009]
  • FIG. 2B is a block diagram illustrating details of the components of the data warehouse system in one embodiment. [0010]
  • FIG. 3 is a block diagram illustrating the sub-components of the data processor component in one embodiment. [0011]
  • FIG. 4 is a block diagram illustrating some of the tables of the local data warehouse and the main data warehouse in one embodiment. [0012]
  • FIG. 5 is a flow diagram illustrating the parse log data routine that implements the parser in one embodiment. [0013]
  • FIG. 6 is a flow diagram of the filter log entry routine in one embodiment. [0014]
  • FIG. 7 is a flow diagram illustrating the normalize log entry routine. [0015]
  • FIG. 8 is a flow diagram of the generate dimensions routine in one embodiment. [0016]
  • FIG. 9 is a flow diagram of the identify logical site routine in one embodiment. [0017]
  • FIG. 10 is a flow diagram of the identify user routine in one embodiment. [0018]
  • FIG. 11 is a flow diagram of the identify page type routine in one embodiment. [0019]
  • FIG. 12 is a flow diagram illustrating the identify events routine in one embodiment. [0020]
  • FIG. 13 is a flow diagram illustrating the identify sessions routine in one embodiment. [0021]
  • FIG. 14 is a flow diagram of the generate aggregate statistics routine in one embodiment. [0022]
  • FIG. 15 is a flow diagram of the import log data routine implementing the importer in one embodiment. [0023]
  • FIG. 16 is a flow diagram of the load dimension table routine and one embodiment. [0024]
  • FIG. 17 is a flow diagram of the load fact table routine in one embodiment. [0025]
  • FIG. 18 is a flow diagram illustrating the identify user aliases routine in one embodiment.[0026]
  • DETAILED DESCRIPTION
  • A method and system for providing customers with access to and analysis of event data (e.g., navigation data collected at customer web sites) is provided. The event data may be stored in log files and supplemented with data from other sources, such as product databases and customer invoices. In one embodiment, a data warehouse system collects customer data from the customer web sites and stores the data at a data warehouse server. The customer data may include application event data (e.g., click stream log files), user attribute data of users of the customer web site (e.g., name, age, and gender), product data (e.g., catalog of products offered for sale by the customer), shopping cart data (i.e., identification of the products currently in a user's shopping cart), and so on. The data warehouse server interacts with the customer servers to collect to the customer data on a periodic basis. The data warehouse server may provide instructions to the customer servers identifying the customer data that is to be uploaded to the data warehouse server. These instructions may include the names of the files that contains the customer data and the name of the web servers on which the files reside. These instructions may also indicate the time the day when the customer data is to be uploaded to the data warehouse server. When the data warehouse server receives customer data, it converts the customer data into a format that is more conducive to processing by decision support system applications by which customers can analyze their data. For example, the data warehouse server may analyze low-level navigation events (e.g., each HTTP request that is received by the customer web site) to identify high-level events (e.g., a user session). The data warehouse server then stores the converted data into a data warehouse. The data warehouse server functions as an application service provider that provides various decision support system applications to the customers. For example, the data warehouse server provides decision support system applications to analyze and graphically display the results of the analysis for a customer. The decision support system applications may be accessed through a web browser. In one embodiment, the customer servers are connected to the data warehouse server via the Internet and the data warehouse server provides data warehousing services to multiple customers. [0027]
  • The data warehouse system may provide a data processor component that converts the log files into a format that is more conducive to processing by the decision support system applications. In one embodiment, the converted data is stored in a data warehouse that includes fact and dimension tables. Each fact table contains entries corresponding to a type of fact derived from the log files. For example, a web page access fact table may contain an entry for each web page access identified in the log files. Each entry may reference attributes of the web page access, such as the identity of the web page and identity of the accessing user. The values for each attribute are stored in a dimension table for that attribute. For example, a user dimension table may include an entry for each user and the entries of the web access fact table may include a user field that contains an index (or some other reference) to the entry of the user dimension table for the accessing user. The user dimension table may contain the names of the users and other user-specific information. Alternatively, the user dimension table may itself also be a fact table that includes references to dimension tables for the attributes of users. The data warehouse may also include fact tables and dimension tables that represent high-level facts and attributes derived from the low-level facts and attributes of the log files. For example, high-level facts and attributes may not be derivable from only the data in a single log entry. For example, the higher level category (e.g., shoes or shirts) of a web page may be identified using a mapping of web page URIs to categories. These categories may be stored in a category dimension table. Also, certain facts, such as the collection of log entries that comprise a single user web access session or visit, may only be derivable by analyzing a series of log entries. [0028]
  • The data processor component may have a parser component and a loader component. The parser of the data processor parses and analyzes a log file and stores the resulting data in a local data warehouse that contains information for only that log file. The local data warehouse may be similar in structure (e.g., similar fact and dimension is tables) to the main data warehouse used by decision support system applications. The local data warehouse may be adapted to allow efficient processing by the parser. For example, the local data warehouse may be stored in primary storage (e.g., main memory) for speed of access, rather than in secondary storage (e.g., disks). The parser may use parser configuration data that defines, on a customer-by-customer basis, the high-level data to be derived from the log entries. For example, the parser configuration data may specify the mapping of URIs to web page categories. The loader of the data processor transfers the data from the local data warehouse to the main data warehouse. The loader may create separate partitions for the main data warehouse. These separate partitions may hold the customer data for a certain time period (e.g., a month's worth of data). The loader adds entries to the main fact tables (i.e., fact tables of the main data warehouse) for each fact in a local fact table (i.e., fact table of the local data warehouse). The loader also adds new entries to the main dimension tables to represent attribute values of the local dimension tables that are not already in the main dimension tables. The loader also maps the local indices (or other references) of the local dimension tables to the main indices used by the main dimension tables. [0029]
  • FIG. 2A is a block diagram illustrating components of the data warehouse system in one embodiment. The data warehouse system includes customer components that execute on the customer servers and data warehouse components that execute on the data warehouse server. The [0030] customer servers 210 and the data warehouse server 260 are interconnected via the Internet 250. Customer components executing on a customer server includes a data collection component 220 and a data viewer 230. The data viewer may reside on a client computer of the customer, rather than a server. The data collection component collects the customer data from the storage devices 240 of the customer servers. The data viewer provides access for viewing of data generated by the decision support system applications of the data warehouse server. In one embodiment, the data viewer may be a web browsed The data warehouse server includes a data receiver component 270, the data processor component 280, the data warehouse 290, and decision support system applications 291. The data receiver component receives customer data sent by the data collection components executing at the various customer web sites. The data processor component processes the customer data and stores it in the data warehouse. The decision support system application provides the customer with tools for analyzing and reviewing the customer data that is stored in the main data warehouse. Analysis performed on and reports generated from are described in U.S. patent application Ser. No. ______ (Attorney Ref. No. 34821-8010US), entitled “Identifying and Reporting on Combinations of Events in Usage Data,” and U.S. patent application Ser. No. ______ (Attorney Ref No. 34821-8013US), entitled “Extracting and Displaying Usage Data for Graphical Structures,” which are being filed concurrently and which are hereby incorporated by reference. In one embodiment, each customer has its own set of dimension and fact tables so that the information of multiple customers are not intermingled.
  • FIG. 2B is a block diagram illustrating details of the components of the data warehouse system in one embodiment. The [0031] data collection component 220 includes a monitor sub-component 221 and a pitcher sub-component 222. The data collection component is described in more detail in U.S. patent application Ser. No. ______ (Attorney Ref. No. 34821-800/US), entitled “Method and System for Monitoring Resource via the Web,” which is being filed concurrently and which is hereby incorporated by reference. The pitcher is responsible for retrieving instructions from the data warehouse server, collecting the customer data in accordance with the retrieved instructions, and uploading the customer data to the data warehouse server. The monitor is responsible for monitoring the operation of the pitcher and detecting when the pitcher may have problems in collecting and uploading the customer data. When the monitor detects that a problem may occur, it notifies the data warehouse server so that corrective action may be taken in advance of the collecting and uploading of the customer data. For example, the pitcher may use certain log on information (e.g., user ID and password) to access a customer web server that contains customer data to be uploaded. The monitor may use that log on information to verify that the log on information will permit access to the customer data. Access may be denied if, for example, a customer administrator inadvertentry deleted from the customer web server the user ID used by the pitcher. When the monitor provides advance notification of a problem, the problem might be corrected before the pitcher attempts to access the customer data. The monitor also periodically checks the pitcher to ensure that the pitcher is executing and, if executing, executing correctly.
  • The data receiver component of the data warehouse server includes a status receiver sub-component [0032] 271, a catcher sub-component 272, an FTP server 273, a status database 274, and a collected data database 275. The status receiver receives status reports from the customer servers and stores the status information in the status database. The catcher receives and processes the customer data that is uploaded from the customer web sites and stores the data in the collected data database. The data processor component includes a parser sub-component 281 and a loader sub-component 282. The parser analyzes the low-level events of the customer data and identifies high-level events and converts the customer data into a format that facilitates processing by the decision support system applications. The loader is responsible for storing the identified high-level events in the data warehouse 290. In one embodiment, a customer may decide not to have the data collection component executing on its computer systems. In such a case, the customer server may include an FTP client 245 that is responsible for periodically transferring the customer data to the FTP server 273 of the data warehouse server. The data receiver may process this customer data at the data warehouse server in the same way as the pitcher processes the data at the customer servers. The processed data is then stored in the collected data database.
  • FIG. 3 is a block diagram illustrating the sub-components of the data processor component in one embodiment. The [0033] data processor component 300 includes a parser 310, data storage area 370, and a loader 330. The data processor component inputs parser configuration data 340 and a log file 350 and updates the main data warehouse 360. The parser configuration data may include a mapping of actual web sites to logical sites and a mapping of a combination of Uniform Resource Identifiers (“URls”) and query strings of the log entries to page definitions (e.g., categories) and event definitions. The parser processes the entries of the log file to generate facts and dimensions to eventually be stored in the main data warehouse. The parser identifies events in accordance with the parser configuration data. The parser includes a filter log entry component 311, a normalize log entry component 312, a generate dimensions component 313, an identify sessions component 314, and a generate aggregate statistics component 315. The filter log entry component identifies which log entries should not be included in the main data warehouse. For example, a log entry that has an invalid format should not be included. The normalize log entry component normalizes the data in a log entry. For example, the component may convert all times to Greenwich Mean Time (“GMT”). The generate dimensions component identifies the various dimensions related to a log entry. For example, a dimension may be the Uniform Resource Identifier of the entry or the logical site identifier. The identify sessions component processes the parsed log file data stored in the local data warehouse to identify user sessions. A user session generally refers to the concept of a series of web page accesses that may be related in some way, such as by temporal proximity. The generate aggregate statistics component aggregates data for the log file being processed as each log entry is processed or after the log file is parsed. The data storage area 320 includes a local data warehouse 321. In one embodiment, the local data warehouse is stored non-persistently (or temporarily) in main memory of the computer system. The local data warehouse may contain fact tables and dimension tables that correspond generally to the tables of the main data warehouse 360. The loader retrieves the information from the local data warehouse and stores the information in the main data warehouse. The loader includes a create partitions component 331, a load dimension table component 332, and a load fact table component 333. The create partitions components creates new partitions for the main data warehouse. A partition may correspond to a collection of information within a certain time range. For example, the main data warehouse may have a partition for each month, which contains all the data for that month. The load dimension table component and the load fact table component are responsible for loading the main data warehouse with the dimensions and facts that are stored in the local data warehouse.
  • In one embodiment, the log file is a web server log file of a customer. The log file may be in the “Extended Log File Foimat” as described in the document “http://www.w3.org/TR/WD-logfile-960323” provided by the World Wide Web Consortium, which is hereby incorporated by reference. According to that description, the log file contains lines that are either directives or entries. An entry corresponds to a single HTTP transaction (e.g., HTTP request and an HTTP response) and consists of a sequence of fields (e.g., integer, fixed, URI, date, time, and string). The meaning of the fields in an entry is specified by a field directive specified in the log file. For example, a field directive may specify that a log entry contains the fields date, time, client IP address, server IP address, and success code. Each entry in the log file would contain these five fields. [0034]
  • The parser configuration data defines logical sites, page definitions, and event definitions. A logical site is a collection of one or more IP addresses and ports that should be treated as a single web site. For example, a web site may actually have five web servers with different IP addresses that handle HTTP requests for the same domain. These five IP addresses may be mapped to the same logical site to be treated as a single web site. The page definitions define the format of the URIs of log entries that are certain page types. For example, a URI with a query string of “category=shoes” may indicate a page type of “shoes.” Each event definition defines an event type and a value for that event type. For example, a log entry with a query string that includes “search=shoes” represents an event type of “search” with an event value of “shoes.” Another log entry with a query string of “add=99ABC” may represent an event type of “add” an item to the shopping cart with an event value of item number “99ABC.”[0035]
  • FIG. 4 is a block diagram illustrating some of the tables of the local data warehouse and the main data warehouse in one embodiment. These data warehouses are databases that include fact tables and dimension tables. A fact table contains an entry for each instance of fact (e.g., web page access). A dimension table contains an entry for each possible attribute value of an attribute (e.g., user). The entries of a fact table contain dimension fields that refer to the entries into the dimension tables for their attribute values. A table may be both a fact table and a dimension table. For example, a user dimension table with an entry for each unique user may also be a fact table that refers to attributes of the users that are stored in other dimension tables. The data warehouses contain a log entry table [0036] 401, a user table 402, a logical site table 403, a URI table 404, a referrer URI table 405, a page type table 406, event type tables 407, a query string table 408, and a referrer query string table 409. The log entry table is a fact table that contains an entry for each log entry that is not filtered out by the parser. The other tables are dimension tables for the log entry table. The user table contains an entry for each unique user identified by the parser. The logical site table contains an entry for each logical site as defined in the parser configuration data. The URI table contains an entry for each unique URI of an entry in the log entry table. The referred URI table contains an entry for each referrer URI of the log entry table. The page type table contains an entry for each page type identified by the parser as defined in the parser configuration data. The data warehouse contains an event table for each type of event defined in the parser configuration data. Each event table contains an entry for each event value of that event type specified in an entry of the log entry table. The query string table contains an entry for each unique query string identified in an entry of the log entry table. The referrer query string contains an entry for each unique referrer query string identified in an entry of the log entry table.
  • Table 1 is an example portion of a log file. The “#fields” directive specifies the meaning of the fields in the log entries. Each field in a log entry is separated by a space and an empty field is represented by a hyphen. The #fields directive in this example indicates that each entry includes the date and time when the transaction was completed (i.e., “date” and “time”), the client IP address (i.e., “c-ip”), and so on. For example, the first log entry has a data and time of “2000-06-01 07:00:04” and a client IP address of “165.21.83.61.” [0037]
    TABLE 1
    #Software: Microsoft Internet Information Server 4.0
    #Version: 1.0
    #Date: 2000-06-01 07:00:04
    #Fields: date time c-ip cs-usemame s-sitename s-computername s-ip cs-method cs-uri-stem cs-uri-query sc-status sc-win32-
    status sc-bytes cs-bytes time-taken s-port cs-version cs(User-Agent) cs(Cookie) cs( Referrer)
    2000-06-01 07:00:04 165.21.83.161 - W3SVC2 COOK_002 206.191.163.41 GET /directory/28.ASP - 200 0 148428 369
    9714  80  HTTP/1.0  Mozilla/3.04+(Win95+1) ASPSESSIONIDQQGGQGPG=JBCCFIPBBHHDANBAFFGLGPH
    http://allrecipes.com/Default.asp
    2000-06-01 07:00:20 4.20.197.70 - W3SVC2 COOK_002 206.191.163.41 GET /Default.asp - 302 0 408 259 30 80
    HTTP/1.0 Mozilla/4.0+(compatible:+Keynote-Perspective+4.0)- -
    2000-06-01 07:00:20 4.20 197.70 - W3SVC2 COOK_002 206.191.163.41 GET /Default.asp - 200 0 41245 266 200 80
    HTTP/1.0 Mozilla/4.0+(compatible:+Keynote-Perspective+4.0)- -
    2000-06-01 07:00.27 204.182.65.192 - W3SVC2 COOK_002 206.191.163.41 HEAD /Default.asp - 302 0 254 66 40 80
    HTTP/1.0 Ipswitch_WhatsUp/3.0 - -
    2000-06-01 07:00:32 24.10.69.137 - W3SVC2 COOK_002 206.191.163.41 GET /directory/541.asp - 200 0 22427 459 421
    80  HTTP/1.0  Mozilla/4.7+[en](Win98+U) ASPSESSIONIDQQGGQGPG=BHBCFIPBEJPNOMDPKCGLKNGC;
    +ARSiteUser=1%2DC2B25364%2D3775%2D11D4%2DBC1%2D0050049BD2E4;+ARSites=ALR=1
    http://allrecipes.com/directory/34.asp
    2000-06-01 07:00:34 192.102.216.101 - W3SVC2 COOK_002 206.191.163.41 GET /encyc/terms/L/7276.asp - 200 0
    20385  471  290  80  HTTP/1.0 Mozilla/4.7+[en]+(X11;+1:+SunOS+5.5.5+sun4u) ASPSESSIONIDQQGGQGPG=
    PKBCFIPBIKONBPDHKDMMEHCE http://search.allrecipes.com/gsearchresults.asp?site=aIlrecipes&allrecipes=allrecipes&
    allsites=1&q1=loin
    2000-06-01 07:00:34 216.88.216.227 - W3SVC2 COOK_002 206.191.163.41 GET /Default.asp -200 0 41253 258 180 80
    HTTP//1.1 Mozila/4.0+(compatible:*MSIE+4.01:+MSN+2.5:MSN+2.5:+Windows+98)- -
    2000-06-01 07:00:36 199.203.4.10 - W3SVC2 COOK_002 206.191.163.41 GET /Default.asp - 302 0 408 485 30 80
    HTTP/1.0 Mozilla/4.0+(compatible:+MSIE:+5.01:+Windows+98:+TUCOWS) SITESERVER=ID=22f117fb3708b2278f3c
    426796a78e2a-
    2000-06-01 07:00:37 199.203.4.10 - W3SVC2 COOK_002 206.191.163.41 GET /Default.asp - 200 0 41277 492 421 80
    HTTP/1.0 Mozilla/4.0+(compatible:+MSIE.+5.01:+Windows+98:+TUCOWS) SITESERVER=ID=22fii71b3708b2278f3c
    426796a78e2a-
    2000-06-01 07:00:43 24.10.69.137 - W3SVC2 COOK_002 206.191.163.41 GET /directory/34.asp - 200 0 17835 458 320
    80  HTTP/1.0  Mozilla/4.7+[en]+(Win98:+U)ASPSESSIONIDQQGGQGPG=BHBCFIPBEJPNOMDPKCGLKNGC;
    +ARSiteUser=1%2DC2B25364%2D3775%,2D11D4%2D11D4%2DBAC1%2D0050049BD2E4;+ARSites=ALR=1 http://allrecipes.
    com/directory/25.asp
    2000-06-01  0700:47  199.203.4.10  -  W3SVC2  COOK_002  206.191.16341  GET  /jumpsite.asp
    jumpsite=5&Go.x=16&Go.y=14 302 0 341 611 40 80 HTTP/1.0 Mozilla/4.0+(compatible;+MSIE+5.01;+Windows+98;+
    TUCOWS)  SITESERVER=ID=22f117tb3708b2278f3c426796a78e2a:+ASPSESSIONIDQQGGQGPG=FCCCFIPBKJM
    BDJJHBNCOEDGII http://allrecipes.com)Default.asp
    2000-06-01 07:00:47 24.10.69.137 - W3SVC2 COOK_002 206.191.163.41 GET /directory/538.asp - 200 0 27471 459 881
    80  HTTP/1.0  Mozilla/4.7+[en]+(Win98:+U)  ASPSESSIONIDQQGGQGPG=BHBCFIPBEJPNOMDPKCGLKNGC;
    +ARSiteUser+1%2DC2B25364%2D3775%2D11D4%2DBAC1%2D0050049BD2E4+ARSites=ALR=1
    http://allrecipes.com/directory/34.asp
    2000-06-01 07:00:47 207.136.48.117 - W3SVC2 COOK_002 206.191.163.41 GET /directory/511.asp - 200 0 77593 369
    12538 80 HTTP/1.0 Mozilla/3.01Gold+(Win95:+) ASPSESSIONIDQQGGQGPG=MFACFIPBDBN PBFBOENJKHJN;
    +ARSiteUser=1%2DC2B251E5%2D3775%2D11D4%2DBAC1%2D0050049BD2E4+ARSites=ALR=1
    http://allrecipes.com/directory/506.asp
    2000-06-01 07:00.49 192.102.216.101 - W3SVC2 COOK_002 206.191.163.41 GET /encyc/Al.asp ARRefSite=
    15&ARRefCookie=1-C2B253B8-3775-11D4-BAC1-0050049BD2E4 200 0 47193 457 260 80 HTTP/1.0 Mozilla/4.7+
    [en]+(X11+1;+SunOS+5.5.1.+sun4u)   ASPSESSIONIDQQGGQGPG=PKBCFIPBIKONBPDHKDMMEHCE
    http://porkrecipe.com/hints/tips.asp
  • Table 2 is an example portion of parser configuration data. The logical site definitions map a server IP address, port, and root URI to a logical site. For example, the entry “LOGICALSITEURIDEFINITION=209.114.94.26,80,/,1” maps all the accesses to port [0038] 80 of IP address 209.114.94.26 at URIs with a prefix “/” to logical site 1. The page type definitions map a logical site identifier, URI pattern, and query string pattern to a page type. For example, the entry “PAGEKEYDEFINITION=news item, news item, 1, {prefix}=homepage include/industrynews_detail. asp, <NewsItemID># {Uri}” indicates that a page type of “news item” is specified for logical site 1 by a URI pattern of “/homepage_include/industrynews_detail.asp.” The definition also indicates that the event value is “<NewsItemID>#{Uri},” where the URI of the log entry is substituted for “{Uri} and the value of NewsItemID in the query string is substituted for “<NewsItemID>.” The event type definitions map a site identifier, URI pattern, and query string pattern to an event type and value. The definitions also specify the name of the event type and the name of the dimension table for that event type. For example, the entry “EVENTDEFINITION=View News Article, View News Article, 1, {prefix}=/homepage_include/industrynews_detail.asp, <NewsItemId>=*, <Newsltemld>” indicates that View News Article event types are stored in the View News Article dimension table. That event type is indicated by a URI with “Homepage_include/industrynews_detail.asp,” and the event value is the sting that follows “<Newsltemld>=” in the query string.
    TABLE 2
    LOGICALSITEURIDEFINITION= 209.114.94.26. 80./.1
    PAGEKEYDEFINITION= news item, news item, 1, {prefix}=/homepage_include/industrynews_detail.asp.,
    <NewsItermId>#{Uri}
    PAGEKEYDEFINITION= page, page, 1...{Uri}
    EVENTDEFINITION= Login, 1, {prefix} =/:registration/login.asp,,
    EVENTDEFINITION= Logout, Logout, 1, {prefix}=/registration/logout.asp,,
    EVENTDEFINITION= Register Page 1, Register Page 1, 1, {prefix}=/registration/register.asp,,
    EVENTDEFINITION= Register Page 2, Register Page 2, 1, {prefix}=/registration/register2.asp. <UserID=,
    EVENTDEFINITION= Registration Confirmation. Registration Confirmation, 1, {prefix}=/registration/register3.asp,,
    EVENTDEFINITION= Abort Registration, Abort Registration, 1, {prefix}=/registration/registrationabort.asp,,
    EVENTDEFINITION= Member Services, Member Services, 1, {prefix}=/registration/memherservices.asp,,
    EVENTDEFINITION= Change Password, Change Password, 1, 55 prefix}=registration/changepassword.asp,,
    EVENTDEFINITION= Profile Edit, Profile Edit, 1, {prefix}=/registration/profile.asp,,
    EVENTDEFINITION= Change Affiliation, Change Affiliation, 1, {prefix}=/registration/changeaffiliation.asp,<UserID>=*,
    EVENTDEFINITION= Change Secret Question, Change Secret Question, 1, {prefix}=/registration/changesecretquestion.asp,,
    EVENTDEFINITION= Forgot Information, Forgot Information, 1, {prefix}=/registration/forgotinfo.asp,,
    EVENTDEFINITION= Forgot Password, Forgot Password, 1. {prefix}=/registration/forgotpassword.asp,,
    EVENTDEFINITION= Forgot Signin, Forgot Signin, 1, {prefix}=/registration/forgotsignin.asp,,
    EVENTDEFINITION= View News Article, View News Article, 1, {prefix}=/homepage_include/industrynews_detail.asp,
    <NewsItemId>=*.<NewsItemId>
  • FIGS. [0039] 5-14 are flow diagrams of components of the parser in one embodiment. FIG. 5 is a flow diagram illustrating the parse log data routine that implements the main routine of parser in one embodiment. The routine processes each entry in the log file based on the parser configuration data. The routine filters out certain log entries, normalizes the attribute values of the log entries, and generates entries in the dimension tables for the attributes of the log entries. After processing all the log entries, the parser identifies user sessions and generates various statistics. In blocks 501-508, the routine loops selecting and processing each log entry. In block 501, the routine selects the next log entry of the log file starting with the first log entry. The routine may also pre-process the header information of the log file to identify the fields of the log entries. In decision block 1502, if all the log entries have already been selected, then the routine continues at block 509, else the routine continues at block 503. In block 503, the routine extracts the values for the fields of the selected log entry. In block 504, the routine invokes the filter log entry routine, which returns an indication as to whether the selected log entry should be filtered out. In decision block 505, if the filter log entry routine indicates that the selected log entry should be filtered out, then the routine skips to block 508, else the routine continues at block 506. In block 506, the routine invokes the normalize log entry routine to normalize the values of the fields of the selected log entry. In block 507, the routine invokes the generate dimensions routine to update the dimension tables based on the selected log entry and to add an entry into the log entry fact table. In block 508, the routine updates the statistics for the log file. For example, the routine may track the number of log entries that have been filtered out. The routine then loops to block 501 to select the next log entry. In block 509, the routine outputs the log file statistics. In block 510, the routine invokes the identify sessions routine that scans the log entry table to identify the user sessions and updates a session dimension table. In block 511, the routine invokes the generate aggregate statistics routine to generate various statistics and then completes.
  • FIG. 6 is a flow diagram of the filter log entry routine in one embodiment. The filter log entry routine is passed a log entry and determines whether the log entry should be filtered out. In blocks [0040] 601-607, the routine determines whether the filter out conditions have been satisfied. In decision block 601, the routine determines whether the log entry has a field count problem. A field count problem arises when the number of fields in the log entry does not correspond to the number of expected fields for that log entry. The number and types of fields may be defined in a “fields” directive line of the log file. In decision block 602, the routine determines whether the log entry is outside of a specified time range. The routine compares the time field of the log entry to the time range. The time range may be specified so that only those log entries within that time range are processed. In decision block 603, the routine determines whether the IP address of the log entry should be ignored. For example, a log entry may be ignored if the entry originated from a server whose function is to ping the customer's web server at periodic intervals. In decision block 604, the routine determines whether the log entry corresponds to a comment (e.g., a “g#remarks” directive). In decision block 605, the routine determines whether the success code associated with the log entry indicates that log entry should be ignored. For example, if the success code indicates a failure, then the log entry may be ignored. In decision block 606, the routine determines whether the log entry is requesting a resource whose extension indicates that the log entry should be ignored. For example, the routine may ignore log entries requesting graphic files, such as those in the “.gif” format. In decision block 607, the routine determines whether the values within the fields of the log entry are corrupt. For example, a value in the date field that indicates a date of February 30th is corrupt. One skilled in the art would appreciate that the various filtering, conditions may be specified in a configuration file. For example, the time range, IP addresses, and so on may be specified in the configuration file. These configuration files may be specified on a customer-by-customer basis.
  • FIG. 7 is a flow diagram illustrating the normalize log entry routine. The routine normalizes the values of the fields in the passed log entry. In [0041] block 701, the routine converts the time of the log entry into a standard time such as Greenwich Mean Time. In block 702, the routine collects the time based on the variation between the times of the customer web servers. For example, the time of one web server may be five minutes ahead of the time of another web server. This correction may be based on current time information collected from computer systems that generated the events and then correlated to base current time information. In block 703, the routine normalizes, the values of the fields of the log entry. This normalization may include processing search strings to place them in a canonical form. For example, a search string of “back pack” may have a canonical form of “backpack.” Other normalization of search strings may include stemming of words (e.g., changing “clothes” and “clothing” to “cloth”), synonym matching, and first and last word grouping. The first word grouping for the search strings of “winter clothing” and “winter shoes” results in the string of “winter.”
  • FIG. 8 is a flow diagram of the generate dimensions routine in one embodiment. This routine identifies a value for each dimension associated with the passed log entry and ensures that the dimension tables contains entries corresponding to those values. In one embodiment, each entry in a dimension table includes the attribute value (e.g., user identifier) and a hash value. The hash value may be used by the loader when transferring information to the main data warehouse. Also, each entry has a local identifier, which may be an index into the local dimension table. The loader maps these local identifiers to their corresponding main identifiers that are used in the main data warehouse. In [0042] block 801, the routine invokes a routine that identifies the logical site associated with the log entry and ensures that an entry for the logical site is in the logical site dimension table. In block 802, the routine invokes a routine that identifies the user associated with the log entry and ensures that an entry for the user is in the user dimension table. In block 803, the routine invokes a routine that identifies the URI associated with log entry and ensures that an entry for that URI is in the URI dimension table. In block 804, the routine invokes a routine that identifies the page type based on the parser configuration data and ensures that an entry for that page type is in the page type dimension table. In block 805, the routine invokes a routine that identifies the various events associated with the log entry based on the parser configuration data and ensures that an entry for each event type is in the corresponding event table. In block 806, the routine identifies other dimensions (e.g., referrer URI) as appropriate. In block 807, the routine adds an entry to the log entry table that is linked to each of the identified dimensions using the local identifiers. In block 808, the routine updates the statistics information based on the log entry and then returns.
  • FIG. 9 is a flow diagram of the identify logical site routine in one embodiment. This routine compares the site information of the passed log entry with the logical site definitions in the parser configuration data. In [0043] block 901, the routine selects the next logical site definition from the parser configuration data. In decision block 902, if all the logical site definitions have already been selected, then the routine continues the block 905, else the routine continues at block 903. In decision block 903, if the URI of the log entry matches the selected logical site definition, then the routine continues at block 904, else the routine loops to block 901 to select the next logical site definition. In block 904, the routine updates the logical site dimension table to ensure that it contains an entry for the logical site defined by the selected logical site definition. The routine then returns. In block 905, the routine updates the logical site dimension table to ensure that it contains a default logical site definition and then returns. The log entries that do not map to a logical site definition are mapped to a default logical site.
  • FIG. 10 is a flow diagram of the identify user routine in one embodiment. This routine may use various techniques to identify the user associated with the passed log entry. In one embodiment, the selection of the technique is configured based on the customer web site. For example, one customer may specify to use a cookie to identify users. In absence of a user identifier in the cookie, the industry norm is to identify users based on their IP addresses. This routine illustrates a technique in which a combination of cookies and IP addresses are used to identify a user. In [0044] block 1001, the routine extracts the user identifier from the cookie associated with the log entry. The format of a cookie may be specified on a customer-by-customer basis. In decision block 1002, if the extraction from the cookie was successful, then the routine continues at block 1006, else the routine continues at block 1003. The extraction may not be successful if, for example, the log entry did not include a cookie. In block 1003, the routine extracts the IP address from the log entry. In decision block 1004, if the IP address is determined to be unique, then routine continues at block 1006, else the routine continues at block 1005. Certain IP addresses may not be unique. For example, an Internet service provider may use one IP address for many of its users. The Internet service provider performs the mapping of the one IP address to the various users. In block 1005, the routine extracts the browser identifier from the log entry. The combination of IP address and browser identifier may uniquely identify a user. In block 1006, the routine updates the user dimension table to ensure that it has an entry for this user and then returns.
  • FIG. 11 is a flow diagram of the identify page type routine in one embodiment. This routine uses the page type definitions of the parser configuration data to identify the page type associated with the log entry. In [0045] block 1101, the routine selects the next page type definition from the parser configuration data. In decision block 1101, if all the page type definitions have already been selected, then no matching page type has been found and the routine returns, else the routine continues at block 1103. In decision block 1103, if the log entry matches the selected page type definition, then the routine continues at block 1104, else the routine loops to block 1101 to select the next page type definition. In block 1104, the routine updates the page type dimension table to ensure that it contains an entry for the page type represented by the selected page type definition. The routine then returns.
  • FIG. 12 is a flow diagram illustrating the identify events routine in one embodiment. This routine determines whether the log entry corresponds to any of the events specified in the parser configuration data. In [0046] block 1201, the routine selects the next type of event from the parser configuration data. In decision block 1202, if all the event types have already been selected, then the routine returns, else the routine continues at block 1203. In block 1203, the routine selects the next event definition of the selected event type. In decision block 1204, if all the event definitions of the selected event type have already been selected, then the log entry does not correspond to this type of event and the routine loops to block 1201 to select the next type of event, else the routine continues at block 1205. In block 1205, if the log entry matches the selected event definition, then the routine continues at block 1206, else the routine loops to block 1203 to select the next event definition of the selected event type. In block 1206, the routine updates the dimension table for the selected type of the event to ensure that it contains an entry for the selected event definition. The routine then loops to block 1201 to select the next type of event. In this way, the routine matches no more than one event definition for a given event type. For example, if there are two event definitions for the event type “Keyword Search,” then if the first one processed matches, then the second one is ignored.
  • FIG. 13 is a flow diagram illustrating the identify sessions routine in one embodiment. This routine scans the log entry table of the local data warehouse to identify user sessions. In one embodiment, a user session may be delimited by a certain period of inactivity (e.g., thirty minutes). The criteria for identifying a session may be configurable on a customer-by-customer basis. In [0047] block 1301, the routine selects the next user from the user dimension table. In decision block 1302, if all the users have already been selected, then the routine returns, else the routine continues at block 1303. In block 1303, the routine selects the next log entry for the selected user in time order. In decision block 1304, if all log entries for the selected user have already been selected, then the routine loops to block 1301 to select the next user, else the routine continues at block 1305. In decision block 1305, if the selected log entry indicates that a new session is starting (e.g., its time is more than 30 minutes greater than that of the last log entry processed), then the routine continues at block 1306, else the routine loops to block 1303 to select the next log entry for the selected user. In block 1306, the routine updates a session fact table to add an indication of the new session. The routine then loops to block 1303 to select the next log entry for the selected user. The routine may also update the log entries to reference their sessions.
  • FIG. 14 is a flow diagram of the generate aggregate statistics routine in one embodiment. This routine generate statistics based on analysis of the fact and dimension tables used by the parser. In [0048] block 1401, the routine selects the next fact table of intent. In decision block 1402, if all the fact tables have already been selected, then the routine returns, else the routine continues at block 1403. In block 1403, the routine selects the next entry of the selected fact table. In decision block 1404, if all the entries of the selected fact table have already been selected, then the routine loops to block 1401 to select the next fact table, else the routine continues at block 1405. In block 1405, the routine aggregates various statistics about the selected fact table. The routine then loops to block 1404 to select the next entry of the fact table.
  • FIGS. [0049] 15-17 are flow diagrams illustrating components of the loader in one embodiment. FIG. 15 is a flow diagram of the load log data routine implementing the main routine of the loader in one embodiment. This routine controls the moving of the data from the local data warehouse (created and used by the parser) into the main data warehouse. In block 1501, the routine invokes the create partitions routine to create partitions for the main data warehouse as appropriate. In blocks 1502-1504, the routine loops loading the dimension tables into the main data warehouse. In block 1502, the routine selects the next dimension table. In decision block 1503, if all the dimension tables have already been selected, then the routine continues at block 1505, else the routine continues at block 1504. In block 1504, the routine invokes the load dimension table routine for the selected dimension table. The routine then loops to block 1502 to select the next dimension table. In blocks 1505-1507, the routine loops adding the entries to the fact tables of the main data warehouse. In block 1505, the routine selects the next fact table in order. The order in which the fact tables are to be loaded may be specified by configuration information. The fact tables may be loaded in order based on their various dependencies. For example, a log entry fact table may be dependent on a user dimension table that is itself a fact table. In decision block 1506, if all the fact tables have already been loaded, then the routine returns, else the routine continues at block 1507. In block 1507, the routine invokes the load fact table routine for the selected fact table. The routine then loops to block 1505 to select the next fact table.
  • FIG. 16 is a flow diagram of the load dimension table routine in one embodiment. This routine maps the local identifiers used in the local data warehouse to the main identifiers used in the main data warehouse. In [0050] block 1601, the routine selects the next entry from the dimension table. In decision block 1602, if all the entries of the dimension table have already been selected, then the routine returns, else the routine continues at block 1603. In block 1603, the routine retrieves an entry from the dimension table of the main data warehouse corresponding to the selected entry. In decision block 1604, if the entry is retrieved, then the routine continues at block 1606, else the dimension table does not contain an entry and the routine continues at block 1605. In block 1605, the routine adds an entry to the dimension table of the main data warehouse corresponding to the selected entry from the dimension table of the local data warehouse. In block 1606, the routine creates a mapping of the local identifier (e.g., index into the local dimension table) of the selected entry to the main identifier (e.g., index into the main dimension table) for that selected entry. The routine then loops to block 1601 to select the next entry of the dimension table.
  • FIG. 17 is a flow diagram of the load fact table routine in one embodiment. This routine adds the facts of the local data warehouse to the main data warehouse. The routine maps the local identifiers for the dimensions used in the local warehouse to the main identifiers of dimensions used in the main data warehouse. In [0051] block 1701, the routine selects the next entry in the fact table. In decision block 1702, if all the entries of the fact table have already been selected, then the routine returns, else the routine continues at block 1703. In block 1703, the routine selects the next dimension for the selected entry. In decision block 1704, if all the dimensions for the selected entry have already been selected, then the routine continues at block 1706, else the routine continues at block 1705. In block 1705, the routine retrieves the main identifier for the selected dimension and then loops to block 1703 to select the next dimension. In block 1706, the routine stores an entry in the fact table of the main data warehouse. The routine then loops to block 1701 to select the next entry in the fact table.
  • FIG. 18 is a flow diagram illustrating the identify user aliases routine in one embodiment. This routine tracks the different user identifiers as a user switches from one web site to another. In particular, the routine maps the user identifiers used by a referrer web site to the user identifiers used by the referred-to web site. In this way, the same user can be tracked even though different web sites use different identifiers for that user. This routine may be invoked as part of the parsing of the log files. In [0052] decision block 1801, if the log entry indicates a referrer web site, then the routine continues at block 1802, else the routine returns. In block 1802, the routine identifies the user identifier for the referrer web site. In block 1803, the routine creates a mapping between the referrer user identifier and the referred-to user identifier. The routine then returns.
  • From the above description it will be appreciated that although specific embodiments of the technology have been described for purposes of illustration, various modifications may be made without deviating from the spirit and scope of the invention. For example, the processing of the parser may be performed by the data collection component before sending the data to the data warehouse server. Accordingly, the invention is not limited except by the appended claims. [0053]

Claims (49)

1. A method of processing data before updating a database based on the processed data, the database having a main table with a main identifier for each entry in the main table, the method comprising:
identifying an entry that should be in the main table;
generating a local identifier for the entry;
adding the entry with the local identifier to a local table;
generating information to be stored in the database that references the entry in the local table using the local identifier; and
after generating the information, generating a main identifier for the entry;
adding an entry with the main identifier to the main table; and
storing the generated information in the database with the local identifier replaced with the main identifier.
2. The method of claim 1 wherein the local table is stored in main memory.
3. The method of claim 1 wherein the table is a dimension table of the database.
4. The method of claim 1 wherein the generated information is stored in a fact table of the database.
5. The method of claim 1 wherein the local identifier is generated based on key information.
6. The method of claim 5 including generating a hash value based on the key information and storing the hash value in the entry of the local table.
7. The method of claim 6 including using the stored hash value to locate entries in the main table.
8. The method of claim 1 wherein the processed data relates to navigation information of a web site.
9. The method of claim 1 wherein the processed data is click stream data.
10. A method in a computer system for parsing information before updating data in a main database, the main database having fact tables and dimension tables, the method comprising:
creating a fact table and a dimension table corresponding to a fact table and dimension table of the main database;
identifying from the information entries for the created fact table and dimension table;
storing the identified entries in the created fact table and dimension table; and
analyzing the entries stored in the created fact table and dimension table.
11. The method of claim 10 wherein the parsed information relates to user interactions with web pages.
12. The method of claim 10 including updating data in the main database based on the entries in the created fact table and dimension table.
13. The method of claim 10 wherein the created fact table and dimension table are stored in main memory.
14. The method of claim 10 wherein the created fact and dimension table are stored in non-persistent memory.
15. The method of claim 10 wherein the created fact and dimension table are stored in temporary memory.
16. The method of claim 10 wherein the created fact and dimension table are destroyed after information in the main database is updated based on the created fact and dimension table.
17. The method of claim 10 wherein the data in the main database is updated based on an ordering of fact and dimension tables.
18. A method in a computer system for identifying navigation paths through web pages based on user navigation information, the method comprising:
analyzing the user navigation information to identify entries associated with the same user; and
for each user,
identifying the web pages that are accessed by the user; and
storing an indication of the identified web pages as a navigation path in a persistent database.
19. The method of claim 18 wherein a sequence of identified web pages for a user is designated as a session.
20. The method of claim 19 wherein a session is delimited by time between access of web pages by the user.
21. A method in a computer for processing navigation information for web pages, the method comprising:
selecting an entry of the navigation information;
identifying a uniform resource identifier of the selected entry; and
when the identified uniform resource identifier satisfies a match criterion, storing in a persistent database an indication that the entry matches the criterion.
22. The method of claim 21 wherein criterion indicates that the entry is for a web pages of a certain category.
23. Tile method of claim 21 wherein the criterion identifies an event.
24. The method of claim 21 wherein the criterion specifies a type of web page.
25. A method in a computer system for identifying a user who accesses a web page, the method comprising:
providing an indication of a request for a second web page, the request including second information identifying a user who requested the second web page and first information identifying a user who requested a first web page that included a reference to the second web page; and
indicating the first information and the second information identify the same user.
26. The method of claim 25 wherein the first information is included as information of a referrer.
27. The method of claim 25 including storing a mapping of the first information to the second information.
28. The method of claim 27 including checking the mapping to determine whether user information corresponds to a user identified with other information.
29. The method of claim 25 wherein the first information and second information are provided by different web domains.
30. A method in a computer system for identifying high-level events from low-level events, the method comprising:
providing a plurality of event definitions that map low-level events to high-level events; and
for each low-level event, determining whether the low-level event matches a provided event definition; and
when a low-level event matches an event definition, persistently storing an indication of the high-level event associated with the matching event definition.
31. The method of claim 30 wherein the low-level events are navigation events.
32. The method of claim 30 wherein the low-level events are derived from click stream information.
33. The method of claim 30 wherein the indication of the high-level event is persistently stored in a data warehouse.
34. A method in a computer system for processing click stream data, the method comprising:
receiving time synchronization information for the click stream data;
adjusting times associated with the click stream data based on the received time synchronization information; and
persistently storing the adjusted times.
35. The method of claim 34 wherein the time synchronization information is based on current time information received from a web server associated with the click stream data.
36. The method of claim 35 including sending a request to the web server for current time information.
37. The method of claim 34 wherein the time synchronization information is based on current time information of a web server and baseline information.
38. A computer-readable medium containing instructions for controlling a computer system to parse information before updating data in a main database, the main database having fact tables and dimension tables, by a method comprising:
creating fact tables and dimension tables corresponding to fact tables and dimension tables of the main database;
identifying from the information entries for the created fact tables and dimension tables;
storing, the identified entries in the created fact tables and dimension tables; and
analyzing the entries stored in the created fact tables and dimension tables.
39. The computer—readable medium of claim 38 wherein the parsed information relates to user interactions with web pages.
40. The computer-readable medium of claim 38 including updating data in the main database based on the entries in the created fact tables and dimension tables.
41. The computer-readable medium of claim 38 wherein the created fact tables and dimension tables are stored in main memory.
42. The computer-readable medium of claim 38 wherein the created fact tables and dimension tables are stored in non-persistent memory.
43. The computer-readable medium of claim 38 wherein the created fact and dimension tables are stored in temporarily memory.
44. The computer-readable medium of claim 38 wherein the created fact and dimension tables are destroyed after information in the main database is updated based on the created fact and dimension tables.
45. The computer-readable medium of claim 38 wherein the data in the main database is updated based on an ordering of fact and dimension tables.
46. A method in a computer system for processing of search strings, the method comprising:
identifying a search sting in a log file;
identifying a canonical form of the search string; and
storing the identified canonical form so that subsequent processing of the search string uses the canonical form.
47. The method of claim 46 wherein the identifying of a canonical form includes stemming of a word in the search string.
48. The method of claim 46 wherein the identifying of a canonical form includes first or last word grouping.
49. The method of claim 46 wherein the identifying of a canonical form includes removal of spaces between words.
US10/320,171 2000-07-11 2002-12-16 Method and system for parsing navigation information Abandoned US20030120670A1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US10/320,171 US20030120670A1 (en) 2000-07-11 2002-12-16 Method and system for parsing navigation information

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US09/613,847 US6785666B1 (en) 2000-07-11 2000-07-11 Method and system for parsing navigation information
US10/320,171 US20030120670A1 (en) 2000-07-11 2002-12-16 Method and system for parsing navigation information

Related Parent Applications (1)

Application Number Title Priority Date Filing Date
US09/613,847 Division US6785666B1 (en) 2000-07-11 2000-07-11 Method and system for parsing navigation information

Publications (1)

Publication Number Publication Date
US20030120670A1 true US20030120670A1 (en) 2003-06-26

Family

ID=24458912

Family Applications (8)

Application Number Title Priority Date Filing Date
US09/613,847 Expired - Lifetime US6785666B1 (en) 2000-07-11 2000-07-11 Method and system for parsing navigation information
US10/005,224 Expired - Lifetime US7117193B1 (en) 2000-07-11 2001-12-05 Parsing navigation information to identify occurrences of events of interest
US10/007,646 Expired - Lifetime US6917972B1 (en) 2000-07-11 2001-12-05 Parsing navigation information to identify occurrences corresponding to defined categories
US10/320,288 Abandoned US20030131081A1 (en) 2000-07-11 2002-12-16 Method and system for parsing navigation information
US10/320,091 Abandoned US20030126111A1 (en) 2000-07-11 2002-12-16 Method and system for parsing navigation information
US10/320,092 Expired - Lifetime US6873981B2 (en) 2000-07-11 2002-12-16 Method and system for parsing navigation information
US10/320,171 Abandoned US20030120670A1 (en) 2000-07-11 2002-12-16 Method and system for parsing navigation information
US11/495,089 Expired - Lifetime US7464122B1 (en) 2000-07-11 2006-07-27 Parsing navigation information to identify occurrences of events of interest

Family Applications Before (6)

Application Number Title Priority Date Filing Date
US09/613,847 Expired - Lifetime US6785666B1 (en) 2000-07-11 2000-07-11 Method and system for parsing navigation information
US10/005,224 Expired - Lifetime US7117193B1 (en) 2000-07-11 2001-12-05 Parsing navigation information to identify occurrences of events of interest
US10/007,646 Expired - Lifetime US6917972B1 (en) 2000-07-11 2001-12-05 Parsing navigation information to identify occurrences corresponding to defined categories
US10/320,288 Abandoned US20030131081A1 (en) 2000-07-11 2002-12-16 Method and system for parsing navigation information
US10/320,091 Abandoned US20030126111A1 (en) 2000-07-11 2002-12-16 Method and system for parsing navigation information
US10/320,092 Expired - Lifetime US6873981B2 (en) 2000-07-11 2002-12-16 Method and system for parsing navigation information

Family Applications After (1)

Application Number Title Priority Date Filing Date
US11/495,089 Expired - Lifetime US7464122B1 (en) 2000-07-11 2006-07-27 Parsing navigation information to identify occurrences of events of interest

Country Status (1)

Country Link
US (8) US6785666B1 (en)

Cited By (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20050267889A1 (en) * 2004-02-09 2005-12-01 Coremetrics, Inc. System and method of managing software product-line customizations
US20060080271A1 (en) * 2004-10-08 2006-04-13 Jeff Williams Dynamic reporting tool for a fact warehouse, a fact contained within the fact warehouse, and a method for refreshing the fact warehouse
EP1716508A1 (en) * 2004-01-19 2006-11-02 Nigel Hamilton Method and system for recording search trails across one or more search engines in a communications network
US20080059310A1 (en) * 2006-09-05 2008-03-06 Thomas Publishing Company Marketing method and system using domain knowledge
US20110264710A1 (en) * 2010-04-27 2011-10-27 International Business Machines Corporation Monitoring and reporting productivity in enterprise environment
US20110307509A1 (en) * 2010-03-23 2011-12-15 Google Inc. Conversion Path Performance Measures And Reports
US8521774B1 (en) 2010-08-20 2013-08-27 Google Inc. Dynamically generating pre-aggregated datasets
US20140214778A1 (en) * 2006-02-17 2014-07-31 Google Inc. Entity Normalization Via Name Normalization
CN105718565A (en) * 2016-01-20 2016-06-29 北京京东尚科信息技术有限公司 Data warehouse model construction method and construction apparatus
CN107241220A (en) * 2017-05-26 2017-10-10 京信通信系统(中国)有限公司 The aging of product monitors automatic analysis method and system
US11055282B1 (en) * 2020-03-31 2021-07-06 Atlassian Pty Ltd. Translating graph queries into efficient network protocol requests
US11093985B2 (en) * 2018-09-25 2021-08-17 Valideck International System, devices, and methods for acquiring and verifying online information

Families Citing this family (153)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7543078B2 (en) * 1999-12-31 2009-06-02 Subdomain Identity Partners Individuals' URL identity exchange and communications
US7266510B1 (en) * 2000-09-01 2007-09-04 International Business Machines Corporation Method for graphically representing clickstream data of a shopping session on a network with a parallel coordinate system
US8255791B2 (en) 2000-11-29 2012-08-28 Dov Koren Collaborative, flexible, interactive real-time displays
JP3276145B1 (en) * 2000-12-26 2002-04-22 株式会社ログ・モンスターズ Internet advertising method and system
US7447182B2 (en) * 2001-04-06 2008-11-04 Nortel Networks Limited Discovering an address of a name server
US7062551B2 (en) * 2001-05-24 2006-06-13 International Business Machines Corporation Method and apparatus to solve compatibility between heterogeneous web server access logs formats
US6963874B2 (en) * 2002-01-09 2005-11-08 Digital River, Inc. Web-site performance analysis system and method utilizing web-site traversal counters and histograms
US20030079136A1 (en) * 2001-08-21 2003-04-24 Emmanuel Ericta Security framework
US8429688B1 (en) * 2001-09-19 2013-04-23 At&T Intellectual Property I, L.P. Broadcast programming guide
US10255362B2 (en) * 2001-11-28 2019-04-09 Benjamin Rodefer Method for performing a search, and computer program product and user interface for same
US7107338B1 (en) * 2001-12-05 2006-09-12 Revenue Science, Inc. Parsing navigation information to identify interactions based on the times of their occurrences
US20030131106A1 (en) * 2002-01-09 2003-07-10 Stephane Kasriel Web-page performance toolbar
US7631035B2 (en) * 2002-01-09 2009-12-08 Digital River, Inc. Path-analysis toolbar
US20030131097A1 (en) * 2002-01-09 2003-07-10 Stephane Kasriel Interactive path analysis
US20030128231A1 (en) * 2002-01-09 2003-07-10 Stephane Kasriel Dynamic path analysis
US20030131093A1 (en) * 2002-01-09 2003-07-10 International Business Machines Corporation System for generating usage data in a distributed information processing environment and method therefor
US20030163586A1 (en) * 2002-02-27 2003-08-28 Steve Schnetzler Server persistence using a URL identifier
JP2003337644A (en) * 2002-03-14 2003-11-28 Sony Corp Electronic device, program, program providing device and recording medium
US20030204490A1 (en) * 2002-04-24 2003-10-30 Stephane Kasriel Web-page collaboration system
US20030202009A1 (en) * 2002-04-24 2003-10-30 Stephane Kasriel Integration toolbar
US7254632B2 (en) * 2002-04-26 2007-08-07 P-Cube Ltd. Apparatus and method for pattern matching in text based protocol
US7613994B2 (en) 2002-05-29 2009-11-03 International Business Machines Corporation Document handling in a web application
US20030233343A1 (en) * 2002-06-18 2003-12-18 International Business Machines Corporation System and method for generating custom business reports for a WEB application
US8219709B2 (en) * 2002-07-05 2012-07-10 Carolyn J Hughes Method for internet name sharing
EP1533958A4 (en) * 2002-07-22 2007-09-05 Fujitsu Ltd Electronic mail server; electronic mail delivery relaying method; and computer program
US8122153B2 (en) * 2002-07-31 2012-02-21 Subdomain Identity Partners Individuals' URL identity exchange and communications
US7640267B2 (en) 2002-11-20 2009-12-29 Radar Networks, Inc. Methods and systems for managing entities in a computing device using semantic objects
US7584208B2 (en) 2002-11-20 2009-09-01 Radar Networks, Inc. Methods and systems for managing offers and requests in a network
US7035838B2 (en) * 2002-12-06 2006-04-25 General Electric Company Methods and systems for organizing information stored within a computer network-based system
US20040158619A1 (en) * 2003-02-10 2004-08-12 Claus Pedersen Method and apparatus for provisioning content
US20040193612A1 (en) * 2003-03-31 2004-09-30 Chang William I. System and method for testing, monitoring, and tracking distributed transactions using a search engine
US20040254919A1 (en) * 2003-06-13 2004-12-16 Microsoft Corporation Log parser
US8112458B1 (en) 2003-06-17 2012-02-07 AudienceScience Inc. User segmentation user interface
US7966333B1 (en) * 2003-06-17 2011-06-21 AudienceScience Inc. User segment population techniques
US20050027695A1 (en) * 2003-07-29 2005-02-03 John Peter Thomas Identifying files suitable for attaching to a text message
US9928522B2 (en) 2003-08-01 2018-03-27 Oath (Americas) Inc. Audience matching network with performance factoring and revenue allocation
US9118812B2 (en) * 2003-08-01 2015-08-25 Advertising.Com Llc Audience server
US8150732B2 (en) * 2003-08-01 2012-04-03 Tacoda Llc Audience targeting system with segment management
US8464290B2 (en) * 2003-08-01 2013-06-11 Tacoda, Inc. Network for matching an audience with deliverable content
US20050125290A1 (en) * 2003-08-01 2005-06-09 Gil Beyda Audience targeting system with profile synchronization
US9117217B2 (en) 2003-08-01 2015-08-25 Advertising.Com Llc Audience targeting with universal profile synchronization
US7805332B2 (en) 2003-08-01 2010-09-28 AOL, Inc. System and method for segmenting and targeting audience members
US8094804B2 (en) 2003-09-26 2012-01-10 Avaya Inc. Method and apparatus for assessing the status of work waiting for service
US20050114321A1 (en) * 2003-11-26 2005-05-26 Destefano Jason M. Method and apparatus for storing and reporting summarized log data
US20050114505A1 (en) 2003-11-26 2005-05-26 Destefano Jason M. Method and apparatus for retrieving and combining summarized log data in a distributed log data processing system
US7599939B2 (en) * 2003-11-26 2009-10-06 Loglogic, Inc. System and method for storing raw log data
US8234256B2 (en) 2003-11-26 2012-07-31 Loglogic, Inc. System and method for parsing, summarizing and reporting log data
US20050114706A1 (en) * 2003-11-26 2005-05-26 Destefano Jason Michael System and method for the collection and transmission of log data over a wide area network
US7617531B1 (en) * 2004-02-18 2009-11-10 Citrix Systems, Inc. Inferencing data types of message components
US7433876B2 (en) * 2004-02-23 2008-10-07 Radar Networks, Inc. Semantic web portal and platform
US7953859B1 (en) 2004-03-31 2011-05-31 Avaya Inc. Data model of participation in multi-channel and multi-party contacts
US7734032B1 (en) 2004-03-31 2010-06-08 Avaya Inc. Contact center and method for tracking and acting on one and done customer contacts
US8000989B1 (en) 2004-03-31 2011-08-16 Avaya Inc. Using true value in routing work items to resources
US8868670B2 (en) * 2004-04-27 2014-10-21 Avaya Inc. Method and apparatus for summarizing one or more text messages using indicative summaries
US7337370B2 (en) * 2004-05-28 2008-02-26 International Business Machines Corporation System and method for capturing significant events at web portlets
US7609846B2 (en) * 2004-07-13 2009-10-27 Eastman Kodak Company Matching of digital images to acquisition devices
US8738412B2 (en) 2004-07-13 2014-05-27 Avaya Inc. Method and apparatus for supporting individualized selection rules for resource allocation
US20060020510A1 (en) * 2004-07-20 2006-01-26 Vest Herb D Method for improved targeting of online advertisements
US20070271145A1 (en) * 2004-07-20 2007-11-22 Vest Herb D Consolidated System for Managing Internet Ads
US8234141B1 (en) 2004-09-27 2012-07-31 Avaya Inc. Dynamic work assignment strategies based on multiple aspects of agent proficiency
US7949121B1 (en) 2004-09-27 2011-05-24 Avaya Inc. Method and apparatus for the simultaneous delivery of multiple contacts to an agent
US7580922B2 (en) * 2005-01-04 2009-08-25 International Business Machines Corporation Methods for relating data in healthcare databases
JP4720213B2 (en) * 2005-02-28 2011-07-13 富士通株式会社 Analysis support program, apparatus and method
US7809752B1 (en) 2005-04-14 2010-10-05 AudienceScience Inc. Representing user behavior information
US7676467B1 (en) 2005-04-14 2010-03-09 AudienceScience Inc. User segment population techniques
US8316446B1 (en) 2005-04-22 2012-11-20 Blue Coat Systems, Inc. Methods and apparatus for blocking unwanted software downloads
US7809127B2 (en) 2005-05-26 2010-10-05 Avaya Inc. Method for discovering problem agent behaviors
US8330759B1 (en) 2005-06-30 2012-12-11 AudienceScience Inc. Defining one or more used segments based upon extent of completion of a process
US20070033233A1 (en) * 2005-08-05 2007-02-08 Hwang Min J Log management system and method of using the same
US7779042B1 (en) 2005-08-08 2010-08-17 Avaya Inc. Deferred control of surrogate key generation in a distributed processing architecture
US7822587B1 (en) 2005-10-03 2010-10-26 Avaya Inc. Hybrid database architecture for both maintaining and relaxing type 2 data entity behavior
GB2446723A (en) * 2005-10-06 2008-08-20 Avaya Tech Llc Database data extensions
US7752230B2 (en) * 2005-10-06 2010-07-06 Avaya Inc. Data extensibility using external database tables
US7787609B1 (en) 2005-10-06 2010-08-31 Avaya Inc. Prioritized service delivery based on presence and availability of interruptible enterprise resources with skills
US20070192461A1 (en) * 2005-11-03 2007-08-16 Robert Reich System and method for dynamically generating and managing an online context-driven interactive social network
US20070106746A1 (en) * 2005-11-04 2007-05-10 Sijian Zhang Method and System Showing The Most Meaningful Page From A Book Or Content To Customers As The First Page
US7565366B2 (en) * 2005-12-14 2009-07-21 Microsoft Corporation Variable rate sampling for sequence analysis
US7571151B1 (en) 2005-12-15 2009-08-04 Gneiss Software, Inc. Data analysis tool for analyzing data stored in multiple text files
US8737173B2 (en) 2006-02-24 2014-05-27 Avaya Inc. Date and time dimensions for contact center reporting in arbitrary international time zones
US20070208784A1 (en) * 2006-03-03 2007-09-06 Beeston Ralph T Parsing computer system logging information collected by common logging
US20070239799A1 (en) * 2006-03-29 2007-10-11 Anirudh Modi Analyzing log files
WO2008021832A2 (en) 2006-08-09 2008-02-21 Radar Networks, Inc. Harvesting data from page
US7936867B1 (en) 2006-08-15 2011-05-03 Avaya Inc. Multi-service request within a contact center
US8391463B1 (en) 2006-09-01 2013-03-05 Avaya Inc. Method and apparatus for identifying related contacts
US8938063B1 (en) 2006-09-07 2015-01-20 Avaya Inc. Contact center service monitoring and correcting
US8811597B1 (en) 2006-09-07 2014-08-19 Avaya Inc. Contact center performance prediction
SE0950005L (en) * 2006-11-10 2008-05-11 Densitech Ab Providing security in relation to mobile terminals
US20090055369A1 (en) * 2007-02-01 2009-02-26 Jonathan Phillips System, method and apparatus for implementing dynamic community formation processes within an online context-driven interactive social network
US7917507B2 (en) * 2007-02-12 2011-03-29 Microsoft Corporation Web data usage platform
US8429185B2 (en) 2007-02-12 2013-04-23 Microsoft Corporation Using structured data for online research
US7822755B2 (en) * 2007-03-06 2010-10-26 Yahoo! Inc. Methods of processing and segmenting web usage information
US20080228773A1 (en) * 2007-03-14 2008-09-18 Hand Held Products, Inc. Apparatus and method for data input device
US7912825B2 (en) * 2007-06-21 2011-03-22 Microsoft Corporation Configurable plug-in architecture for manipulating XML-formatted information
US8381193B2 (en) * 2007-09-06 2013-02-19 International Business Machines Corporation Apparatus, system, and method for visual log analysis
US20090076887A1 (en) 2007-09-16 2009-03-19 Nova Spivack System And Method Of Collecting Market-Related Data Via A Web-Based Networking Environment
US8504534B1 (en) 2007-09-26 2013-08-06 Avaya Inc. Database structures and administration techniques for generalized localization of database items
US8306967B2 (en) 2007-10-02 2012-11-06 Loglogic, Inc. Searching for associated events in log data
US8244667B1 (en) * 2007-10-18 2012-08-14 Google Inc. Querying multidimensional data with independent fact and dimension pipelines combined at query time
US7925694B2 (en) 2007-10-19 2011-04-12 Citrix Systems, Inc. Systems and methods for managing cookies via HTTP content layer
US9552356B1 (en) * 2007-12-21 2017-01-24 Amazon Technologies, Inc. Merging client-side and server-side logs
US8856182B2 (en) 2008-01-25 2014-10-07 Avaya Inc. Report database dependency tracing through business intelligence metadata
US8090877B2 (en) 2008-01-26 2012-01-03 Citrix Systems, Inc. Systems and methods for fine grain policy driven cookie proxying
US20090271497A1 (en) * 2008-04-29 2009-10-29 Lee Roberts Affiliate Program Redirection System
US8244224B2 (en) * 2008-11-20 2012-08-14 Research In Motion Limited Providing customized information to a user based on identifying a trend
US8260926B2 (en) 2008-11-25 2012-09-04 Citrix Systems, Inc. Systems and methods for GSLB site persistence
CN101499098B (en) * 2009-03-04 2012-07-11 阿里巴巴集团控股有限公司 Web page assessed value confirming and employing method and system
US8200617B2 (en) 2009-04-15 2012-06-12 Evri, Inc. Automatic mapping of a location identifier pattern of an object to a semantic type using object metadata
US10628847B2 (en) 2009-04-15 2020-04-21 Fiver Llc Search-enhanced semantic advertising
WO2010120929A2 (en) * 2009-04-15 2010-10-21 Evri Inc. Generating user-customized search results and building a semantics-enhanced search engine
US8862579B2 (en) 2009-04-15 2014-10-14 Vcvc Iii Llc Search and search optimization using a pattern of a location identifier
CN102301652A (en) * 2009-04-27 2011-12-28 国际商业机器公司 Message switching
JP5338461B2 (en) * 2009-05-01 2013-11-13 ブラザー工業株式会社 Management device, information generation program, and information generation method
US8549019B2 (en) * 2009-05-26 2013-10-01 Google Inc. Dynamically generating aggregate tables
US9305105B2 (en) * 2009-05-26 2016-04-05 Google Inc. System and method for aggregating analytics data
US20100306363A1 (en) * 2009-05-26 2010-12-02 Erwien Saputra Determining completion of a web server download session at a database server
US8751544B2 (en) * 2009-09-02 2014-06-10 Google Inc. Method and system for pivoting a multidimensional dataset
US8412719B1 (en) 2009-09-02 2013-04-02 Google Inc. Method and system for segmenting a multidimensional dataset
US8565386B2 (en) 2009-09-29 2013-10-22 Avaya Inc. Automatic configuration of soft phones that are usable in conjunction with special-purpose endpoints
US8583584B2 (en) * 2009-10-20 2013-11-12 Google Inc. Method and system for using web analytics data for detecting anomalies
US20110119100A1 (en) * 2009-10-20 2011-05-19 Jan Matthias Ruhl Method and System for Displaying Anomalies in Time Series Data
US9516069B2 (en) 2009-11-17 2016-12-06 Avaya Inc. Packet headers as a trigger for automatic activation of special-purpose softphone applications
US8286258B2 (en) * 2009-12-04 2012-10-09 Institute For Information Industry Monitor method and monitor apparatus for monitoring data of hardware
US8108377B2 (en) * 2009-12-15 2012-01-31 Facebook, Inc. Predictive resource identification and phased delivery of structured documents
US8768926B2 (en) * 2010-01-05 2014-07-01 Yahoo! Inc. Techniques for categorizing web pages
US20150213484A1 (en) * 2010-03-22 2015-07-30 Ashok Amara System and method for tracking related events
US8645384B1 (en) * 2010-05-05 2014-02-04 Google Inc. Updating taxonomy based on webpage
US8060414B1 (en) * 2010-05-25 2011-11-15 Amazon Technologies, Inc. Encoding and decoding of product properties in a uniform resource locator
US8712999B2 (en) * 2010-06-10 2014-04-29 Aol Inc. Systems and methods for online search recirculation and query categorization
US8954451B2 (en) * 2010-06-30 2015-02-10 Hewlett-Packard Development Company, L.P. Selecting microblog entries based on web pages, via path similarity within hierarchy of categories
US8799310B2 (en) * 2010-07-23 2014-08-05 Hewlett-Packard Development Company, L.P. Method and system for processing a uniform resource locator
US20120078925A1 (en) * 2010-09-27 2012-03-29 International Business Machines Corporation Searching within log files
US8775606B2 (en) * 2010-12-02 2014-07-08 Yahoo! Inc. System and method for counting network users
US8521655B2 (en) * 2011-06-06 2013-08-27 Bizequity Llc Engine, system and method for providing cloud-based business intelligence
US20130067115A1 (en) * 2011-09-12 2013-03-14 Isaac Omar Lapanc Method And System For Mapping Domain Prefixes To Qualified URLs
US9020463B2 (en) * 2011-12-29 2015-04-28 The Nielsen Company (Us), Llc Systems, methods, apparatus, and articles of manufacture to measure mobile device usage
US9659095B2 (en) * 2012-03-04 2017-05-23 International Business Machines Corporation Managing search-engine-optimization content in web pages
US9507825B2 (en) 2012-09-28 2016-11-29 Oracle International Corporation Techniques for partition pruning based on aggregated zone map information
US8996544B2 (en) 2012-09-28 2015-03-31 Oracle International Corporation Pruning disk blocks of a clustered table in a relational database management system
US9430550B2 (en) 2012-09-28 2016-08-30 Oracle International Corporation Clustering a table in a relational database management system
US10642837B2 (en) 2013-03-15 2020-05-05 Oracle International Corporation Relocating derived cache during data rebalance to maintain application performance
US20140324422A1 (en) * 2013-04-18 2014-10-30 WTF Technology Partners, Inc. Synchronous audio distribution to portable computing devices
US10223453B2 (en) * 2015-02-18 2019-03-05 Ubunifu, LLC Dynamic search set creation in a search engine
WO2016137426A1 (en) * 2015-02-23 2016-09-01 Hewlett Packard Enterprise Development Lp Navigation menu based on crowd data
EP3403187A4 (en) * 2016-01-14 2019-07-31 Sumo Logic Single click delta analysis
CN107547409B (en) * 2016-06-24 2020-12-25 华为技术有限公司 Route searching method and device and routing equipment
US11282091B2 (en) * 2016-09-30 2022-03-22 Transitiv, Inc. Systems, methods, and devices for dynamic page feed management
US11127044B2 (en) 2016-12-22 2021-09-21 The Nielsen Company (Us), Llc Methods and apparatus to expand panelist enrollment
US10942937B2 (en) 2017-04-14 2021-03-09 Seagate Technology Llc Data mining systems
CA3062634A1 (en) * 2017-05-10 2018-11-15 Embee Mobile, Inc. System and method for the capture of mobile behavior, usage, or content exposure
KR102408476B1 (en) * 2017-07-10 2022-06-14 십일번가 주식회사 Method for predicing purchase probability based on behavior sequence of user and apparatus therefor
US11086876B2 (en) 2017-09-29 2021-08-10 Oracle International Corporation Storing derived summaries on persistent memory of a storage device
US11232166B2 (en) * 2019-01-17 2022-01-25 Salesforce.Com, Inc. Networked page access and addressing based on partial categorization indexing
CN111651701A (en) * 2020-06-05 2020-09-11 厦门理工学院 Internet of things big data window browsing system

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6006225A (en) * 1998-06-15 1999-12-21 Amazon.Com Refining search queries by the suggestion of correlated terms from prior searches
US6157618A (en) * 1999-01-26 2000-12-05 Microsoft Corporation Distributed internet user experience monitoring system
US6633316B1 (en) * 1999-05-13 2003-10-14 International Business Machines Corporation Method and apparatus for implementing direct link selection of cached, previously visited links in nested web pages

Family Cites Families (116)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
DE59108895D1 (en) * 1991-12-18 1998-01-08 Siemens Ag Procedures for keeping the time in computer networks
US5557686A (en) * 1993-01-13 1996-09-17 University Of Alabama Method and apparatus for verification of a computer user's identification, based on keystroke characteristics
US5619709A (en) * 1993-09-20 1997-04-08 Hnc, Inc. System and method of context vector generation and retrieval
US5950173A (en) * 1996-10-25 1999-09-07 Ipf, Inc. System and method for delivering consumer product related information to consumers within retail environments using internet-based information servers and sales agents
US5715399A (en) * 1995-03-30 1998-02-03 Amazon.Com, Inc. Secure method and system for communicating a list of credit card numbers over a non-secure network
US5742762A (en) * 1995-05-19 1998-04-21 Telogy Networks, Inc. Network management gateway
US6018342A (en) * 1995-07-03 2000-01-25 Sun Microsystems, Inc. Automatically generated content-based history mechanism
US5712979A (en) * 1995-09-20 1998-01-27 Infonautics Corporation Method and apparatus for attaching navigational history information to universal resource locator links on a world wide web page
US5930786A (en) 1995-10-20 1999-07-27 Ncr Corporation Method and apparatus for providing shared data to a requesting client
US5809478A (en) * 1995-12-08 1998-09-15 Allstate Insurance Company Method for accessing and evaluating information for processing an application for insurance
US6016482A (en) * 1996-01-11 2000-01-18 Merrill Lynch & Co., Inc. Enhanced collateralized funding processor
US5821937A (en) * 1996-02-23 1998-10-13 Netsuite Development, L.P. Computer method for updating a network design
US5809235A (en) * 1996-03-08 1998-09-15 International Business Machines Corporation Object oriented network event management framework
US5894554A (en) * 1996-04-23 1999-04-13 Infospinner, Inc. System for managing dynamic web page generation requests by intercepting request at web server and routing to page server thereby releasing web server to process other requests
US5848396A (en) * 1996-04-26 1998-12-08 Freedom Of Information, Inc. Method and apparatus for determining behavioral profile of a computer user
US5864871A (en) 1996-06-04 1999-01-26 Multex Systems Information delivery system and method including on-line entitlements
US5796633A (en) 1996-07-12 1998-08-18 Electronic Data Systems Corporation Method and system for performance monitoring in computer networks
US5787425A (en) 1996-10-01 1998-07-28 International Business Machines Corporation Object-oriented data mining framework mechanism
US5983200A (en) * 1996-10-09 1999-11-09 Slotznick; Benjamin Intelligent agent for executing delegated tasks
US5974572A (en) * 1996-10-15 1999-10-26 Mercury Interactive Corporation Software system and methods for generating a load test using a server access log
US5870559A (en) * 1996-10-15 1999-02-09 Mercury Interactive Software system and associated methods for facilitating the analysis and management of web sites
US6983478B1 (en) * 2000-02-01 2006-01-03 Bellsouth Intellectual Property Corporation Method and system for tracking network use
US6012087A (en) * 1997-01-14 2000-01-04 Netmind Technologies, Inc. Unique-change detection of dynamic web pages using history tables of signatures
JPH10214284A (en) * 1997-01-30 1998-08-11 Victor Co Of Japan Ltd On-line shopping system and server for the same
US6112238A (en) 1997-02-14 2000-08-29 Webtrends Corporation System and method for analyzing remote traffic data in a distributed computing environment
US5899992A (en) * 1997-02-14 1999-05-04 International Business Machines Corporation Scalable set oriented classifier
AUPO525497A0 (en) * 1997-02-21 1997-03-20 Mills, Dudley John Network-based classified information systems
US6026433A (en) * 1997-03-17 2000-02-15 Silicon Graphics, Inc. Method of creating and editing a web site in a client-server environment using customizable web site templates
US6023507A (en) * 1997-03-17 2000-02-08 Sun Microsystems, Inc. Automatic remote computer monitoring system
US5945989A (en) * 1997-03-25 1999-08-31 Premiere Communications, Inc. Method and apparatus for adding and altering content on websites
US6424974B1 (en) * 1997-03-31 2002-07-23 International Business Machines Corporation Storing P-code in a database
US6144944A (en) * 1997-04-24 2000-11-07 Imgis, Inc. Computer system for efficiently selecting and providing information
US6553416B1 (en) 1997-05-13 2003-04-22 Micron Technology, Inc. Managing computer system alerts
US6094672A (en) * 1997-05-19 2000-07-25 Novell, Inc. Method and system for time synchronization management
US6012086A (en) * 1997-06-24 2000-01-04 Sony Corporation Internet event timer recording for video and/or audio
US6029141A (en) * 1997-06-27 2000-02-22 Amazon.Com, Inc. Internet-based customer referral system
US5905985A (en) * 1997-06-30 1999-05-18 International Business Machines Corporation Relational database modifications based on multi-dimensional database modifications
US6205447B1 (en) * 1997-06-30 2001-03-20 International Business Machines Corporation Relational database management of multi-dimensional data
US5987135A (en) 1997-07-25 1999-11-16 Prc Inc. System and method for controlling and monitoring remote distributed processing system
US5999924A (en) * 1997-07-25 1999-12-07 Amazon.Com, Inc. Method and apparatus for producing sequenced queries
US6076092A (en) * 1997-08-19 2000-06-13 Sun Microsystems, Inc. System and process for providing improved database interfacing using query objects
US5960428A (en) * 1997-08-28 1999-09-28 International Business Machines Corporation Star/join query optimization
US6000832A (en) * 1997-09-24 1999-12-14 Microsoft Corporation Electronic online commerce card with customer generated transaction proxy number for online transactions
US6574661B1 (en) 1997-09-26 2003-06-03 Mci Communications Corporation Integrated proxy interface for web based telecommunication toll-free network management using a network manager for downloading a call routing tree to client
US6035332A (en) * 1997-10-06 2000-03-07 Ncr Corporation Method for monitoring user interactions with web pages from web server using data and command lists for maintaining information visited and issued by participants
US6108004A (en) * 1997-10-21 2000-08-22 International Business Machines Corporation GUI guide for data mining
JP3811874B2 (en) * 1997-10-28 2006-08-23 富士通株式会社 Time correction method and mobile phone terminal device used in the method
US6112181A (en) * 1997-11-06 2000-08-29 Intertrust Technologies Corporation Systems and methods for matching, selecting, narrowcasting, and/or classifying based on rights management and/or other information
US6151601A (en) 1997-11-12 2000-11-21 Ncr Corporation Computer architecture and method for collecting, analyzing and/or transforming internet and/or electronic commerce data for storage into a data storage area
US6128624A (en) * 1997-11-12 2000-10-03 Ncr Corporation Collection and integration of internet and electronic commerce data in a database during web browsing
US6151584A (en) 1997-11-20 2000-11-21 Ncr Corporation Computer architecture and method for validating and collecting and metadata and data about the internet and electronic commerce environments (data discoverer)
US6052785A (en) * 1997-11-21 2000-04-18 International Business Machines Corporation Multiple remote data access security mechanism for multitiered internet computer networks
US6289342B1 (en) * 1998-01-05 2001-09-11 Nec Research Institute, Inc. Autonomous citation indexing and literature browsing using citation context
US6078924A (en) 1998-01-30 2000-06-20 Aeneid Corporation Method and apparatus for performing data collection, interpretation and analysis, in an information platform
US6073135A (en) * 1998-03-10 2000-06-06 Alta Vista Company Connectivity server for locating linkage information between Web pages
US6199063B1 (en) * 1998-03-27 2001-03-06 Red Brick Systems, Inc. System and method for rewriting relational database queries
JP2002510088A (en) * 1998-03-27 2002-04-02 インフォミックス ソフトウェア, インコーポレイテッド Processing precomputed views
US6154738A (en) * 1998-03-27 2000-11-28 Call; Charles Gainor Methods and apparatus for disseminating product information via the internet using universal product codes
US6233618B1 (en) * 1998-03-31 2001-05-15 Content Advisor, Inc. Access control of networked data
GB2336007B (en) 1998-04-01 2003-01-29 Mitel Corp Agent-based data mining and warehousing
US6101486A (en) * 1998-04-20 2000-08-08 Nortel Networks Corporation System and method for retrieving customer information at a transaction center
US6219639B1 (en) * 1998-04-28 2001-04-17 International Business Machines Corporation Method and apparatus for recognizing identity of individuals employing synchronized biometrics
US6169997B1 (en) * 1998-04-29 2001-01-02 Ncr Corporation Method and apparatus for forming subject (context) map and presenting Internet data according to the subject map
US6714931B1 (en) * 1998-04-29 2004-03-30 Ncr Corporation Method and apparatus for forming user sessions and presenting internet data according to the user sessions
US6189004B1 (en) * 1998-05-06 2001-02-13 E. Piphany, Inc. Method and apparatus for creating a datamart and for creating a query structure for the datamart
US6212524B1 (en) * 1998-05-06 2001-04-03 E.Piphany, Inc. Method and apparatus for creating and populating a datamart
US6161103A (en) * 1998-05-06 2000-12-12 Epiphany, Inc. Method and apparatus for creating aggregates for use in a datamart
US6233341B1 (en) * 1998-05-19 2001-05-15 Visto Corporation System and method for installing and using a temporary certificate at a remote site
US6138121A (en) * 1998-05-29 2000-10-24 Hewlett-Packard Company Network management event storage and manipulation using relational database technology in a data warehouse
CA2261460A1 (en) * 1998-06-04 1999-12-04 Richard B. Mcevoy Mailing label
US6278966B1 (en) * 1998-06-18 2001-08-21 International Business Machines Corporation Method and system for emulating web site traffic to identify web site usage patterns
FR2780178B1 (en) 1998-06-18 2001-08-10 Inst Nat Rech Inf Automat METHOD FOR TRANSFORMING AND TRANSPORTING DATA BETWEEN AGENT SERVERS PRESENT ON MACHINES AND A CENTRAL AGENT SERVER PRESENT ON ANOTHER MACHINE
JP4105293B2 (en) 1998-06-30 2008-06-25 富士通株式会社 Network monitoring system, monitoring device and monitored device
US6192364B1 (en) 1998-07-24 2001-02-20 Jarg Corporation Distributed computer database system and method employing intelligent agents
US6178418B1 (en) * 1998-07-28 2001-01-23 Noetix Corporation Distributed data warehouse query and resource management system
US6317787B1 (en) * 1998-08-11 2001-11-13 Webtrends Corporation System and method for analyzing web-server log files
AU5797599A (en) 1998-08-31 2000-03-21 Aprisma Management Technologies, Inc. Method and apparatus for managing data for use by data applications
US6311205B1 (en) * 1998-10-19 2001-10-30 International Business Machines Corporation Persistent user groups on servers managed by central servers
US6289378B1 (en) * 1998-10-20 2001-09-11 Triactive Technologies, L.L.C. Web browser remote computer management system
US6192357B1 (en) * 1998-11-03 2001-02-20 Platinum Technology, Inc. Method and apparatus for optimizing query generation by selectively utilizing attributes or key values
US6360246B1 (en) * 1998-11-13 2002-03-19 The Nasdaq Stock Market, Inc. Report generation architecture for remotely generated data
US6256676B1 (en) * 1998-11-18 2001-07-03 Saga Software, Inc. Agent-adapter architecture for use in enterprise application integration systems
US6802042B2 (en) * 1999-06-01 2004-10-05 Yodlee.Com, Inc. Method and apparatus for providing calculated and solution-oriented personalized summary-reports to a user through a single user-interface
US6466970B1 (en) * 1999-01-27 2002-10-15 International Business Machines Corporation System and method for collecting and analyzing information about content requested in a network (World Wide Web) environment
US20010012346A1 (en) * 1999-01-29 2001-08-09 Alex Terry Interactive billing system utilizing a thin web client interface
US6246998B1 (en) * 1999-02-25 2001-06-12 Fujitsu Limited System and method for home grocery shopping including item categorization for efficient delivery and pick-up
US6584464B1 (en) * 1999-03-19 2003-06-24 Ask Jeeves, Inc. Grammar template query system
US6519585B1 (en) * 1999-04-27 2003-02-11 Infospace, Inc. System and method for facilitating presentation of subject categorizations for use in an on-line search query engine
US6618719B1 (en) * 1999-05-19 2003-09-09 Sybase, Inc. Database system with methodology for reusing cost-based optimization decisions
US6401077B1 (en) * 1999-05-28 2002-06-04 Network Commerce, Inc. Method and system for providing additional behavior through a web page
US6279033B1 (en) * 1999-05-28 2001-08-21 Microstrategy, Inc. System and method for asynchronous control of report generation using a network interface
US6341290B1 (en) 1999-05-28 2002-01-22 Electronic Data Systems Corporation Method and system for automating the communication of business information
US6467052B1 (en) * 1999-06-03 2002-10-15 Microsoft Corporation Method and apparatus for analyzing performance of data processing system
US6446086B1 (en) * 1999-06-30 2002-09-03 Computer Sciences Corporation System and method for logging transaction records in a computer system
US6665682B1 (en) * 1999-07-19 2003-12-16 International Business Machines Corporation Performance of table insertion by using multiple tables or multiple threads
US6408292B1 (en) * 1999-08-04 2002-06-18 Hyperroll, Israel, Ltd. Method of and system for managing multi-dimensional databases using modular-arithmetic based address data mapping processes on integer-encoded business dimensions
US6286104B1 (en) * 1999-08-04 2001-09-04 Oracle Corporation Authentication and authorization in a multi-tier relational database management system
US6532023B1 (en) * 1999-08-12 2003-03-11 International Business Machines Corporation Recording selected applet events of a user interaction sequence
US6539391B1 (en) * 1999-08-13 2003-03-25 At&T Corp. Method and system for squashing a large data set
US6449739B1 (en) * 1999-09-01 2002-09-10 Mercury Interactive Corporation Post-deployment monitoring of server performance
US7533344B1 (en) * 1999-09-10 2009-05-12 Ricoh Company, Ltd. Application unit monitoring and reporting system and method
US6792458B1 (en) * 1999-10-04 2004-09-14 Urchin Software Corporation System and method for monitoring and analyzing internet traffic
US6105020A (en) * 1999-10-11 2000-08-15 International Business Machines Corporation System and method for identifying and constructing star joins for execution by bitmap ANDing
US6484179B1 (en) * 1999-10-25 2002-11-19 Oracle Corporation Storing multidimensional data in a relational database management system
US6611846B1 (en) * 1999-10-30 2003-08-26 Medtamic Holdings Method and system for medical patient data analysis
US6798413B1 (en) * 1999-12-03 2004-09-28 Dcs, Inc. Workflow management system
US6665658B1 (en) * 2000-01-13 2003-12-16 International Business Machines Corporation System and method for automatically gathering dynamic content and resources on the world wide web by stimulating user interaction and managing session information
US6496857B1 (en) * 2000-02-08 2002-12-17 Mirror Worlds Technologies, Inc. Delivering targeted, enhanced advertisements across electronic networks
US6389510B1 (en) * 2000-04-25 2002-05-14 Lucent Technologies Inc. Method and apparatus for caching web-based information
US6678700B1 (en) * 2000-04-27 2004-01-13 General Atomics System of and method for transparent management of data objects in containers across distributed heterogenous resources
US6728769B1 (en) * 2000-05-04 2004-04-27 Sun Microsystems, Inc. Method and apparatus for providing a highly interactive transaction environment in a distributed network
US20020038388A1 (en) * 2000-09-13 2002-03-28 Netter Zvi Itzhak System and method for capture and playback of user interaction with web browser content
US6836773B2 (en) * 2000-09-28 2004-12-28 Oracle International Corporation Enterprise web mining system and method
US20020082911A1 (en) * 2000-12-22 2002-06-27 Dunn Charles L. Online revenue sharing
US20020143925A1 (en) * 2000-12-29 2002-10-03 Ncr Corporation Identifying web-log data representing a single user session
US7003565B2 (en) * 2001-04-03 2006-02-21 International Business Machines Corporation Clickstream data collection technique

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6006225A (en) * 1998-06-15 1999-12-21 Amazon.Com Refining search queries by the suggestion of correlated terms from prior searches
US6157618A (en) * 1999-01-26 2000-12-05 Microsoft Corporation Distributed internet user experience monitoring system
US6633316B1 (en) * 1999-05-13 2003-10-14 International Business Machines Corporation Method and apparatus for implementing direct link selection of cached, previously visited links in nested web pages

Cited By (26)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8572100B2 (en) 2004-01-19 2013-10-29 Nigel Hamilton Method and system for recording search trails across one or more search engines in a communications network
EP1716508A1 (en) * 2004-01-19 2006-11-02 Nigel Hamilton Method and system for recording search trails across one or more search engines in a communications network
EP1716508A4 (en) * 2004-01-19 2008-01-23 Nigel Hamilton Method and system for recording search trails across one or more search engines in a communications network
US20090030876A1 (en) * 2004-01-19 2009-01-29 Nigel Hamilton Method and system for recording search trails across one or more search engines in a communications network
US7650344B2 (en) * 2004-02-09 2010-01-19 Coremetrics, Inc. System and method of managing software product-line customizations
US20050267889A1 (en) * 2004-02-09 2005-12-01 Coremetrics, Inc. System and method of managing software product-line customizations
US20060080271A1 (en) * 2004-10-08 2006-04-13 Jeff Williams Dynamic reporting tool for a fact warehouse, a fact contained within the fact warehouse, and a method for refreshing the fact warehouse
US10223406B2 (en) 2006-02-17 2019-03-05 Google Llc Entity normalization via name normalization
US9710549B2 (en) * 2006-02-17 2017-07-18 Google Inc. Entity normalization via name normalization
US20140214778A1 (en) * 2006-02-17 2014-07-31 Google Inc. Entity Normalization Via Name Normalization
US8788321B2 (en) 2006-09-05 2014-07-22 Thomas Publishing Company Marketing method and system using domain knowledge
US20080059310A1 (en) * 2006-09-05 2008-03-06 Thomas Publishing Company Marketing method and system using domain knowledge
US8548851B2 (en) 2010-03-23 2013-10-01 Google Inc. Conversion path performance measures and reports
US11544739B1 (en) 2010-03-23 2023-01-03 Google Llc Conversion path performance measures and reports
US20110307509A1 (en) * 2010-03-23 2011-12-15 Google Inc. Conversion Path Performance Measures And Reports
US9245279B2 (en) * 2010-03-23 2016-01-26 Google Inc. Conversion path performance measures and reports
US10360586B2 (en) 2010-03-23 2019-07-23 Google Llc Conversion path performance measures and reports
US20110264710A1 (en) * 2010-04-27 2011-10-27 International Business Machines Corporation Monitoring and reporting productivity in enterprise environment
US9715669B2 (en) * 2010-04-27 2017-07-25 International Business Machines Corporation Monitoring and reporting productivity in enterprise environment
US10216805B1 (en) 2010-08-20 2019-02-26 Google Llc Dynamically generating pre-aggregated datasets
US9430519B1 (en) 2010-08-20 2016-08-30 Google Inc. Dynamically generating pre-aggregated datasets
US8521774B1 (en) 2010-08-20 2013-08-27 Google Inc. Dynamically generating pre-aggregated datasets
CN105718565A (en) * 2016-01-20 2016-06-29 北京京东尚科信息技术有限公司 Data warehouse model construction method and construction apparatus
CN107241220A (en) * 2017-05-26 2017-10-10 京信通信系统(中国)有限公司 The aging of product monitors automatic analysis method and system
US11093985B2 (en) * 2018-09-25 2021-08-17 Valideck International System, devices, and methods for acquiring and verifying online information
US11055282B1 (en) * 2020-03-31 2021-07-06 Atlassian Pty Ltd. Translating graph queries into efficient network protocol requests

Also Published As

Publication number Publication date
US20030131081A1 (en) 2003-07-10
US7117193B1 (en) 2006-10-03
US6785666B1 (en) 2004-08-31
US7464122B1 (en) 2008-12-09
US6873981B2 (en) 2005-03-29
US20030144988A1 (en) 2003-07-31
US20030126111A1 (en) 2003-07-03
US6917972B1 (en) 2005-07-12

Similar Documents

Publication Publication Date Title
US6873981B2 (en) Method and system for parsing navigation information
US7035925B1 (en) Parsing navigation information to identify interactions based on the times of their occurrences
US7949702B2 (en) Method and apparatus for synchronizing cookies across multiple client machines
US7752394B1 (en) Method and system for an extensible caching framework
Eirinaki et al. Web mining for web personalization
US7698281B2 (en) Use of extensible markup language in a system and method for influencing a position on a search result list generated by a computer network search engine
US6094657A (en) Apparatus and method for dynamic meta-tagging of compound documents
US6286043B1 (en) User profile management in the presence of dynamic pages using content templates
EP1315114B1 (en) System and method of presenting channelized data
US6993529B1 (en) Importing data using metadata
US5987454A (en) Method and apparatus for selectively augmenting retrieved text, numbers, maps, charts, still pictures and/or graphics, moving pictures and/or graphics and audio information from a network resource
US6959319B1 (en) System and method for automatically personalizing web portals and web services based upon usage history
US6564251B2 (en) Scalable computing system for presenting customized aggregation of information
US20030191737A1 (en) Indexing system and method
US20040260680A1 (en) Personalized indexing and searching for information in a distributed data processing system
US7631263B2 (en) Methods, systems, and computer program products for characterizing links to resources not activated
US20100185599A1 (en) Exposing rich internet application content to search engines
US20040249824A1 (en) Semantics-bases indexing in a distributed data processing system
US20040205554A1 (en) Systems and methods for accessing multiple internal information sources of a business from a composite web document
CA2402437C (en) Methods and systems for monitoring quality assurance
De Meo et al. EC-XAMAS: supporting e-commerce activities by an XML-based adaptive multi-agent system
US20080177761A1 (en) Dynamically optimized storage system for online user activities
JP3597104B2 (en) Electronic catalog automatic collection system
AU773065B2 (en) An indexing system and method
Manandhar An efficient search paradigm implementing a unique identifier

Legal Events

Date Code Title Description
AS Assignment

Owner name: REVENUE SCIENCE, INC., WASHINGTON

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:DIGIMINE, INC.;REEL/FRAME:014185/0138

Effective date: 20030527

STCB Information on status: application discontinuation

Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION

AS Assignment

Owner name: COMERICA BANK, CALIFORNIA

Free format text: SECURITY AGREEMENT;ASSIGNOR:REVENUE SCIENCE, INC.;REEL/FRAME:020393/0843

Effective date: 20010302

AS Assignment

Owner name: AUDIENCESCIENCE, INC. FKA REVENUE SCIENCE, INC. FK

Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:COMERICA BANK;REEL/FRAME:024023/0574

Effective date: 20100226