US20110246816A1 - Configuring a system to collect and aggregate datasets - Google Patents

Configuring a system to collect and aggregate datasets Download PDF

Info

Publication number
US20110246816A1
US20110246816A1 US12/877,902 US87790210A US2011246816A1 US 20110246816 A1 US20110246816 A1 US 20110246816A1 US 87790210 A US87790210 A US 87790210A US 2011246816 A1 US2011246816 A1 US 2011246816A1
Authority
US
United States
Prior art keywords
node
data
sink
master
collector
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
US12/877,902
Other versions
US9317572B2 (en
Inventor
Jonathan Ming-Cyn Hsieh
Henry Noel Robinson
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Cloudera Inc
Original Assignee
Cloudera Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Cloudera Inc filed Critical Cloudera Inc
Priority to US12/877,902 priority Critical patent/US9317572B2/en
Assigned to Cloudera, Inc. reassignment Cloudera, Inc. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: ROBINSON, HENRY NOEL, HSIEH, JONATHAN MING-CYN
Publication of US20110246816A1 publication Critical patent/US20110246816A1/en
Priority to US15/098,198 priority patent/US10187461B2/en
Application granted granted Critical
Publication of US9317572B2 publication Critical patent/US9317572B2/en
Assigned to CITIBANK, N.A., AS COLLATERAL AGENT reassignment CITIBANK, N.A., AS COLLATERAL AGENT SECURITY INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: Cloudera, Inc., Hortonworks, Inc.
Assigned to JPMORGAN CHASE BANK, N.A. reassignment JPMORGAN CHASE BANK, N.A. FIRST LIEN NOTICE AND CONFIRMATION OF GRANT OF SECURITY INTEREST IN PATENTS Assignors: Cloudera, Inc., Hortonworks, Inc.
Assigned to JPMORGAN CHASE BANK, N.A. reassignment JPMORGAN CHASE BANK, N.A. SECOND LIEN NOTICE AND CONFIRMATION OF GRANT OF SECURITY INTEREST IN PATENTS Assignors: Cloudera, Inc., Hortonworks, Inc.
Assigned to Cloudera, Inc., Hortonworks, Inc. reassignment Cloudera, Inc. RELEASE OF SECURITY INTERESTS IN PATENTS Assignors: CITIBANK, N.A.
Active legal-status Critical Current
Adjusted expiration legal-status Critical

Links

Images

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00Network arrangements or protocols for supporting network services or applications
    • H04L67/01Protocols
    • H04L67/10Protocols in which an application is distributed across nodes in the network
    • H04L67/104Peer-to-peer [P2P] networks
    • H04L67/1042Peer-to-peer [P2P] networks using topology management mechanisms
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/07Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/16Error detection or correction of the data by redundancy in hardware
    • G06F11/20Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
    • G06F11/202Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where processing functionality is redundant
    • G06F11/2023Failover techniques
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/25Integrating or interfacing systems involving database management systems
    • G06F16/254Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
    • H04L41/12Discovery or management of network topologies
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00Network arrangements or protocols for supporting network services or applications
    • H04L67/01Protocols
    • H04L67/10Protocols in which an application is distributed across nodes in the network
    • H04L67/104Peer-to-peer [P2P] networks
    • H04L67/1087Peer-to-peer [P2P] networks using cross-functional networking aspects
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/30Monitoring
    • G06F11/34Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
    • G06F11/3466Performance evaluation by tracing or monitoring
    • G06F11/3476Data logging

Definitions

  • FIG. 1 illustrates a block diagram of client devices that generate datasets (log data) to be collected and aggregated for storage in the storage device via interacting nodes in various tiers, in some instances, through a network.
  • datasets log data
  • FIG. 2A depicts a diagram showing one example configuration of nodes that are controlled/configured by a master and dataflow therein from a data source to a data sink (e.g., storage).
  • a data sink e.g., storage
  • FIG. 2B depicts diagrams showing examples of configuration of nodes controlled/configured by a master and the dataflow therein from a data source to a data sink.
  • FIG. 3A depicts a diagram showing one example configuration of a node (nodes) controlled by a master residing on the same machine and dataflow therein from a data source to a data sink.
  • FIG. 3B depicts a diagram showing one example configuration of a node controlled by a master residing on a different machine and dataflow therein from a data source to a data sink.
  • FIG. 4 depicts a diagram showing an example configuration of nodes in multiple tiers on the same machine controlled by multiple masters residing on different machines and dataflow therein from data sources to a data sink.
  • FIG. 5 depicts a diagram showing an example configuration of nodes in multiple tiers on different machines controlled by multiple masters residing on different machines and dataflow therein from data sources to a data sink.
  • FIG. 6 depicts a block diagram illustrating example components of a node through which dataflow occur.
  • FIG. 7 depicts a block diagram illustrating example components of a decorator module in a node.
  • FIG. 8 depicts a block diagram illustrating example components of a master that controls the nodes in the system.
  • FIG. 9 depicts an example of an extensible data model used to represent events or data flow in a distributed system from a source to a sink.
  • FIG. 10A depicts a screenshot of an example user interface showing node status and configurations with node mappings.
  • FIG. 10B depicts a screenshot of an example user interface for configuring a node.
  • FIG. 10C depicts a screenshot of an example user interface for configuring multiple nodes.
  • FIG. 10D depicts a screenshot of an example user interface for issuing raw commands.
  • FIG. 10E depicts a screenshot of an example user interface showing statistics for a node.
  • FIG. 10F depicts a screenshot of an example user interface showing statistics for a node.
  • FIG. 11 depicts a flowchart of an example process of facilitating collecting and aggregating datasets that are machine or user-generated for analysis.
  • FIG. 12 depicts a flowchart of an example process for configuring a system to collect and aggregate datasets.
  • FIG. 13 depicts a flowchart of an example for scaling the system by implementing multiple masters on multiple machines.
  • FIG. 14 depicts a flowchart of another example process for configuring a system to collect and aggregate datasets and to perform analytics on the datasets.
  • FIG. 15 depicts a flowchart of an example process for dynamically processing an event including a dataset that is streamed from a source to a sink via nodes.
  • FIG. 16 depicts a flowchart of an example process for collecting and aggregating datasets for storage in a file system with fault tolerance.
  • FIG. 17 depicts a flowchart of an example process for deleting the WAL in response to acknowledgement of receipt by the master.
  • FIG. 18 depicts a flowchart of an example process for collecting and aggregating datasets with fault tolerance using a store on failure mechanism.
  • FIG. 19 shows a diagrammatic representation of a machine in the example form of a computer system within which a set of instructions, for causing the machine to perform any one or more of the methodologies discussed herein, may be executed.
  • Embodiments of the present disclosure include systems and methods for configuring a system to collect aggregate datasets.
  • FIG. 1 illustrates a block diagram of client devices 102 A-N that generate datasets (log data) to be collected and aggregated for storage in the storage device 108 via interacting nodes in various tiers, in some instances, through a network 106 .
  • datasets log data
  • the client devices 102 A-N can be any system and/or device, and/or any combination of devices/systems that is able to establish a connection with another device, a server and/or other systems.
  • the client devices 102 A-N typically include display or other output functionalities to present data exchanged between the devices to a user.
  • the client devices and content providers can be, but are not limited to, a server desktop, a desktop computer, a thin-client device, an interne kiosk, a computer cluster, a mobile computing device such as a notebook, a laptop computer, a handheld computer, a mobile phone, a smart phone, a PDA, a Blackberry device, a Treo, and/or an iPhone, etc.
  • the client devices 102 A-N are coupled to a network 106 .
  • the client devices may be directly connected to one another.
  • users interact with user devices 102 A-N (e.g., machines or devices).
  • the devices 102 A-N can generate datasets such as log files to be collected and aggregated.
  • the file can include logs, information, and other metadata about clicks, feeds, status updates, data from applications, and associated properties and attributes.
  • User devices 102 A-N can have nodes executing or running thereon that collect the datasets that are user-generated or machine-generated, for example, based on user-interaction with applications or websites running on the devices. Such nodes can interact and/or communicate with one or more other nodes (e.g., either running on the same device/machine or another device/machine (e.g., machine/device 104 ) to facilitate collection and aggregation of datasets thus generated.
  • the datasets are eventually written to a file and stored, for example, in storage (e.g., repository 130 ) on a physical disk.
  • the dataflow among nodes can be configured at a master.
  • the nodes executed on the machines 102 or 104 can contact the master(s) to obtain configuration information, which have been set by default or configured by a user
  • the master can be executed on the same devices 102 A-N, 104 , or at the host device 100 .
  • One or multiple masters can be involved in the mapping of data flow among the nodes and various machines.
  • FIG. 2-5 Some examples of architectural configurations among nodes, machines, and masters are illustrated and described with further reference to the example of FIG. 2-5 .
  • the functionalities of the nodes and configuration are described with further reference to the examples of FIG. 6-8 .
  • the network 106 over which the client devices 102 A-N, 104 , host, and the nodes and masters therein communicate may be a telephonic network, an open network, such as the Internet, or a private network, such as an intranet and/or the extranet.
  • the Internet can provide file transfer, remote log in, email, news, RSS, and other services through any known or convenient protocol, such as, but is not limited to the TCP/IP protocol, Open System Interconnections (OSI), FTP, UPnP, iSCSI, NSF, ISDN, PDH, RS-232, SDH, SONET, etc.
  • OSI Open System Interconnections
  • the network 106 can be any collection of distinct networks operating wholly or partially in conjunction to provide connectivity to the client devices, host server, and may appear as one or more networks to the serviced systems and devices.
  • communications to and from the client devices 102 A-N can be achieved by, an open network, such as the Internet, or a private network, such as an intranet and/or the extranet.
  • communications can be achieved by a secure communications protocol, such as secure sockets layer (SSL), or transport layer security (TLS).
  • SSL secure sockets layer
  • TLS transport layer security
  • Internet refers to a network of networks that uses certain protocols, such as the TCP/IP protocol, and possibly other protocols such as the hypertext transfer protocol (HTTP) for hypertext markup language (HTML) documents that make up the World Wide Web (the web).
  • HTTP hypertext transfer protocol
  • HTML hypertext markup language
  • a web server which is one type of content server, is typically at least one computer system which operates as a server computer system and is configured to operate with the protocols of the World Wide Web and is coupled to the Internet.
  • the physical connections of the Internet and the protocols and communication procedures of the Internet and the web are well known to those of skill in the relevant art.
  • the network 106 broadly includes anything from a minimalist coupling of the components illustrated in the example of FIG. 1 , to every component of the Internet and networks coupled to the Internet.
  • communications can be achieved via one or more wireless networks, such as, but is not limited to, one or more of a Local Area Network (LAN), Wireless Local Area Network (WLAN), a Personal area network (PAN), a Campus area network (CAN), a Metropolitan area network (MAN), a Wide area network (WAN), a Wireless wide area network (WWAN), Global System for Mobile Communications (GSM), Personal Communications Service (PCS), Digital Advanced Mobile Phone Service (D-Amps), Bluetooth, Wi-Fi, Fixed Wireless Data, 2G, 2.5G, 3G networks, enhanced data rates for GSM evolution (EDGE), General packet radio service (GPRS), enhanced GPRS, messaging protocols such as, TCP/IP, SMS, MMS, extensible messaging and presence protocol (XMPP), real time messaging protocol (RTMP), instant messaging and presence protocol (IMPP), instant messaging, USSD, IRC, or any other wireless data networks or messaging protocols.
  • LAN Local Area Network
  • WLAN Wireless Local Area Network
  • PAN Personal area network
  • CAN Campus area network
  • MAN Metropolitan area network
  • the client devices 102 A-N can be coupled to the network (e.g., Internet) via a dial up connection, a digital subscriber loop (DSL, ADSL), cable modem, and/or other types of connection.
  • the client devices 102 A-N can communicate with remote servers (e.g., web server, host server, mail server, and instant messaging server) that provide access to user interfaces of the World Wide Web via a web browser, for example.
  • remote servers e.g., web server, host server, mail server, and instant messaging server
  • the repository 130 can store software, descriptive data, images, system information, drivers, collected datasets, aggregated datasets, log files, analytics of collected datasets, enriched datasets, etc.
  • the repository may be managed by a database management system (DBMS), for example but not limited to, Oracle, DB2, Microsoft Access, Microsoft SQL Server, MySQL, FileMaker, etc.
  • DBMS database management system
  • the repositories can be implemented via object-oriented technology and/or via text files, and can be managed by a distributed database management system, an object-oriented database management system (OODBMS) (e.g., ConceptBase, FastDB Main Memory Database Management System, JDOlnstruments, ObjectDB, etc.), an object-relational database management system (ORDBMS) (e.g., Informix, OpenLink Virtuoso, VMDS, etc.), a file system, and/or any other convenient or known database management package.
  • OODBMS object-oriented database management system
  • ORDBMS object-relational database management system
  • the repository is managed by a distributed file system or network file system that allows access to files from multiple hosts/machines over a network.
  • the distributed file system can include by way of example, the Hadoop Distributed File system (HDFS).
  • HDFS Hadoop Distributed File system
  • file systems can be used as well, for example, through integration of Hadoop's interface which provides an abstraction layer for the file system.
  • a local file system where a node resides can be used.
  • the HDFS native distributed file system can also be used.
  • S3 a remote file system hosted by Amazon web services
  • FTP a remote file system hosted by Amazon web services
  • KFS Kosmos file system—another distributed file system
  • Clients can also write to different file systems (NFS), or other file systems.
  • NFS file systems
  • the user devices 102 and 104 are able to write files (e.g., files including by way of example, collected and aggregated datasets/logs/log files) to the repository 130 , either through the network 106 or without utilizing the network 106 .
  • the host server 100 can be implemented on a known or convenient computer system, such as is illustrated in FIG. 19 .
  • FIG. 2A depicts a diagram showing one example configuration 200 of nodes 202 and 204 that are controlled/configured by a master 206 and dataflow therein from a data source 201 to a data sink 207 (e.g., repository 208 ).
  • a data sink 207 e.g., repository 208
  • the node 202 can be mapped to the data source 201 on the machine that generates that dataset.
  • the dataset can include data, metadata, complex data, including logs of clicks, social networking sites, feeds, status updates, logs from local or remote applications, etc.
  • the data source can include, by way of example, but not limitation, a syslog port, an incoming network connection, and an IRC channel, output from execution of a program, a text file.
  • a data source can include a Hadpp sequence file formatted file.
  • the node 202 can map the data source 201 to a receiving location such as a data sink 203 .
  • a master 206 controls the mapping of the data source 201 to the receiving location 203 via the node 202 .
  • data flow of the dataset from the data source 201 to the receiving location 203 is configurable and reconfigurable at the master 206 by a user (e.g., system administrator, developer, etc.).
  • the node 202 can contact the master 206 to obtain its mapping/configuration information.
  • nodes contact the master 206 to obtain configuration information allows node configuration to dynamically change without having to login and to restart the daemon (e.g., a server process on a UNIX-based machine).
  • the node's configuration can be changed to a new one without taking the system offline.
  • nodes are configured using a web interface to the master. Screenshots of example interfaces are illustrated with further reference to the example of FIG. 10 .
  • one or more masters can be implemented using one or more machines, as illustrated in subsequent figures.
  • the master 206 can be executed on the machine on which the collected dataset is received or generated or one or more different machines. Different types of architectural configurations for the master 206 are illustrated in subsequent figures.
  • a receiving location can include, by way of example, not limitation, a network connection, a physical disk, a console, a text file.
  • the receiving location can also include, a file in the Hadoop sequence file format, an HDFS path, a port, a monitoring or alerting application, and/or an IRC.
  • the collected dataset from the data source 201 can be aggregated at the receiving location (data sink 203 ).
  • the receiving location e.g., or data sink 203
  • analytics can be performed on the dataset upon collection at node 202 and/or aggregation at node 204 .
  • the master 206 controls the functions/properties of the nodes 202 and/or 204 .
  • the nodes 202 / 204 can be associated with properties and/or functions, including but not limited to, adding encryption or digital signature, performing batching/unbatching, performing compression/uncompression, generating a checksum of a dataset, performing sampling, performing benchmarking (e.g., by injecting benchmark tags).
  • the analytic that are performed can include, feature extraction, filtering, transformation, generating aggregate counts, statistics, etc.
  • the node 204 in aggregating the dataset, can also be mapped to a data source 205 and a data sink 207 , as controlled and configured at the master 206 .
  • the data source 205 of the node 204 can be mapped to the sink 203 of node 202 . Additional functions and properties of nodes are illustrated with further reference to the example of FIG. 6-7 .
  • the node 204 (e.g., a collector node) writes the aggregated data to a storage location.
  • the storage location can be localized or distributed among multiple machines and the dataset is stored redundantly.
  • the data sink 207 of the node 204 can be mapped to a repository 208 , by the master 206 , for example.
  • the repository 208 may be managed by a file system.
  • the file system can be distributed (e.g., the the Hadoop Distributed File System (HDFS)).
  • HDFS Hadoop Distributed File System
  • node 202 although illustrated in the example of FIG. 2A as having being mapped to a single node 204 , can be mapped to multiple nodes, as further illustrated in the example of FIG. 2B .
  • FIG. 2B depicts diagrams showing examples of configuration of nodes controlled/configured by a master and the dataflow therein from a data source to a data sink.
  • an agent node 212 can send incoming dataset/event from a data source 211 to multiple collector nodes (e.g., node 214 and node 216 ) which further maps the data flow to the repository 218 .
  • the collector nodes are failover nodes for handling failures when appending new events. Failover sinks can be used to specify alternate collectors to contact in the event the assigned collector fails. Fail over sinks can also be used to specify a local disk sink to store data until the assigned collector recovers. For example, if node 214 goes down, the agent node 212 can deliver events via collector agent 216 to the repository 218 or any other receiving location or destination.
  • the collector 216 can queue their logs until node 214 comes back online.
  • the failover assignment can be specified through the master 220 .
  • the failover mechanism is built in by default by disbursing collector node assignments evenly among agent nodes. This can mitigate the chances of one collector from being overloaded when one collector fails.
  • multiple collectors 224 are assigned to receive events/data flow from agent node 222 .
  • the collectors 224 can store the event/data flow in the repository 228 .
  • the collectors 224 may be failover chains.
  • the collectors 224 can be fan out sinks, each of which receives the incoming event from the data source 221 .
  • the fan out sinks can be configured by the master 230 as collectors 224 .
  • the multiple collectors 224 as fan out sinks can be used for data replication for processing data off a main data flow path.
  • FIG. 3A depicts a diagram showing one example configuration of a node (nodes) controlled by a master 306 residing on the same machine 300 and dataflow therein from a data source 301 to a data sink 305 .
  • the agent node 302 and the master 306 used for controlling, configuring, and/or monitoring the node 302 can be implemented on the same machine 300 .
  • the master 306 can be used to specify functions and/or properties of the agent node 302 in sending an incoming event/data set from the source 310 to the sink 305 .
  • a collector node 304 is also implemented on the same machine 304 as the agent node 302 and the master 306 .
  • the master 306 can control, configure, and/or monitor both nodes 302 and 304 in mapping events/data flow fro the source 301 to the sink 305 .
  • FIG. 3B depicts a diagram showing example configurations of a node 312 or nodes 312 and 314 executed on a machine 320 and controlled by a master 316 which is executed on a different machine 330 and dataflow therein from a data source 311 to a data sink 315 .
  • the master 316 executed on machine 330 controls, configures, and/or monitors the node 312 on the machine 320 in sending data sets/events from the data source 311 to the data sink 315 .
  • the collector node 314 which is mapped to receive datasets/events from the node 312 can also be executed on machine 320 .
  • the collector node can be configured to send datasets received from the node 312 to a data sink 315 .
  • the master 316 can also control, configure, and/or monitor the collector node 314 .
  • having a single master is a single point of failure in the system. If the master 316 fails, the ability to update nodes (e.g., nodes 314 and/or 314 ), configure nodes, control, and/or monitor nodes, etc. goes down.
  • nodes e.g., nodes 314 and/or 314
  • configure nodes, control, and/or monitor nodes, etc. goes down.
  • multiple masters can be implemented on multiple machines to maintain a consistent state in the system.
  • using multiple masters on multiple machines allows the system to scale with increasing amount of log data and events to process and analyze.
  • using multiple collectors can also increase log collection throughput and can improve timeliness of event/dataset delivery since more collector nodes are available at any given time during operation. Since data collection is parallelizable, the load from agent nodes can be shared and distributed among many collector nodes. Example architectures with multiple masters and multiple collectors are illustrated in FIG. 4-5 .
  • FIG. 4 depicts a block diagram showing an example configuration of nodes in multiple tiers on the same machine 400 which controlled by multiple masters 406 and 408 residing on different machines 420 and 430 respectively, and dataflow therein from data sources 401 , 403 , and 405 to a data sink 407 .
  • the example system of FIG. 4 utilizes multiple agent nodes 402 in an agent tier 412 and multiple collector nodes 404 in a collector tier 414 .
  • the nodes 402 in the agent tier 412 are located on the machine 400 which is also producing logs that are being collected.
  • the data source 401 of the node 402 can be configured to be a port on the machine 400 .
  • the sink of the node 402 can be configured to be node 404 in the collector tier 414 .
  • collector nodes 404 can listen and receive data from multiple agent nodes 412 and aggregate logs. In addition, collector nodes 414 can further write the logs to a data sink 407 in the storage tier 416 .
  • agent nodes In reliable mode agent nodes generally write data to the respective local disks. This data might not be deleted until after it knows that the data has reached its final destination. Collectors generally don't write to their local disk—they actually forward data to the storage tier which eventually points the collector to storage tier nodes that write to storage tier disks.
  • An agent is generally on a system that is responsible for doing other work (serving web pages) and thus agent nodes typically strive to have low resource utilization (e.g., cpu, memory, disk).
  • Collector nodes are generally dedicated to processing large amounts of data and can use more expensive algorithms because its processing units are generally less occupied with other tasks.
  • the data sink 407 may or may not be a storage unit/device.
  • the collector nodes 404 can also send the data sets downstream to cluster for further processing, a network, etc.
  • FIG. 5 depicts a diagram showing an example configuration of nodes 502 and 504 in multiple tiers on different machines controlled by multiple masters 506 residing on different machines and dataflow therein from data sources 501 to a data sink 507 .
  • Agent nodes 502 can be executed on different machines and map incoming data/events to collector nodes 504 .
  • the multiple machines may be in a cluster and can include web servers.
  • the multiple machines generate the log data to be collected and each of the multiple machines is associated with at least one agent node to facilitate data collection and/or aggregation.
  • the agent nodes 502 can forward the collected log data or events to a collector tier 512 which includes at least one collector node 504 but in some instances, multiple collector nodes 504 which can combine streams of log data/events into larger streams, for example, for writing to the storage tier 516 .
  • the collector node(s) 504 can aggregate the log data collected from the machines on which the agent nodes are executed.
  • the collector nodes 504 can also be executed on different machines with each machine having various numbers of collector nodes 504 , which passes events on to the data sink 507 .
  • the collector node(s) 504 outputs the log data to multiple destinations in the same or different formats.
  • Some of the supported formats include, by way of example not limitation, raw text file, json (a web format), avro binary (an apache serialization format), a hadoop native sequence file, text emulating a syslog formatted entry, an apache web server formatted entry, and a “log4j” formatting pattern, etc.
  • File formats are the format for batches of records. Record formats are the formatting for individual events/records. A collector writing to storage tier can be responsible for file formats. The work for formatting individual records however can be pushed upstream.
  • One embodiment of the topology includes a storage tier 516 having a storage system coupled to the machines in the system.
  • the storage system can include a distributed file system to which the collector node(s) 504 in the collector tier 514 stores the log data aggregated from the set of machines in the system.
  • analytics can be performed on the log data/aggregated events by the agent nodes 502 and/or the collector nodes 504 .
  • the master 506 can be used to control/configure the types of analytics that are performed.
  • Agent nodes 502 and collector nodes 504 can be configured, re-configured via the masters 506 , which can be executed on multiple machines that are different from the machines that the collector 504 and agent nodes 502 are executed on.
  • the masters 506 specify the configuration/topology of the system 500 and the agent node(s) 502 and the collector node(s) 504 contact the master 506 to retrieve configuration information.
  • the master 506 can specify that an agent data sink for the agent node 502 is a collector data source for the collector node 504 in the collector tier 514 and that a collector data sink 507 for the collector node 504 is the distributed file system in the storage tier 516 .
  • the master 506 can be used by a user to configure or reconfigure the agent data sink and agent data source and/or to configure or reconfigure the collector data source and the collector data sink.
  • FIG. 6 depicts a block diagram illustrating example components of a node 602 through which dataflow occur.
  • the node 602 can include a source specifier module 604 , a sink specifier module 606 , a configuration file 610 , and/or a decorator module 620 .
  • a “module” or an “engine” includes a dedicated or shared processor and, typically, firmware or software modules that are executed by the processor. Depending upon implementation-specific or other considerations, an engine can be centralized or its functionality distributed. An engine can include special purpose hardware, firmware, or software embodied in a computer-readable medium for execution by the processor.
  • a computer-readable medium is intended to include all mediums that are statutory (e.g., in the United States, under 35 U.S.C. 101), and to specifically exclude all mediums that are non-statutory in nature to the extent that the exclusion is necessary for a claim that includes the computer-readable medium to be valid.
  • Known statutory computer-readable mediums include hardware (e.g., registers, random access memory (RAM), non-volatile (NV) storage, to name a few), but may or may not be limited to hardware.
  • the node includes a source specifier module 604 and a sink specifier module 606 .
  • the source specifier module 604 and sink specifier module 606 can be implemented, example, as software embodied in a computer-readable medium or computer-readable storage medium on a machine, in firmware, in hardware, in a combination thereof, or in any applicable known or convenient device or system.
  • This and other modules or engines described in this specification are intended to include any machine, manufacture, or composition of matter capable of carrying out at least some of the functionality described implicitly, explicitly, or inherently in this specification, and/or carrying out equivalent functionality.
  • the source specifier module 604 can include an identifier or mapper to a source that produces or accepts events or data fed into the system.
  • the node 604 reads or tails the event source.
  • the module 604 can detect, recognize, identify, and/or store a port ID (e.g., a port name, a TCP or UDP port, port number) and a file ID (e.g., a file name, a file path, etc.).
  • the file can be any type of file, including but not limited to, messages, system log files, Hadoop log files, text files, etc.
  • the event source can include, a console, a sequence file (e.g., serialized events in a sequence file format), a source that synthetically generates a specified number of messages of a select size.
  • the sink specifier module 606 includes an identifier or mapper to a location that consumes data or an event (e.g., a receiving or destination location for data or events).
  • a sink can include, by way of example, a disk, a repository, a distributed file system on a disk (e.g., the Hadoop distributed file system), a console, a device/machine, and/or a network/network connection over which the data/event is to be forwarded over.
  • the sink specifier module 606 can detect, recognize, identify, and or store, a machine/device ID, a port ID, a file ID, and/or a path ID.
  • the path ID is a distributed sequence file path.
  • the source and sink specifier modules can store or retrieve the configuration information stored in the configuration file 610 .
  • the configuration file 610 may be loaded with default values upon installation.
  • the configuration file 610 may be updated by a master that controls the node 602 . The updates the configuration file 610 may occur while the system is running or when the system is off.
  • the node 602 may be assigned to have an agent role or a collector rule.
  • an agent node is co-located on machines/devices with the service/application that is generating, producing, creating events/logs/data.
  • a collector node can receive, accept, check, listen for data from one or more agent nodes.
  • a collector node can be mapped to the data sink of an agent node, for example, to aggregate logs/data and/or to extract analytics.
  • a collector node can further write the dataset, events, and/or any generated events to storage (e.g., a distributed file system).
  • the node 602 includes a decorator module 620 .
  • the decorator module 620 can be implemented, example, as software embodied in a computer-readable medium or computer-readable storage medium on a machine, in firmware, in hardware, in a combination thereof, or in any applicable known or convenient device or system. This and other modules or engines described in this specification are intended to include any machine, manufacture, or composition of matter capable of carrying out at least some of the functionality described implicitly, explicitly, or inherently in this specification, and/or carrying out equivalent functionality.
  • the decorator module 620 can add properties/functionalities of a sink and modify the data/event streams that pass through them, like a wrapper. For example, the decorator module 620 can be used to increase reliability, robustness of the system or to perform analytics.
  • a decorator can correspond to a sink that interposes between the source and the sink.
  • the decorator processes events and then sends it down the pipeline to another sink (which may be another decorator wrapping another sink) on the same node/dataflow.
  • Sources produce events, and pass them through the decorator to sink.
  • FIG. 7 depicts a block diagram illustrating example components of a decorator module 720 in a node.
  • the decorator module 720 can include, a write ahead module 702 , a sampler module 704 , an open module 706 , an append module 708 , a batch/unbatch module 710 , and/or a GZIP/GUNZIP module 712 .
  • a “module” or an “engine” includes a dedicated or shared processor and, typically, firmware or software modules that are executed by the processor. Depending upon implementation-specific or other considerations, an engine can be centralized or its functionality distributed. An engine can include special purpose hardware, firmware, or software embodied in a computer-readable medium for execution by the processor.
  • a computer-readable medium is intended to include all mediums that are statutory (e.g., in the United States, under 35 U.S.C. 101), and to specifically exclude all mediums that are non-statutory in nature to the extent that the exclusion is necessary for a claim that includes the computer-readable medium to be valid.
  • Known statutory computer-readable mediums include hardware (e.g., registers, random access memory (RAM), non-volatile (NV) storage, to name a few), but may or may not be limited to hardware.
  • the decorator module 720 includes a write ahead module 702 .
  • the decorator module 620 can be implemented, example, as software embodied in a computer-readable medium or computer-readable storage medium on a machine, in firmware, in hardware, in a combination thereof, or in any applicable known or convenient device or system.
  • This and other modules or engines described in this specification are intended to include any machine, manufacture, or composition of matter capable of carrying out at least some of the functionality described implicitly, explicitly, or inherently in this specification, and/or carrying out equivalent functionality.
  • the write ahead module 702 provides reliability and durability and writes/stores events/datasets to persistent storage on the machine generating the event (writeAhead( . . . )) prior to forwarding them. In the event and any of the downstream nodes fails, the system has a copy of the dataset that can still be retrieved. In response to determining that the dataset/event has safely reached its destination, the copy can be deleted from persistent storage.
  • the write ahead module 702 can also assign batch identifiers for batches of messages/events to track
  • the decorator module 720 includes a sampler module 704 .
  • the decorator module 704 can be implemented, example, as software embodied in a computer-readable medium or computer-readable storage medium on a machine, in firmware, in hardware, in a combination thereof, or in any applicable known or convenient device or system.
  • This and other modules or engines described in this specification are intended to include any machine, manufacture, or composition of matter capable of carrying out at least some of the functionality described implicitly, explicitly, or inherently in this specification, and/or carrying out equivalent functionality.
  • the sampler module 704 can sample data input or events at select intervals (e.g., intervalSampler(N)). For example, the sample module 704 can be configured to send every nth element/event from the source to the data sink. In one embodiment, the sampler module 704 samples based on a probability such that each event or message has a specified chance of being forwarded (e.g., probSampler (p)). In addition, the sampler module 704 can be a reservoir sampler (e.g., reserverSampler (K) such that when flushed, a selected number (K) of elements are forwarded and each event that passes through has the same probability of being selected for forwarding.
  • a reservoir sampler e.g., reserverSampler (K) such that when flushed, a selected number (K) of elements are forwarded and each event that passes through has the same probability of being selected for forwarding.
  • the decorator module 720 includes an open module 706 (or re-try open module).
  • the open module 706 can be implemented, example, as software embodied in a computer-readable medium or computer-readable storage medium on a machine, in firmware, in hardware, in a combination thereof, or in any applicable known or convenient device or system.
  • This and other modules or engines described in this specification are intended to include any machine, manufacture, or composition of matter capable of carrying out at least some of the functionality described implicitly, explicitly, or inherently in this specification, and/or carrying out equivalent functionality.
  • the open module 706 in reading or opening a file or incoming event, continues to re-try the opening process if the initial attempt fails.
  • the decorator module 720 includes an append module 708 (or re-try append module).
  • the append module 708 can be implemented, example, as software embodied in a computer-readable medium or computer-readable storage medium on a machine, in firmware, in hardware, in a combination thereof, or in any applicable known or convenient device or system.
  • This and other modules or engines described in this specification are intended to include any machine, manufacture, or composition of matter capable of carrying out at least some of the functionality described implicitly, explicitly, or inherently in this specification, and/or carrying out equivalent functionality.
  • the append module 708 can append events to a target. In the event that the append fails, the module 708 can close the target, re-open it and attempt to append again.
  • sinks and decorator modules include open, append, and close functions. Typically, if one of these operations fail, the modules can give up after a first try.
  • the sink/decorator can be configured to keep trying the failed operation (e.g.,retry opening or appending).
  • the decorator module 720 includes a batch/unbatch module 710 .
  • the batch/unbatch module 710 can be implemented, example, as software embodied in a computer-readable medium or computer-readable storage medium on a machine, in firmware, in hardware, in a combination thereof, or in any applicable known or convenient device or system.
  • This and other modules or engines described in this specification are intended to include any machine, manufacture, or composition of matter capable of carrying out at least some of the functionality described implicitly, explicitly, or inherently in this specification, and/or carrying out equivalent functionality.
  • the batch/unbatch module 710 can buffer a select number of events/message together to be forwarded or sent as a single aggregate event.
  • the batch/unbatch module 710 can take an aggregate event and split and forward the original events.
  • the decorator module 720 includes a GZIP/GUNZIP module 712 .
  • the GZIP/GUNZIP module 712 can be implemented, example, as software embodied in a computer-readable medium or computer-readable storage medium on a machine, in firmware, in hardware, in a combination thereof, or in any applicable known or convenient device or system. This and other modules or engines described in this specification are intended to include any machine, manufacture, or composition of matter capable of carrying out at least some of the functionality described implicitly, explicitly, or inherently in this specification, and/or carrying out equivalent functionality.
  • the GZIP/ZIP module 712 can zip or unzip an event/message or multiple events/messages.
  • multiple decorators can be assigned to an event, message, or dataset.
  • decorators can be combined to batch multiple messages together and compress the messages before forwarding them to the mapped data sink.
  • FIG. 8 depicts a block diagram illustrating example components of a master 800 that controls the nodes in the system.
  • the master 800 can include a node configurator module 802 , a system health monitor 804 , a consistency manager 806 , a map module 808 , and/or an acknowledger module 810 .
  • the master 800 can include additional or less modules.
  • the master 800 although illustrated as being a discrete component, can be implemented on multiple machines, as shown in the examples of FIG. 4-5 on full or partial redundancy.
  • the master 800 as implemented on a single or multiple machines can perform as a central control and monitor for the system.
  • the master 800 can track and monitor the status, health of nodes, and the data flow in the system. In tracking and monitoring system status, the master 800 can automatically perform any system re-configurations that may be needed to resolve node failure or load imbalance, for example.
  • the master 800 can identify, detect, or be used to specify new nodes or deleted nodes in the system and perform any adjustments in data flow by updating the mapping of data sinks and sources.
  • the nodes contact the master 800 to retrieve configuration information.
  • the node detects that the configuration version has been changed and can instantiate and activate the configuration. This allows updates to the system and mapping to be performed in dynamically, in operation, without starting or rebooting the system. For example, a node's configuration can be updated during operation.
  • a user can reconfigure the setting by accessing the master 800 .
  • One of accessing the master 800 is through the master's web page. Examples of screenshots are illustrated in FIG. 10A-F .
  • the configuration link is displayed on the master's web interface. In the interface, for example, node names are depicted and sources and sinks can be viewed, updated, or deleted. Once a configuration has been changed, the version identifier or stamp on the master's web page will be updated.
  • a “module” or an “engine” includes a dedicated or shared processor and, typically, firmware or software modules that are executed by the processor. Depending upon implementation-specific or other considerations, an engine can be centralized or its functionality distributed. An engine can include special purpose hardware, firmware, or software embodied in a computer-readable medium for execution by the processor.
  • a computer-readable medium is intended to include all mediums that are statutory (e.g., in the United States, under 35 U.S.C. 101), and to specifically exclude all mediums that are non-statutory in nature to the extent that the exclusion is necessary for a claim that includes the computer-readable medium to be valid.
  • Known statutory computer-readable mediums include hardware (e.g., registers, random access memory (RAM), non-volatile (NV) storage, to name a few), but may or may not be limited to hardware.
  • the master includes a node configurator module 802 .
  • the node configurator module 802 can be implemented, example, as software embodied in a computer-readable medium or computer-readable storage medium on a machine, in firmware, in hardware, in a combination thereof, or in any applicable known or convenient device or system.
  • This and other modules or engines described in this specification are intended to include any machine, manufacture, or composition of matter capable of carrying out at least some of the functionality described implicitly, explicitly, or inherently in this specification, and/or carrying out equivalent functionality.
  • the node configurator module 802 detects, updates, identifies, stores configuration settings of the nodes in the system. For example, the source and sink mappers can configure the data source(s) and/or data sink(s) for each node. In addition, the module 802 can assign and/or track functions/properties (e.g., analytics functions) that a node can perform.
  • the source and sink mappers can configure the data source(s) and/or data sink(s) for each node.
  • the module 802 can assign and/or track functions/properties (e.g., analytics functions) that a node can perform.
  • the master includes a system health monitor 804 .
  • the system health monitor 804 can be implemented, example, as software embodied in a computer-readable medium or computer-readable storage medium on a machine, in firmware, in hardware, in a combination thereof, or in any applicable known or convenient device or system.
  • This and other modules or engines described in this specification are intended to include any machine, manufacture, or composition of matter capable of carrying out at least some of the functionality described implicitly, explicitly, or inherently in this specification, and/or carrying out equivalent functionality.
  • the system health monitor 804 can track, detect, determine, monitor, system traffic, node health/status, dataflow, load balance/imbalance in the system.
  • the master 800 can update/adjust data flow based on system traffic and any load imbalances. In addition, if certain nodes have failed, the master 800 can assign failover nodes such that data flow is not interrupted.
  • the master includes a consistency module 806 .
  • the consistency module 806 can be implemented, example, as software embodied in a computer-readable medium or computer-readable storage medium on a machine, in firmware, in hardware, in a combination thereof, or in any applicable known or convenient device or system.
  • This and other modules or engines described in this specification are intended to include any machine, manufacture, or composition of matter capable of carrying out at least some of the functionality described implicitly, explicitly, or inherently in this specification, and/or carrying out equivalent functionality.
  • the consistency module 806 maintains consistency among multiple masters implemented on multiple machines in a system, by leveraging the consensus protocol and/or a gossip protocol.
  • the master 800 can maintain consistency using a consensus protocol for order critical state changes and using a gossip protocol for weak consistency among data including but not limited to node status updates, reporting information, and metrics information.
  • the master includes a map module 808 and an acknowledger module 810 .
  • the map module 808 and the acknowledger module 810 can be implemented, example, as software embodied in a computer-readable medium or computer-readable storage medium on a machine, in firmware, in hardware, in a combination thereof, or in any applicable known or convenient device or system.
  • This and other modules or engines described in this specification are intended to include any machine, manufacture, or composition of matter capable of carrying out at least some of the functionality described implicitly, explicitly, or inherently in this specification, and/or carrying out equivalent functionality.
  • the map module 808 monitors and manages a set of tags associated with batches of messages to track which messages or batches or messages have been successfully sent to a receiving location or stored in a data repository.
  • an agent node generates a batch comprising multiple messages from the log data and assigns a tag (e.g., can include a begin event tag and an end event tag) to the batch.
  • the agent can also compute a checksum for the batch of multiple messages and stores he tag and the checksum to a write-ahead-log (WAL) in persistent storage of the machine from which the dataset was generated.
  • WAL write-ahead-log
  • the agent node sends the batch of messages to a collector which verifies the checksum of the batch, and if successful, adds the tag to a map of tags for batches of messages.
  • the collector agent publishes the tag to the map module 808 of the master.
  • the map module 808 includes identifiers for events, messages, or batches of messages that have reached their mapped locations.
  • the master 800 can use the identifiers in the map module 808 to notify the agent nodes of batches that have been successfully stored or sent.
  • the acknowledger module 810 using the tags that are associated with batches in the map, acknowledges the agent node and the collector node of the batches that have been safely stored in the file system or sent down stream.
  • FIG. 9 depicts an example of an extensible data model used to represent events or data flow in a distributed system from a source to a sink.
  • the data model can include one or more timestamp fields 902 and 904 , a priority field 906 , a source host field 908 , a body field 910 , and/or a metadata table field 912 .
  • the data model is extensible and thus additional fields can be added on the fly during data flow. The data model is described with further reference to the example of FIG. 15 .
  • FIG. 10A depicts a screenshot of an example user interface showing node status and configurations with node mappings. This user interface can be accessed from the master.
  • FIG. 10B depicts a screenshot of an example user interface for configuring a node.
  • the configuration page can be accessed from the master's web page. For example, a node to be configured can be selected from the list and its source/sink can be specified.
  • FIG. 10C depicts a screenshot of an example user interface for configuring multiple nodes.
  • FIG. 10D depicts a screenshot of an example user interface for issuing raw commands.
  • Commands can be used to modify the global state of the system.
  • the “config” command sets the configuration of a flow on the master, who in turn sends the info to a node.
  • the “refresh” command takes an existing config and forces the node to reset (close then open).
  • the “decommission” command removes a logical node.
  • the “unmap” command removes a logical node from a physical node.
  • the “spawn” command creates a new logical node (could be used to associate an unmapped logical node onto another physical machine).
  • FIG. 10E depicts a screenshot of an example user interface showing statistics for a node.
  • the user interface shows the data flow configuration settings, and memory usage of a particular node.
  • this figure shows information about the node “LogicalNode[0]” and includes two analytics reports: a graph showing the number of incoming events and a graph showing a histogram of the frequency of certain words in the stream.
  • the source in this example is Twitter, and the sinks for the node include the graphs.
  • FIG. 10F depicts a screenshot of an example user interface showing statistics for a node. This figure illustrates the configuration info for this data flow, and similarly has an event graph and a histogram (showing an even distribution of substrings).
  • FIG. 11 depicts a flowchart of an example process of facilitating collecting and aggregating datasets that are machine or user-generated for analysis.
  • a dataset is collected on a machine on which the dataset is received or generated.
  • the data set can include logs from an application that is local or remote, a service, a social networking site, feeds, status updates, etc.
  • the dataset can be collected from a data source on the machine.
  • datasets can be collected from machines in a cluster which can include web servers, for example.
  • the dataset that is collected from the data source is aggregated at a receiving location.
  • the receiving location can be, for example, a network, a network connection, a physical disk, a consol, a text file, a file in a sequence file format, an HDFS path, a port, a monitoring or alerting application, and/or an IRC.
  • One or more receiving location can be specified.
  • the mapping of the data source to the receiving location is controlled by a master.
  • the master can be executed on a different machine than the machine on which the dataset is generated or received.
  • the master can also be executed on the same machine as the one on which the dataset is received or generated.
  • the data source is mapped to a receiving location via a logical node.
  • the logical node can contact the master to obtain configuration information including but not limited to, functions, data source, data sink, etc.
  • analytics are performed on the dataset upon collection or aggregation. Analytics that can be performed include, by way of example, feature extraction, filtering, and transformation, generation of aggregate counts, and/or generation of statistics.
  • the analytics can be performed in near real time relative to collection of the dataset.
  • the dataset aggregated at the receiving location is written to a storage location.
  • IO there is typically a tradeoff between throughput and latency.
  • sending lots of small things will take a lot longer than a few large things, even if the actual amount of payload is the same. This is generally due to overhead associated with each sent value (sent events in Flume's case).
  • delay can be added to help the scheduler improve on throughput.
  • the storage location can be distributed among multiple machines and the dataset is stored redundantly on a distributed file system
  • FIG. 12 depicts a flowchart of an example process for configuring a system to collect and aggregate datasets.
  • a client e.g., a web page/interface or a command line interface, another program (a master), or a device (e.g., a laptop, a cell phone, a smartphone, iPhone, etc.) is launched on a machine to access a master for the system, for example, using the master's universal resource locator (URL).
  • the master's web page can be used to view, change, and/or update a node's configuration, as illustrated in the example of FIG. 10A .
  • the system may include multiple machines (e.g., in a cluster) which generate datasets that are to be collected.
  • a data source in the system from where dataset is to be collected is identified.
  • a machine in the system that generates the dataset to be collected is configured to send the dataset to the data source.
  • an arrival location where the dataset that is collected is to be aggregated or written is identified.
  • an agent node is configured by specifying a source for the agent node as the data source in the system and specifying a sink for the agent node as the arrival location.
  • the agent node is generally configurable to perform analytics on the dataset.
  • the arrival location is specified as a collector source of a collector node.
  • a distributed file system is specified as a collector sink of the collector node.
  • the distributed file system can be, for example, the Hadoop distributed file system (HDFS)
  • FIG. 13 depicts a flowchart of an example for scaling the system by implementing multiple masters on multiple machines.
  • a master provides a centralized management point to monitor and update data flows. Changes in the system are told to the master, and changes can come from the master. When new resources are added to the system, the information is updated at the master. In addition, if a user issues a command that changes global state, the master can be used such that the master can react to these state changes. In addition, the master can dynamically respond to system changes such as load imbalances, node failure, machine failure, new hardware, etc. For example, a user configures a new node to be a collector. The master detects this, and automatically reacts by changing the configuration of some nodes to send events downstream to the new collector. The master can do this by changing global state.
  • a collector node hasn't communicated with the master for a while.
  • the master may decide that the machine might have fialed and automatically reacts by sending an event to an admin (via email/irc/im/sms, etc.).
  • nodes periodically report their performance metrics to the master (e.g., in the examples of FIGS. 10E , 10 F). The reports indicate that a collector is being overloaded with work, and reacts to new metrics data by shifting some of the work from the overloaded collector to another with less load. This can be performed by changing global state.
  • Nodes in the system can be configured/reconfigured dynamically via the master.
  • the nodes can be arranged in a three-tier topology (e.g., agent tier, collector tier, storage tier) or any other types of topology with any number of tiers.
  • a user can use the master to configure the topology and the nodes, for example, by accessing a link on the web page for the master, as illustrate in the example screenshot of FIG. 10B .
  • the system is scaled by implementing multiple masters on multiple machines.
  • consistency is maintained among the multiple masters using a consensus protocol for order critical state changes.
  • order critical state changes can include data flow configurations and node-to-machine mappings.
  • consistency is maintained among the multiple masters using a gossip protocol. For example, node status updates, reporting information, and metrics information are maintained using the gossip protocol. Typically, stronger consistency is maintained using the consensus protocol and weaker consistency is maintained using the gossip protocol.
  • FIG. 14 depicts a flowchart of another example process for configuring a system to collect and aggregate datasets and to perform analytics on the datasets.
  • process 1402 data sources on the multiple machines wherein datasets are to be collected from are identified.
  • process 1404 the multiple machines in the system that generate the datasets to be collected are configured to send the datasets to the data source.
  • process 1406 an arrival location where dataset that is collected is to be logged is identified.
  • configurations for the multiple machines are specified simultaneously by accessing a master through a web page and specifying the data sources for agent nodes.
  • the master can be used to configure and/or dynamically reconfigure the agent nodes.
  • statuses of the agent nodes are visible via the master.
  • a sink is specified for each of the agent node as the arrival location.
  • the sink is configured to be a fan-out sink that sends incoming datasets to child sinks coupled to the sink for data replication or for splitting off a CPU intensive tap.
  • the sink can be configured as a failover sink that specifies alternate collectors in case the collector fails.
  • the failover sink can specify a local disk to store data until the sink that has failed recovers.
  • the arrival location is specified as a collector source of a collector node.
  • a distributed file system is specified as a collector sink of the collector node.
  • FIG. 15 depicts a flowchart of an example process for dynamically processing an event including a dataset that is streamed from a source to a sink via nodes.
  • Events or messages are transmitted in a flexible and extensible data model.
  • events can be formatted as unstructured blobs and can be annotated with key-value attribute pairs which can be arbitrary.
  • This allows analytic features include feature extraction, filtering, and transformation to be performed as the system forwards datasets downstream.
  • the general pattern can utilize use a feature extractor to add key-value attributes, and then use a analysis module that reads key-value attributes and acts on it.
  • feature extraction modules in the system can scan raw data and pull out structured data (e.g., numbers). This data can be written as a key-value attribute.
  • raw data can take upon the form: “[31 Mar. 2010, 12:06 AM] Jon wrote this example”.
  • the system can use a regular expression to pull out the data and turn it into a single number representation (unix time).
  • the system can add a metadata attribute called ‘eventTime’ with the numerical representation as its value.
  • a threshold predicate can be: “was ‘eventtime’ before 31 Mar?”
  • a histogramming module can use that value and determine the number of events that were on 31 Mar (and 30 Mar, and 1 Apr, etc).
  • the example graphs shown in the examples of FIGS. 10E and 10F can be generated using this pattern.
  • attributes of the event is specified in a data model.
  • An example of the data model is illustrated in FIG. 9 .
  • the data model is extensible to add properties to the event as the dataset is streamed from the source to the sink.
  • additional data fields can be appended to the data model in response to additional sources being added to the event.
  • the attributes can include a priority, which can be specified by a logging system. If a machine is heavily loaded, some events can be prioritize over others, using the ‘priority field’. For example, FATAL (highest priority) messages may skip ahead of line and be sent before lesser prioritized messages (e.g., ERROR or WARN priority messages).
  • the DEBUG priority data can be erased before any of the INFO data gets erased, and INFO before WARN, etc.
  • the attributes can include a source machine, a body, a time stamp, a metadata table, etc. Events will generally include each of these attributes but the body may have zero length and the metadata table may be empty.
  • the timestamp can be a UNIX timestamp which is the time stamp from the source machine.
  • the UNIX timestamp can be measured in milliseconds, for example.
  • the timestamp may be a nanosecond timestamp, or any other monotonically increasing identifier which is a machine specific counter from the source machine.
  • the priority field of the message can take on example values: TRACE, DEGBUG, INFO, WARN, ERROR, OR FATAL.
  • a logging system e.g., syslog or log4j
  • the source host can include a machine name or IP address.
  • the body can include the raw log entry body. By default, the body can be truncated to a max of 32 KB per event. The max size can be reconfigured.
  • event/message output to directories or files can be controlled based on the values of data model representing the event.
  • the system or any analytics being performed uses the timestamp to bucket the dataset based on time or date, specified by day of week, month, date and time, day of month, hour, minute, locale's equivalent of AM or PM, seconds, seconds since, year, etc.
  • logs can be bucketed based on time, page hit, or the browser that is used, using extraction mechanisms.
  • logs can be bucketed based on time and data node name, for example.
  • the data group can be grouped based on a project identifier or person, for example, when tracking a feed of JIRA tickets from an apache feed.
  • the event described by the data model is annotated with key-value attribute pairs in a metadata table to enable analytic features to be performed on the dataset.
  • the metadata table includes a map from a string attribute name to an array of bytes.
  • the metadata table can be used to specify routing of the event to multiple or different storage locations.
  • the metadata table can be used to specify outputting of the dataset at the sink in multiple or different formats.
  • the acktags, rolltags, and checksum tags/values are examples of values that use the metadata table.
  • Metrics information generated by nodes can use the metadata table to store individual metrics. Any additional features can also utilize tag/values and the metadata table to allow new fields without rebuilding the program).
  • an event including a dataset that is streamed from a source to a sink via nodes is dynamically processed.
  • FIG. 16 depicts a flowchart of an example process for collecting and aggregating datasets for storage in a file system with fault tolerance.
  • datasets are collected from a data source on a machine where the datasets are generated.
  • the data can be collected from an agent node on the machine.
  • a batch ID comprising multiple messages is generated from the datasets.
  • the agent node generates the batch ID.
  • a tag is assigned to the batch ID and a checksum is computed for the batch, for example, also by the agent node.
  • the batch ID and messages are written to an entry in a write-ahead-log (WAL) in storage.
  • the writing can be performed by the agent node.
  • the storage for the WAL is on the machine where the datasets are generated.
  • the datasets are sent to a receiving location (e.g., a collector node).
  • the data set on the agent can be sent when the size or time elapsed reaches a certain threshold.
  • a file is written to the destination location, for example, by the collector node.
  • the tag is added to a map, in response to verifying the checksum of the batch of multiple messages at a known location (e.g., the master).
  • the map can be associated with multiple tags assigned to multiple batches of messages from the datasets.
  • tags associated with the batches in the file that have been written to the destination location are identified in the map. The process continues in the flow chart of FIG. 17 .
  • FIG. 17 depicts a flowchart of an example process for deleting the WAL in response to acknowledgement of receipt by the master.
  • process 1702 an acknowledgement message that the batch has been written to the destination location is queried for.
  • process 1704 it is determined whether the batch has been written in the destination location, which can be a storage tier comprising a distributed file system. If so, in process 1706 , the entry is deleted from the storage of the machine. If not, in process 1708 , the batch is resent.
  • FIG. 18 depicts a flowchart of an example process for collecting and aggregating datasets with fault tolerance using a store on failure mechanism.
  • a dataset is collected from a data source on a machine where the dataset is generated.
  • the dataset is sent to a receiving location which aggregates the dataset.
  • the receiving location is a collector in a collector tier on the machine where the dataset is generated or another machine.
  • the receiving location may also be a file system in a storage tier comprised in one or more storage devices.
  • the dataset is stored in persistent storage of the machine until the receiving location has been repaired or until another destination is identified in response to determining that receiving location which is mapped to receive the dataset has failed.
  • FIG. 19 shows a diagrammatic representation of a machine 1900 in the example form of a computer system within which a set of instructions, for causing the machine to perform any one or more of the methodologies discussed herein, may be executed.
  • the computer system 900 includes a processor, memory, non-volatile memory, and an interface device. Various common components (e.g., cache memory) are omitted for illustrative simplicity.
  • the computer system 900 is intended to illustrate a hardware device on which any of the components depicted in the example of FIG. 1 (and any other components described in this specification) can be implemented.
  • the computer system 900 can be of any applicable known or convenient type.
  • the components of the computer system 900 can be coupled together via a bus or through some other known or convenient device.
  • the processor may be, for example, a conventional microprocessor such as an Intel Pentium microprocessor or Motorola power PC microprocessor.
  • Intel Pentium microprocessor or Motorola power PC microprocessor.
  • machine-readable (storage) medium or “computer-readable (storage) medium” include any type of device that is accessible by the processor.
  • the memory is coupled to the processor by, for example, a bus.
  • the memory can include, by way of example but not limitation, random access memory (RAM), such as dynamic RAM (DRAM) and static RAM (SRAM).
  • RAM random access memory
  • DRAM dynamic RAM
  • SRAM static RAM
  • the memory can be local, remote, or distributed.
  • the bus also couples the processor to the non-volatile memory and drive unit.
  • the non-volatile memory is often a magnetic floppy or hard disk, a magnetic-optical disk, an optical disk, a read-only memory (ROM), such as a CD-ROM, EPROM, or EEPROM, a magnetic or optical card, or another form of storage for large amounts of data. Some of this data is often written, by a direct memory access process, into memory during execution of software in the computer 900 .
  • the non-volatile storage can be local, remote, or distributed.
  • the non-volatile memory is optional because systems can be created with all applicable data available in memory.
  • a typical computer system will usually include at least a processor, memory, and a device (e.g., a bus) coupling the memory to the processor.
  • Software is typically stored in the non-volatile memory and/or the drive unit. Indeed, for large programs, it may not even be possible to store the entire program in the memory. Nevertheless, it should be understood that for software to run, if necessary, it is moved to a computer readable location appropriate for processing, and for illustrative purposes, that location is referred to as the memory in this paper. Even when software is moved to the memory for execution, the processor will typically make use of hardware registers to store values associated with the software, and local cache that, ideally, serves to speed up execution.
  • a software program is assumed to be stored at any known or convenient location (from non-volatile storage to hardware registers) when the software program is referred to as “implemented in a computer-readable medium.”
  • a processor is considered to be “configured to execute a program” when at least one value associated with the program is stored in a register readable by the processor.
  • the bus also couples the processor to the network interface device.
  • the interface can include one or more of a modem or network interface. It will be appreciated that a modem or network interface can be considered to be part of the computer system 1900 .
  • the interface can include an analog modem, isdn modem, cable modem, token ring interface, satellite transmission interface (e.g. “direct PC”), or other interfaces for coupling a computer system to other computer systems.
  • the interface can include one or more input and/or output devices.
  • the I/O devices can include, by way of example but not limitation, a keyboard, a mouse or other pointing device, disk drives, printers, a scanner, and other input and/or output devices, including a display device.
  • the display device can include, by way of example but not limitation, a cathode ray tube (CRT), liquid crystal display (LCD), or some other applicable known or convenient display device.
  • CTR cathode ray tube
  • LCD liquid crystal display
  • controllers of any devices not depicted in the example of FIG. 19 reside in the interface.
  • the computer system 1900 can be controlled by operating system software that includes a file management system, such as a disk operating system.
  • a file management system such as a disk operating system.
  • operating system software with associated file management system software is the family of operating systems known as Windows® from Microsoft Corporation of Redmond, Wash., and their associated file management systems.
  • Windows® from Microsoft Corporation of Redmond, Wash.
  • Windows® is the family of operating systems known as Windows® from Microsoft Corporation of Redmond, Wash.
  • Windows® Windows® from Microsoft Corporation of Redmond, Wash.
  • Linux operating system is another example of operating system software with its associated file management system software.
  • the file management system is typically stored in the non-volatile memory and/or drive unit and causes the processor to execute the various acts required by the operating system to input and output data and to store data in the memory, including storing files on the non-volatile memory and/or drive unit.
  • the machine operates as a standalone device or may be connected (e.g., networked) to other machines.
  • the machine may operate in the capacity of a server or a client machine in a client-server network environment, or as a peer machine in a peer-to-peer (or distributed) network environment.
  • the machine may be a server computer, a client computer, a personal computer (PC), a tablet PC, a laptop computer, a set-top box (STB), a personal digital assistant (PDA), a cellular telephone, an iPhone, a Blackberry, a processor, a telephone, a web appliance, a network router, switch or bridge, or any machine capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by that machine.
  • PC personal computer
  • PDA personal digital assistant
  • machine-readable medium or machine-readable storage medium is shown in an exemplary embodiment to be a single medium, the term “machine-readable medium” and “machine-readable storage medium” should be taken to include a single medium or multiple media (e.g., a centralized or distributed database, and/or associated caches and servers) that store the one or more sets of instructions.
  • the term “machine-readable medium” and “machine-readable storage medium” shall also be taken to include any medium that is capable of storing, encoding or carrying a set of instructions for execution by the machine and that cause the machine to perform any one or more of the methodologies of the presently disclosed technique and innovation.
  • routines executed to implement the embodiments of the disclosure may be implemented as part of an operating system or a specific application, component, program, object, module or sequence of instructions referred to as “computer programs.”
  • the computer programs typically comprise one or more instructions set at various times in various memory and storage devices in a computer, and that, when read and executed by one or more processing units or processors in a computer, cause the computer to perform operations to execute elements involving the various aspects of the disclosure.
  • machine-readable storage media machine-readable media, or computer-readable (storage) media
  • recordable type media such as volatile and non-volatile memory devices, floppy and other removable disks, hard disk drives, optical disks (e.g., Compact Disk Read-Only Memory (CD ROMS), Digital Versatile Disks, (DVDs), etc.), among others, and transmission type media such as digital and analog communication links.
  • CD ROMS Compact Disk Read-Only Memory
  • DVDs Digital Versatile Disks
  • transmission type media such as digital and analog communication links.
  • the words “comprise,” “comprising,” and the like are to be construed in an inclusive sense, as opposed to an exclusive or exhaustive sense; that is to say, in the sense of “including, but not limited to.”
  • the terms “connected,” “coupled,” or any variant thereof means any connection or coupling, either direct or indirect, between two or more elements; the coupling of connection between the elements can be physical, logical, or a combination thereof.
  • the words “herein,” “above,” “below,” and words of similar import when used in this application, shall refer to this application as a whole and not to any particular portions of this application.
  • words in the above Detailed Description using the singular or plural number may also include the plural or singular number respectively.
  • the word “or,” in reference to a list of two or more items, covers all of the following interpretations of the word: any of the items in the list, all of the items in the list, and any combination of the items in the list.

Abstract

Methods for configuring a system to collect and aggregate datasets are disclosed. One embodiment includes, identifying a data source in the system from where dataset is to be collected, configuring a machine in the system that generates the dataset to be collected, to send the dataset to the data source, identifying an arrival location where the dataset that is collected is to be aggregated or written, and/or configuring an agent node by specifying a source for the agent node as the data source in the system and specifying a sink for the agent node as the arrival location.

Description

    CLAIM OF PRIORITY
  • This application claims priority to U.S. Provisional Patent Application No. 61/319,816 entitled “CONFIGURING A SYSTEM TO COLLECT AND AGGREGATE DATASETS”, which was filed on Mar. 31, 2010, Attorney Docket No. 68784-8002.US00, the contents of which are expressly incorporated by reference herein.
  • BACKGROUND
  • Increasingly large amounts of data are generated with expanded user base of online applications such as social networks. Online activity of users interacting with one another, or with applications generates valuable information about users. Such data and analytics therein extracted can potentially provide service providers with useful information, for example, regarding traffic, content popularity, trends, etc. Additionally, advertisers can potentially glean information regarding user information and interests.
  • BRIEF DESCRIPTION OF THE DRAWINGS
  • FIG. 1 illustrates a block diagram of client devices that generate datasets (log data) to be collected and aggregated for storage in the storage device via interacting nodes in various tiers, in some instances, through a network.
  • FIG. 2A depicts a diagram showing one example configuration of nodes that are controlled/configured by a master and dataflow therein from a data source to a data sink (e.g., storage).
  • FIG. 2B depicts diagrams showing examples of configuration of nodes controlled/configured by a master and the dataflow therein from a data source to a data sink.
  • FIG. 3A depicts a diagram showing one example configuration of a node (nodes) controlled by a master residing on the same machine and dataflow therein from a data source to a data sink.
  • FIG. 3B depicts a diagram showing one example configuration of a node controlled by a master residing on a different machine and dataflow therein from a data source to a data sink.
  • FIG. 4 depicts a diagram showing an example configuration of nodes in multiple tiers on the same machine controlled by multiple masters residing on different machines and dataflow therein from data sources to a data sink.
  • FIG. 5 depicts a diagram showing an example configuration of nodes in multiple tiers on different machines controlled by multiple masters residing on different machines and dataflow therein from data sources to a data sink.
  • FIG. 6 depicts a block diagram illustrating example components of a node through which dataflow occur.
  • FIG. 7 depicts a block diagram illustrating example components of a decorator module in a node.
  • FIG. 8 depicts a block diagram illustrating example components of a master that controls the nodes in the system.
  • FIG. 9 depicts an example of an extensible data model used to represent events or data flow in a distributed system from a source to a sink.
  • FIG. 10A depicts a screenshot of an example user interface showing node status and configurations with node mappings.
  • FIG. 10B depicts a screenshot of an example user interface for configuring a node.
  • FIG. 10C depicts a screenshot of an example user interface for configuring multiple nodes.
  • FIG. 10D depicts a screenshot of an example user interface for issuing raw commands.
  • FIG. 10E depicts a screenshot of an example user interface showing statistics for a node.
  • FIG. 10F depicts a screenshot of an example user interface showing statistics for a node.
  • FIG. 11 depicts a flowchart of an example process of facilitating collecting and aggregating datasets that are machine or user-generated for analysis.
  • FIG. 12 depicts a flowchart of an example process for configuring a system to collect and aggregate datasets.
  • FIG. 13 depicts a flowchart of an example for scaling the system by implementing multiple masters on multiple machines.
  • FIG. 14 depicts a flowchart of another example process for configuring a system to collect and aggregate datasets and to perform analytics on the datasets.
  • FIG. 15 depicts a flowchart of an example process for dynamically processing an event including a dataset that is streamed from a source to a sink via nodes.
  • FIG. 16 depicts a flowchart of an example process for collecting and aggregating datasets for storage in a file system with fault tolerance.
  • FIG. 17 depicts a flowchart of an example process for deleting the WAL in response to acknowledgement of receipt by the master.
  • FIG. 18 depicts a flowchart of an example process for collecting and aggregating datasets with fault tolerance using a store on failure mechanism.
  • FIG. 19 shows a diagrammatic representation of a machine in the example form of a computer system within which a set of instructions, for causing the machine to perform any one or more of the methodologies discussed herein, may be executed.
  • DETAILED DESCRIPTION
  • The following description and drawings are illustrative and are not to be construed as limiting. Numerous specific details are described to provide a thorough understanding of the disclosure. However, in certain instances, well-known or conventional details are not described in order to avoid obscuring the description. References to one or an embodiment in the present disclosure can be, but not necessarily are, references to the same embodiment; and, such references mean at least one of the embodiments.
  • Reference in this specification to “one embodiment” or “an embodiment” means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the disclosure. The appearances of the phrase “in one embodiment” in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. Moreover, various features are described which may be exhibited by some embodiments and not by others. Similarly, various requirements are described which may be requirements for some embodiments but not other embodiments.
  • The terms used in this specification generally have their ordinary meanings in the art, within the context of the disclosure, and in the specific context where each term is used. Certain terms that are used to describe the disclosure are discussed below, or elsewhere in the specification, to provide additional guidance to the practitioner regarding the description of the disclosure. For convenience, certain terms may be highlighted, for example using italics and/or quotation marks. The use of highlighting has no influence on the scope and meaning of a term; the scope and meaning of a term is the same, in the same context, whether or not it is highlighted. It will be appreciated that same thing can be said in more than one way.
  • Consequently, alternative language and synonyms may be used for any one or more of the terms discussed herein, nor is any special significance to be placed upon whether or not a term is elaborated or discussed herein. Synonyms for certain terms are provided. A recital of one or more synonyms does not exclude the use of other synonyms. The use of examples anywhere in this specification including examples of any terms discussed herein is illustrative only, and is not intended to further limit the scope and meaning of the disclosure or of any exemplified term. Likewise, the disclosure is not limited to various embodiments given in this specification.
  • Without intent to further limit the scope of the disclosure, examples of instruments, apparatus, methods and their related results according to the embodiments of the present disclosure are given below. Note that titles or subtitles may be used in the examples for convenience of a reader, which in no way should limit the scope of the disclosure. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains. In the case of conflict, the present document, including definitions will control.
  • Embodiments of the present disclosure include systems and methods for configuring a system to collect aggregate datasets.
  • FIG. 1 illustrates a block diagram of client devices 102A-N that generate datasets (log data) to be collected and aggregated for storage in the storage device 108 via interacting nodes in various tiers, in some instances, through a network 106.
  • The client devices 102A-N can be any system and/or device, and/or any combination of devices/systems that is able to establish a connection with another device, a server and/or other systems. The client devices 102A-N typically include display or other output functionalities to present data exchanged between the devices to a user. For example, the client devices and content providers can be, but are not limited to, a server desktop, a desktop computer, a thin-client device, an interne kiosk, a computer cluster, a mobile computing device such as a notebook, a laptop computer, a handheld computer, a mobile phone, a smart phone, a PDA, a Blackberry device, a Treo, and/or an iPhone, etc. In one embodiment, the client devices 102A-N are coupled to a network 106. In some embodiments, the client devices may be directly connected to one another.
  • In one embodiment, users interact with user devices 102A-N (e.g., machines or devices). As a results of the user interaction, the devices 102A-N can generate datasets such as log files to be collected and aggregated. The file can include logs, information, and other metadata about clicks, feeds, status updates, data from applications, and associated properties and attributes.
  • User devices 102A-N can have nodes executing or running thereon that collect the datasets that are user-generated or machine-generated, for example, based on user-interaction with applications or websites running on the devices. Such nodes can interact and/or communicate with one or more other nodes (e.g., either running on the same device/machine or another device/machine (e.g., machine/device 104) to facilitate collection and aggregation of datasets thus generated. In one embodiment, the datasets are eventually written to a file and stored, for example, in storage (e.g., repository 130) on a physical disk.
  • Additionally, functionalities and properties can be assigned to the nodes such that various analytics can be performed on the collected dataset and additional information can be extracted or embedded. The dataflow among nodes can be configured at a master. In one embodiment, the nodes executed on the machines 102 or 104 can contact the master(s) to obtain configuration information, which have been set by default or configured by a user The master can be executed on the same devices 102A-N, 104, or at the host device 100. One or multiple masters can be involved in the mapping of data flow among the nodes and various machines.
  • Some examples of architectural configurations among nodes, machines, and masters are illustrated and described with further reference to the example of FIG. 2-5. The functionalities of the nodes and configuration are described with further reference to the examples of FIG. 6-8.
  • The network 106, over which the client devices 102A-N, 104, host, and the nodes and masters therein communicate may be a telephonic network, an open network, such as the Internet, or a private network, such as an intranet and/or the extranet. For example, the Internet can provide file transfer, remote log in, email, news, RSS, and other services through any known or convenient protocol, such as, but is not limited to the TCP/IP protocol, Open System Interconnections (OSI), FTP, UPnP, iSCSI, NSF, ISDN, PDH, RS-232, SDH, SONET, etc.
  • The network 106 can be any collection of distinct networks operating wholly or partially in conjunction to provide connectivity to the client devices, host server, and may appear as one or more networks to the serviced systems and devices. In one embodiment, communications to and from the client devices 102A-N can be achieved by, an open network, such as the Internet, or a private network, such as an intranet and/or the extranet. In one embodiment, communications can be achieved by a secure communications protocol, such as secure sockets layer (SSL), or transport layer security (TLS).
  • The term “Internet” as used herein refers to a network of networks that uses certain protocols, such as the TCP/IP protocol, and possibly other protocols such as the hypertext transfer protocol (HTTP) for hypertext markup language (HTML) documents that make up the World Wide Web (the web). Content is often provided by content servers, which are referred to as being “on” the Internet. A web server, which is one type of content server, is typically at least one computer system which operates as a server computer system and is configured to operate with the protocols of the World Wide Web and is coupled to the Internet. The physical connections of the Internet and the protocols and communication procedures of the Internet and the web are well known to those of skill in the relevant art. For illustrative purposes, it is assumed the network 106 broadly includes anything from a minimalist coupling of the components illustrated in the example of FIG. 1, to every component of the Internet and networks coupled to the Internet.
  • In addition, communications can be achieved via one or more wireless networks, such as, but is not limited to, one or more of a Local Area Network (LAN), Wireless Local Area Network (WLAN), a Personal area network (PAN), a Campus area network (CAN), a Metropolitan area network (MAN), a Wide area network (WAN), a Wireless wide area network (WWAN), Global System for Mobile Communications (GSM), Personal Communications Service (PCS), Digital Advanced Mobile Phone Service (D-Amps), Bluetooth, Wi-Fi, Fixed Wireless Data, 2G, 2.5G, 3G networks, enhanced data rates for GSM evolution (EDGE), General packet radio service (GPRS), enhanced GPRS, messaging protocols such as, TCP/IP, SMS, MMS, extensible messaging and presence protocol (XMPP), real time messaging protocol (RTMP), instant messaging and presence protocol (IMPP), instant messaging, USSD, IRC, or any other wireless data networks or messaging protocols.
  • The client devices 102A-N can be coupled to the network (e.g., Internet) via a dial up connection, a digital subscriber loop (DSL, ADSL), cable modem, and/or other types of connection. Thus, the client devices 102A-N can communicate with remote servers (e.g., web server, host server, mail server, and instant messaging server) that provide access to user interfaces of the World Wide Web via a web browser, for example.
  • The repository 130 can store software, descriptive data, images, system information, drivers, collected datasets, aggregated datasets, log files, analytics of collected datasets, enriched datasets, etc. The repository may be managed by a database management system (DBMS), for example but not limited to, Oracle, DB2, Microsoft Access, Microsoft SQL Server, MySQL, FileMaker, etc.
  • The repositories can be implemented via object-oriented technology and/or via text files, and can be managed by a distributed database management system, an object-oriented database management system (OODBMS) (e.g., ConceptBase, FastDB Main Memory Database Management System, JDOlnstruments, ObjectDB, etc.), an object-relational database management system (ORDBMS) (e.g., Informix, OpenLink Virtuoso, VMDS, etc.), a file system, and/or any other convenient or known database management package. In one embodiment, the repository is managed by a distributed file system or network file system that allows access to files from multiple hosts/machines over a network. The distributed file system can include by way of example, the Hadoop Distributed File system (HDFS). Other file systems can be used as well, for example, through integration of Hadoop's interface which provides an abstraction layer for the file system. For example, a local file system where a node resides can be used. The HDFS native distributed file system can also be used. In addition, S3 (a remote file system hosted by Amazon web services), FTP, and KFS (Kosmos file system—another distributed file system) can also be used. Clients can also write to different file systems (NFS), or other file systems.
  • In general, the user devices 102 and 104 are able to write files (e.g., files including by way of example, collected and aggregated datasets/logs/log files) to the repository 130, either through the network 106 or without utilizing the network 106. The host server 100 can be implemented on a known or convenient computer system, such as is illustrated in FIG. 19.
  • FIG. 2A depicts a diagram showing one example configuration 200 of nodes 202 and 204 that are controlled/configured by a master 206 and dataflow therein from a data source 201 to a data sink 207 (e.g., repository 208).
  • In order to collect dataset on a machine for aggregation and to perform any analytics, the node 202 can be mapped to the data source 201 on the machine that generates that dataset. The dataset can include data, metadata, complex data, including logs of clicks, social networking sites, feeds, status updates, logs from local or remote applications, etc. The data source can include, by way of example, but not limitation, a syslog port, an incoming network connection, and an IRC channel, output from execution of a program, a text file. In addition, a data source can include a Hadpp sequence file formatted file.
  • The node 202 can map the data source 201 to a receiving location such as a data sink 203. In one embodiment, a master 206 controls the mapping of the data source 201 to the receiving location 203 via the node 202. In addition, data flow of the dataset from the data source 201 to the receiving location 203 is configurable and reconfigurable at the master 206 by a user (e.g., system administrator, developer, etc.). The node 202 can contact the master 206 to obtain its mapping/configuration information.
  • Having nodes contact the master 206 to obtain configuration information allows node configuration to dynamically change without having to login and to restart the daemon (e.g., a server process on a UNIX-based machine). The node's configuration can be changed to a new one without taking the system offline. In one embodiment, nodes are configured using a web interface to the master. Screenshots of example interfaces are illustrated with further reference to the example of FIG. 10.
  • In general, one or more masters can be implemented using one or more machines, as illustrated in subsequent figures. The master 206 can be executed on the machine on which the collected dataset is received or generated or one or more different machines. Different types of architectural configurations for the master 206 are illustrated in subsequent figures.
  • A receiving location, represented by data sink 203, can include, by way of example, not limitation, a network connection, a physical disk, a console, a text file. The receiving location can also include, a file in the Hadoop sequence file format, an HDFS path, a port, a monitoring or alerting application, and/or an IRC.
  • In general, the collected dataset from the data source 201 can be aggregated at the receiving location (data sink 203). The receiving location (e.g., or data sink 203) may further be mapped to node 204 (e.g., a collector node), where the dataset collected from the dataset can be aggregated. In some instances, analytics can be performed on the dataset upon collection at node 202 and/or aggregation at node 204. In one embodiment, the master 206 controls the functions/properties of the nodes 202 and/or 204.
  • For example, the nodes 202/204 (e.g., logical nodes) can be associated with properties and/or functions, including but not limited to, adding encryption or digital signature, performing batching/unbatching, performing compression/uncompression, generating a checksum of a dataset, performing sampling, performing benchmarking (e.g., by injecting benchmark tags). In addition, the analytic that are performed can include, feature extraction, filtering, transformation, generating aggregate counts, statistics, etc.
  • The node 204, in aggregating the dataset, can also be mapped to a data source 205 and a data sink 207, as controlled and configured at the master 206. To aggregate data collected by the node 202, the data source 205 of the node 204 can be mapped to the sink 203 of node 202. Additional functions and properties of nodes are illustrated with further reference to the example of FIG. 6-7.
  • In one embodiment, the node 204 (e.g., a collector node) writes the aggregated data to a storage location. The storage location can be localized or distributed among multiple machines and the dataset is stored redundantly. The data sink 207 of the node 204 can be mapped to a repository 208, by the master 206, for example. The repository 208 may be managed by a file system. The file system can be distributed (e.g., the the Hadoop Distributed File System (HDFS)). Results of any analytics performed by nodes 202 and/or 204 can also be written to storage. Data/metadata extracted from the collected dataset may be written to storage 208 as well.
  • Note that node 202, although illustrated in the example of FIG. 2A as having being mapped to a single node 204, can be mapped to multiple nodes, as further illustrated in the example of FIG. 2B.
  • FIG. 2B depicts diagrams showing examples of configuration of nodes controlled/configured by a master and the dataflow therein from a data source to a data sink.
  • In the example flow 215, an agent node 212 can send incoming dataset/event from a data source 211 to multiple collector nodes (e.g., node 214 and node 216) which further maps the data flow to the repository 218. In one embodiment, the collector nodes are failover nodes for handling failures when appending new events. Failover sinks can be used to specify alternate collectors to contact in the event the assigned collector fails. Fail over sinks can also be used to specify a local disk sink to store data until the assigned collector recovers. For example, if node 214 goes down, the agent node 212 can deliver events via collector agent 216 to the repository 218 or any other receiving location or destination. In some instances, the collector 216 can queue their logs until node 214 comes back online. The failover assignment can be specified through the master 220. In addition, in one embodiment, the failover mechanism is built in by default by disbursing collector node assignments evenly among agent nodes. This can mitigate the chances of one collector from being overloaded when one collector fails.
  • Similarly, in the example flow 225, multiple collectors 224 are assigned to receive events/data flow from agent node 222. The collectors 224 can store the event/data flow in the repository 228. The collectors 224 may be failover chains. In addition, the collectors 224 can be fan out sinks, each of which receives the incoming event from the data source 221. The fan out sinks can be configured by the master 230 as collectors 224. The multiple collectors 224 as fan out sinks can be used for data replication for processing data off a main data flow path.
  • FIG. 3A depicts a diagram showing one example configuration of a node (nodes) controlled by a master 306 residing on the same machine 300 and dataflow therein from a data source 301 to a data sink 305.
  • The agent node 302 and the master 306 used for controlling, configuring, and/or monitoring the node 302 can be implemented on the same machine 300. The master 306 can be used to specify functions and/or properties of the agent node 302 in sending an incoming event/data set from the source 310 to the sink 305.
  • In one embodiment, a collector node 304 is also implemented on the same machine 304 as the agent node 302 and the master 306. The master 306 can control, configure, and/or monitor both nodes 302 and 304 in mapping events/data flow fro the source 301 to the sink 305.
  • FIG. 3B depicts a diagram showing example configurations of a node 312 or nodes 312 and 314 executed on a machine 320 and controlled by a master 316 which is executed on a different machine 330 and dataflow therein from a data source 311 to a data sink 315.
  • The master 316 executed on machine 330 controls, configures, and/or monitors the node 312 on the machine 320 in sending data sets/events from the data source 311 to the data sink 315. Alternatively, the collector node 314 which is mapped to receive datasets/events from the node 312 can also be executed on machine 320. The collector node can be configured to send datasets received from the node 312 to a data sink 315. The master 316 can also control, configure, and/or monitor the collector node 314.
  • However, having a single master is a single point of failure in the system. If the master 316 fails, the ability to update nodes (e.g., nodes 314 and/or 314), configure nodes, control, and/or monitor nodes, etc. goes down. Thus, in one embodiment, multiple masters can be implemented on multiple machines to maintain a consistent state in the system. In addition, using multiple masters on multiple machines allows the system to scale with increasing amount of log data and events to process and analyze.
  • Moreover, using multiple collectors can also increase log collection throughput and can improve timeliness of event/dataset delivery since more collector nodes are available at any given time during operation. Since data collection is parallelizable, the load from agent nodes can be shared and distributed among many collector nodes. Example architectures with multiple masters and multiple collectors are illustrated in FIG. 4-5.
  • FIG. 4 depicts a block diagram showing an example configuration of nodes in multiple tiers on the same machine 400 which controlled by multiple masters 406 and 408 residing on different machines 420 and 430 respectively, and dataflow therein from data sources 401, 403, and 405 to a data sink 407.
  • The example system of FIG. 4 utilizes multiple agent nodes 402 in an agent tier 412 and multiple collector nodes 404 in a collector tier 414. In one embodiment, the nodes 402 in the agent tier 412 are located on the machine 400 which is also producing logs that are being collected. For example, the data source 401 of the node 402 can be configured to be a port on the machine 400. The sink of the node 402 can be configured to be node 404 in the collector tier 414.
  • In one embodiment, collector nodes 404 can listen and receive data from multiple agent nodes 412 and aggregate logs. In addition, collector nodes 414 can further write the logs to a data sink 407 in the storage tier 416. In reliable mode agent nodes generally write data to the respective local disks. This data might not be deleted until after it knows that the data has reached its final destination. Collectors generally don't write to their local disk—they actually forward data to the storage tier which eventually points the collector to storage tier nodes that write to storage tier disks. An agent is generally on a system that is responsible for doing other work (serving web pages) and thus agent nodes typically strive to have low resource utilization (e.g., cpu, memory, disk). Collector nodes are generally dedicated to processing large amounts of data and can use more expensive algorithms because its processing units are generally less occupied with other tasks. Note that the data sink 407 may or may not be a storage unit/device. For example, the collector nodes 404 can also send the data sets downstream to cluster for further processing, a network, etc.
  • FIG. 5 depicts a diagram showing an example configuration of nodes 502 and 504 in multiple tiers on different machines controlled by multiple masters 506 residing on different machines and dataflow therein from data sources 501 to a data sink 507.
  • Agent nodes 502 can be executed on different machines and map incoming data/events to collector nodes 504. The multiple machines may be in a cluster and can include web servers. In general, the multiple machines generate the log data to be collected and each of the multiple machines is associated with at least one agent node to facilitate data collection and/or aggregation. In one embodiment, the agent nodes 502 can forward the collected log data or events to a collector tier 512 which includes at least one collector node 504 but in some instances, multiple collector nodes 504 which can combine streams of log data/events into larger streams, for example, for writing to the storage tier 516. The collector node(s) 504 can aggregate the log data collected from the machines on which the agent nodes are executed. The collector nodes 504 can also be executed on different machines with each machine having various numbers of collector nodes 504, which passes events on to the data sink 507.
  • In one embodiment, the collector node(s) 504 outputs the log data to multiple destinations in the same or different formats. Some of the supported formats include, by way of example not limitation, raw text file, json (a web format), avro binary (an apache serialization format), a hadoop native sequence file, text emulating a syslog formatted entry, an apache web server formatted entry, and a “log4j” formatting pattern, etc. File formats are the format for batches of records. Record formats are the formatting for individual events/records. A collector writing to storage tier can be responsible for file formats. The work for formatting individual records however can be pushed upstream. As an example, let's say the agent gets an event that has raw data An agent node can extract features and/or reformat the data found in the event/record and write this intermediate result to the metadata table. The downstream collector node can use the preformatted data and write each formatted record in the file format. One embodiment of the topology includes a storage tier 516 having a storage system coupled to the machines in the system. The storage system can include a distributed file system to which the collector node(s) 504 in the collector tier 514 stores the log data aggregated from the set of machines in the system.
  • Note that analytics can be performed on the log data/aggregated events by the agent nodes 502 and/or the collector nodes 504. The master 506 can be used to control/configure the types of analytics that are performed.
  • Agent nodes 502 and collector nodes 504 can be configured, re-configured via the masters 506, which can be executed on multiple machines that are different from the machines that the collector 504 and agent nodes 502 are executed on. In one embodiment, the masters 506 specify the configuration/topology of the system 500 and the agent node(s) 502 and the collector node(s) 504 contact the master 506 to retrieve configuration information. For example, the master 506 can specify that an agent data sink for the agent node 502 is a collector data source for the collector node 504 in the collector tier 514 and that a collector data sink 507 for the collector node 504 is the distributed file system in the storage tier 516. In addition, the master 506 can be used by a user to configure or reconfigure the agent data sink and agent data source and/or to configure or reconfigure the collector data source and the collector data sink.
  • FIG. 6 depicts a block diagram illustrating example components of a node 602 through which dataflow occur.
  • The node 602 can include a source specifier module 604, a sink specifier module 606, a configuration file 610, and/or a decorator module 620.
  • As used in this paper, a “module” or an “engine” includes a dedicated or shared processor and, typically, firmware or software modules that are executed by the processor. Depending upon implementation-specific or other considerations, an engine can be centralized or its functionality distributed. An engine can include special purpose hardware, firmware, or software embodied in a computer-readable medium for execution by the processor. As used in this paper, a computer-readable medium is intended to include all mediums that are statutory (e.g., in the United States, under 35 U.S.C. 101), and to specifically exclude all mediums that are non-statutory in nature to the extent that the exclusion is necessary for a claim that includes the computer-readable medium to be valid. Known statutory computer-readable mediums include hardware (e.g., registers, random access memory (RAM), non-volatile (NV) storage, to name a few), but may or may not be limited to hardware.
  • One embodiment of the node includes a source specifier module 604 and a sink specifier module 606. The source specifier module 604 and sink specifier module 606 can be implemented, example, as software embodied in a computer-readable medium or computer-readable storage medium on a machine, in firmware, in hardware, in a combination thereof, or in any applicable known or convenient device or system. This and other modules or engines described in this specification are intended to include any machine, manufacture, or composition of matter capable of carrying out at least some of the functionality described implicitly, explicitly, or inherently in this specification, and/or carrying out equivalent functionality.
  • The source specifier module 604 can include an identifier or mapper to a source that produces or accepts events or data fed into the system. The node 604 reads or tails the event source. For example, the module 604 can detect, recognize, identify, and/or store a port ID (e.g., a port name, a TCP or UDP port, port number) and a file ID (e.g., a file name, a file path, etc.). The file can be any type of file, including but not limited to, messages, system log files, Hadoop log files, text files, etc. In addition, the event source can include, a console, a sequence file (e.g., serialized events in a sequence file format), a source that synthetically generates a specified number of messages of a select size.
  • The sink specifier module 606 includes an identifier or mapper to a location that consumes data or an event (e.g., a receiving or destination location for data or events). A sink can include, by way of example, a disk, a repository, a distributed file system on a disk (e.g., the Hadoop distributed file system), a console, a device/machine, and/or a network/network connection over which the data/event is to be forwarded over. Thus, the sink specifier module 606 can detect, recognize, identify, and or store, a machine/device ID, a port ID, a file ID, and/or a path ID. In one embodiment, the path ID is a distributed sequence file path.
  • The source and sink specifier modules can store or retrieve the configuration information stored in the configuration file 610. The configuration file 610 may be loaded with default values upon installation. In addition, the configuration file 610 may be updated by a master that controls the node 602. The updates the configuration file 610 may occur while the system is running or when the system is off.
  • Note that the node 602 may be assigned to have an agent role or a collector rule. In general, an agent node is co-located on machines/devices with the service/application that is generating, producing, creating events/logs/data. A collector node can receive, accept, check, listen for data from one or more agent nodes. A collector node can be mapped to the data sink of an agent node, for example, to aggregate logs/data and/or to extract analytics. A collector node can further write the dataset, events, and/or any generated events to storage (e.g., a distributed file system).
  • One embodiment of the node 602 includes a decorator module 620. The decorator module 620 can be implemented, example, as software embodied in a computer-readable medium or computer-readable storage medium on a machine, in firmware, in hardware, in a combination thereof, or in any applicable known or convenient device or system. This and other modules or engines described in this specification are intended to include any machine, manufacture, or composition of matter capable of carrying out at least some of the functionality described implicitly, explicitly, or inherently in this specification, and/or carrying out equivalent functionality. The decorator module 620 can add properties/functionalities of a sink and modify the data/event streams that pass through them, like a wrapper. For example, the decorator module 620 can be used to increase reliability, robustness of the system or to perform analytics. An example of the decorator module is illustrated in detail with further reference to the example of FIG. 7. A decorator can correspond to a sink that interposes between the source and the sink. The decorator processes events and then sends it down the pipeline to another sink (which may be another decorator wrapping another sink) on the same node/dataflow. Sources produce events, and pass them through the decorator to sink.
  • FIG. 7 depicts a block diagram illustrating example components of a decorator module 720 in a node.
  • The decorator module 720 can include, a write ahead module 702, a sampler module 704, an open module 706, an append module 708, a batch/unbatch module 710, and/or a GZIP/GUNZIP module 712.
  • As used in this paper, a “module” or an “engine” includes a dedicated or shared processor and, typically, firmware or software modules that are executed by the processor. Depending upon implementation-specific or other considerations, an engine can be centralized or its functionality distributed. An engine can include special purpose hardware, firmware, or software embodied in a computer-readable medium for execution by the processor. As used in this paper, a computer-readable medium is intended to include all mediums that are statutory (e.g., in the United States, under 35 U.S.C. 101), and to specifically exclude all mediums that are non-statutory in nature to the extent that the exclusion is necessary for a claim that includes the computer-readable medium to be valid. Known statutory computer-readable mediums include hardware (e.g., registers, random access memory (RAM), non-volatile (NV) storage, to name a few), but may or may not be limited to hardware.
  • One embodiment of the decorator module 720 includes a write ahead module 702. The decorator module 620 can be implemented, example, as software embodied in a computer-readable medium or computer-readable storage medium on a machine, in firmware, in hardware, in a combination thereof, or in any applicable known or convenient device or system. This and other modules or engines described in this specification are intended to include any machine, manufacture, or composition of matter capable of carrying out at least some of the functionality described implicitly, explicitly, or inherently in this specification, and/or carrying out equivalent functionality.
  • The write ahead module 702 provides reliability and durability and writes/stores events/datasets to persistent storage on the machine generating the event (writeAhead( . . . )) prior to forwarding them. In the event and any of the downstream nodes fails, the system has a copy of the dataset that can still be retrieved. In response to determining that the dataset/event has safely reached its destination, the copy can be deleted from persistent storage. The write ahead module 702 can also assign batch identifiers for batches of messages/events to track
  • One embodiment of the decorator module 720 includes a sampler module 704. The decorator module 704 can be implemented, example, as software embodied in a computer-readable medium or computer-readable storage medium on a machine, in firmware, in hardware, in a combination thereof, or in any applicable known or convenient device or system. This and other modules or engines described in this specification are intended to include any machine, manufacture, or composition of matter capable of carrying out at least some of the functionality described implicitly, explicitly, or inherently in this specification, and/or carrying out equivalent functionality.
  • The sampler module 704 can sample data input or events at select intervals (e.g., intervalSampler(N)). For example, the sample module 704 can be configured to send every nth element/event from the source to the data sink. In one embodiment, the sampler module 704 samples based on a probability such that each event or message has a specified chance of being forwarded (e.g., probSampler (p)). In addition, the sampler module 704 can be a reservoir sampler (e.g., reserverSampler (K) such that when flushed, a selected number (K) of elements are forwarded and each event that passes through has the same probability of being selected for forwarding.
  • One embodiment of the decorator module 720 includes an open module 706 (or re-try open module). The open module 706 can be implemented, example, as software embodied in a computer-readable medium or computer-readable storage medium on a machine, in firmware, in hardware, in a combination thereof, or in any applicable known or convenient device or system. This and other modules or engines described in this specification are intended to include any machine, manufacture, or composition of matter capable of carrying out at least some of the functionality described implicitly, explicitly, or inherently in this specification, and/or carrying out equivalent functionality. In one embodiment, the open module 706, in reading or opening a file or incoming event, continues to re-try the opening process if the initial attempt fails.
  • One embodiment of the decorator module 720 includes an append module 708 (or re-try append module). The append module 708 can be implemented, example, as software embodied in a computer-readable medium or computer-readable storage medium on a machine, in firmware, in hardware, in a combination thereof, or in any applicable known or convenient device or system. This and other modules or engines described in this specification are intended to include any machine, manufacture, or composition of matter capable of carrying out at least some of the functionality described implicitly, explicitly, or inherently in this specification, and/or carrying out equivalent functionality. In one embodiment, the append module 708, can append events to a target. In the event that the append fails, the module 708 can close the target, re-open it and attempt to append again. In general, sinks and decorator modules include open, append, and close functions. Typically, if one of these operations fail, the modules can give up after a first try. Using the open module (or re-try open module) or the append module (e.g., re-try append module), the sink/decorator can be configured to keep trying the failed operation (e.g.,retry opening or appending).
  • One embodiment of the decorator module 720 includes a batch/unbatch module 710. The batch/unbatch module 710 can be implemented, example, as software embodied in a computer-readable medium or computer-readable storage medium on a machine, in firmware, in hardware, in a combination thereof, or in any applicable known or convenient device or system. This and other modules or engines described in this specification are intended to include any machine, manufacture, or composition of matter capable of carrying out at least some of the functionality described implicitly, explicitly, or inherently in this specification, and/or carrying out equivalent functionality.
  • The batch/unbatch module 710 can buffer a select number of events/message together to be forwarded or sent as a single aggregate event. The batch/unbatch module 710 can take an aggregate event and split and forward the original events.
  • One embodiment of the decorator module 720 includes a GZIP/GUNZIP module 712. The GZIP/GUNZIP module 712 can be implemented, example, as software embodied in a computer-readable medium or computer-readable storage medium on a machine, in firmware, in hardware, in a combination thereof, or in any applicable known or convenient device or system. This and other modules or engines described in this specification are intended to include any machine, manufacture, or composition of matter capable of carrying out at least some of the functionality described implicitly, explicitly, or inherently in this specification, and/or carrying out equivalent functionality. The GZIP/ZIP module 712 can zip or unzip an event/message or multiple events/messages.
  • In general, multiple decorators can be assigned to an event, message, or dataset. For example, decorators can be combined to batch multiple messages together and compress the messages before forwarding them to the mapped data sink.
  • FIG. 8 depicts a block diagram illustrating example components of a master 800 that controls the nodes in the system.
  • The master 800 can include a node configurator module 802, a system health monitor 804, a consistency manager 806, a map module 808, and/or an acknowledger module 810. The master 800 can include additional or less modules. The master 800, although illustrated as being a discrete component, can be implemented on multiple machines, as shown in the examples of FIG. 4-5 on full or partial redundancy.
  • The master 800, as implemented on a single or multiple machines can perform as a central control and monitor for the system. The master 800 can track and monitor the status, health of nodes, and the data flow in the system. In tracking and monitoring system status, the master 800 can automatically perform any system re-configurations that may be needed to resolve node failure or load imbalance, for example.
  • The master 800 can identify, detect, or be used to specify new nodes or deleted nodes in the system and perform any adjustments in data flow by updating the mapping of data sinks and sources. In general, the nodes contact the master 800 to retrieve configuration information. When a node contacts the master, the node detects that the configuration version has been changed and can instantiate and activate the configuration. This allows updates to the system and mapping to be performed in dynamically, in operation, without starting or rebooting the system. For example, a node's configuration can be updated during operation.
  • Additionally, a user can reconfigure the setting by accessing the master 800. One of accessing the master 800 is through the master's web page. Examples of screenshots are illustrated in FIG. 10A-F. The configuration link is displayed on the master's web interface. In the interface, for example, node names are depicted and sources and sinks can be viewed, updated, or deleted. Once a configuration has been changed, the version identifier or stamp on the master's web page will be updated.
  • As used in this paper, a “module” or an “engine” includes a dedicated or shared processor and, typically, firmware or software modules that are executed by the processor. Depending upon implementation-specific or other considerations, an engine can be centralized or its functionality distributed. An engine can include special purpose hardware, firmware, or software embodied in a computer-readable medium for execution by the processor. As used in this paper, a computer-readable medium is intended to include all mediums that are statutory (e.g., in the United States, under 35 U.S.C. 101), and to specifically exclude all mediums that are non-statutory in nature to the extent that the exclusion is necessary for a claim that includes the computer-readable medium to be valid. Known statutory computer-readable mediums include hardware (e.g., registers, random access memory (RAM), non-volatile (NV) storage, to name a few), but may or may not be limited to hardware.
  • One embodiment of the master includes a node configurator module 802. The node configurator module 802 can be implemented, example, as software embodied in a computer-readable medium or computer-readable storage medium on a machine, in firmware, in hardware, in a combination thereof, or in any applicable known or convenient device or system. This and other modules or engines described in this specification are intended to include any machine, manufacture, or composition of matter capable of carrying out at least some of the functionality described implicitly, explicitly, or inherently in this specification, and/or carrying out equivalent functionality.
  • In one embodiment, the node configurator module 802 detects, updates, identifies, stores configuration settings of the nodes in the system. For example, the source and sink mappers can configure the data source(s) and/or data sink(s) for each node. In addition, the module 802 can assign and/or track functions/properties (e.g., analytics functions) that a node can perform.
  • One embodiment of the master includes a system health monitor 804. The system health monitor 804 can be implemented, example, as software embodied in a computer-readable medium or computer-readable storage medium on a machine, in firmware, in hardware, in a combination thereof, or in any applicable known or convenient device or system. This and other modules or engines described in this specification are intended to include any machine, manufacture, or composition of matter capable of carrying out at least some of the functionality described implicitly, explicitly, or inherently in this specification, and/or carrying out equivalent functionality.
  • The system health monitor 804 can track, detect, determine, monitor, system traffic, node health/status, dataflow, load balance/imbalance in the system. The master 800 can update/adjust data flow based on system traffic and any load imbalances. In addition, if certain nodes have failed, the master 800 can assign failover nodes such that data flow is not interrupted.
  • One embodiment of the master includes a consistency module 806. The consistency module 806 can be implemented, example, as software embodied in a computer-readable medium or computer-readable storage medium on a machine, in firmware, in hardware, in a combination thereof, or in any applicable known or convenient device or system. This and other modules or engines described in this specification are intended to include any machine, manufacture, or composition of matter capable of carrying out at least some of the functionality described implicitly, explicitly, or inherently in this specification, and/or carrying out equivalent functionality.
  • The consistency module 806 maintains consistency among multiple masters implemented on multiple machines in a system, by leveraging the consensus protocol and/or a gossip protocol. For example, the master 800 can maintain consistency using a consensus protocol for order critical state changes and using a gossip protocol for weak consistency among data including but not limited to node status updates, reporting information, and metrics information.
  • One embodiment of the master includes a map module 808 and an acknowledger module 810. The map module 808 and the acknowledger module 810 can be implemented, example, as software embodied in a computer-readable medium or computer-readable storage medium on a machine, in firmware, in hardware, in a combination thereof, or in any applicable known or convenient device or system. This and other modules or engines described in this specification are intended to include any machine, manufacture, or composition of matter capable of carrying out at least some of the functionality described implicitly, explicitly, or inherently in this specification, and/or carrying out equivalent functionality.
  • The map module 808 monitors and manages a set of tags associated with batches of messages to track which messages or batches or messages have been successfully sent to a receiving location or stored in a data repository.
  • In one embodiment, an agent node generates a batch comprising multiple messages from the log data and assigns a tag (e.g., can include a begin event tag and an end event tag) to the batch. The agent can also compute a checksum for the batch of multiple messages and stores he tag and the checksum to a write-ahead-log (WAL) in persistent storage of the machine from which the dataset was generated.
  • The agent node sends the batch of messages to a collector which verifies the checksum of the batch, and if successful, adds the tag to a map of tags for batches of messages. When the collector node determines that a batch of messages has successfully reached a receiving location or stored in storage/file system, the collector agent publishes the tag to the map module 808 of the master. Thus, the map module 808 includes identifiers for events, messages, or batches of messages that have reached their mapped locations. The master 800 can use the identifiers in the map module 808 to notify the agent nodes of batches that have been successfully stored or sent. In one embodiment, the acknowledger module 810, using the tags that are associated with batches in the map, acknowledges the agent node and the collector node of the batches that have been safely stored in the file system or sent down stream.
  • FIG. 9 depicts an example of an extensible data model used to represent events or data flow in a distributed system from a source to a sink.
  • The data model, for example, can include one or more timestamp fields 902 and 904, a priority field 906, a source host field 908, a body field 910, and/or a metadata table field 912. The data model is extensible and thus additional fields can be added on the fly during data flow. The data model is described with further reference to the example of FIG. 15.
  • FIG. 10A depicts a screenshot of an example user interface showing node status and configurations with node mappings. This user interface can be accessed from the master. FIG. 10B depicts a screenshot of an example user interface for configuring a node. The configuration page can be accessed from the master's web page. For example, a node to be configured can be selected from the list and its source/sink can be specified.
  • FIG. 10C depicts a screenshot of an example user interface for configuring multiple nodes. FIG. 10D depicts a screenshot of an example user interface for issuing raw commands. Commands can be used to modify the global state of the system. For example, the “config” command sets the configuration of a flow on the master, who in turn sends the info to a node. The “refresh” command takes an existing config and forces the node to reset (close then open). The “decommission” command removes a logical node. The “unmap” command removes a logical node from a physical node. The “spawn” command creates a new logical node (could be used to associate an unmapped logical node onto another physical machine). Confirmation that command was received by the master by reviewing the command history section of FIG. 10A. Confirmation that a node has installed the changes the command may have caused by looking at the web page in the examples of FIGS. 10E and 10F. FIG. 10E depicts a screenshot of an example user interface showing statistics for a node. The user interface shows the data flow configuration settings, and memory usage of a particular node. For example, this figure shows information about the node “LogicalNode[0]” and includes two analytics reports: a graph showing the number of incoming events and a graph showing a histogram of the frequency of certain words in the stream. The source in this example is Twitter, and the sinks for the node include the graphs. FIG. 10F depicts a screenshot of an example user interface showing statistics for a node. This figure illustrates the configuration info for this data flow, and similarly has an event graph and a histogram (showing an even distribution of substrings).
  • FIG. 11 depicts a flowchart of an example process of facilitating collecting and aggregating datasets that are machine or user-generated for analysis.
  • In process 1102, a dataset is collected on a machine on which the dataset is received or generated. The data set can include logs from an application that is local or remote, a service, a social networking site, feeds, status updates, etc. The dataset can be collected from a data source on the machine. In addition, datasets can be collected from machines in a cluster which can include web servers, for example. In process 1104, the dataset that is collected from the data source is aggregated at a receiving location. The receiving location can be, for example, a network, a network connection, a physical disk, a consol, a text file, a file in a sequence file format, an HDFS path, a port, a monitoring or alerting application, and/or an IRC. One or more receiving location can be specified.
  • In one embodiment, the mapping of the data source to the receiving location is controlled by a master. The master can be executed on a different machine than the machine on which the dataset is generated or received. The master can also be executed on the same machine as the one on which the dataset is received or generated. In one embodiment, the data source is mapped to a receiving location via a logical node. The logical node can contact the master to obtain configuration information including but not limited to, functions, data source, data sink, etc. In process 1106, analytics are performed on the dataset upon collection or aggregation. Analytics that can be performed include, by way of example, feature extraction, filtering, and transformation, generation of aggregate counts, and/or generation of statistics. The analytics can be performed in near real time relative to collection of the dataset. In process 1108, the dataset aggregated at the receiving location is written to a storage location. With IO there is typically a tradeoff between throughput and latency. Usually, sending lots of small things will take a lot longer than a few large things, even if the actual amount of payload is the same. This is generally due to overhead associated with each sent value (sent events in Flume's case). In one embodiment, delay (latency) can be added to help the scheduler improve on throughput.
  • The storage location can be distributed among multiple machines and the dataset is stored redundantly on a distributed file system
  • FIG. 12 depicts a flowchart of an example process for configuring a system to collect and aggregate datasets.
  • In process 1202, a client (e.g., a web page/interface or a command line interface, another program (a master), or a device (e.g., a laptop, a cell phone, a smartphone, iPhone, etc.) is launched on a machine to access a master for the system, for example, using the master's universal resource locator (URL). The master's web page can be used to view, change, and/or update a node's configuration, as illustrated in the example of FIG. 10A. The system may include multiple machines (e.g., in a cluster) which generate datasets that are to be collected. In process 1204, a data source in the system from where dataset is to be collected is identified.
  • In process 1206, a machine in the system that generates the dataset to be collected is configured to send the dataset to the data source. In process 1208, an arrival location where the dataset that is collected is to be aggregated or written is identified. In process 1210, an agent node is configured by specifying a source for the agent node as the data source in the system and specifying a sink for the agent node as the arrival location. In addition, the agent node is generally configurable to perform analytics on the dataset.
  • In process 1212, the arrival location is specified as a collector source of a collector node. In process 1214, a distributed file system is specified as a collector sink of the collector node. The distributed file system can be, for example, the Hadoop distributed file system (HDFS)
  • FIG. 13 depicts a flowchart of an example for scaling the system by implementing multiple masters on multiple machines.
  • A master provides a centralized management point to monitor and update data flows. Changes in the system are told to the master, and changes can come from the master. When new resources are added to the system, the information is updated at the master. In addition, if a user issues a command that changes global state, the master can be used such that the master can react to these state changes. In addition, the master can dynamically respond to system changes such as load imbalances, node failure, machine failure, new hardware, etc. For example, a user configures a new node to be a collector. The master detects this, and automatically reacts by changing the configuration of some nodes to send events downstream to the new collector. The master can do this by changing global state. In another example, if a collector node hasn't communicated with the master for a while. The master may decide that the machine might have fialed and automatically reacts by sending an event to an admin (via email/irc/im/sms, etc.). Similarly, in yet another example, nodes periodically report their performance metrics to the master (e.g., in the examples of FIGS. 10E, 10F). The reports indicate that a collector is being overloaded with work, and reacts to new metrics data by shifting some of the work from the overloaded collector to another with less load. This can be performed by changing global state.
  • Nodes in the system can be configured/reconfigured dynamically via the master. The nodes can be arranged in a three-tier topology (e.g., agent tier, collector tier, storage tier) or any other types of topology with any number of tiers. A user can use the master to configure the topology and the nodes, for example, by accessing a link on the web page for the master, as illustrate in the example screenshot of FIG. 10B.
  • Since the master provides centralized management, having only one master is a single point of failure in the system. Thus, in one embodiment, in process 1302, the system is scaled by implementing multiple masters on multiple machines.
  • In process 1304, consistency is maintained among the multiple masters using a consensus protocol for order critical state changes. For example, order critical state changes can include data flow configurations and node-to-machine mappings. In process 1306, consistency is maintained among the multiple masters using a gossip protocol. For example, node status updates, reporting information, and metrics information are maintained using the gossip protocol. Typically, stronger consistency is maintained using the consensus protocol and weaker consistency is maintained using the gossip protocol.
  • FIG. 14 depicts a flowchart of another example process for configuring a system to collect and aggregate datasets and to perform analytics on the datasets.
  • In process 1402, data sources on the multiple machines wherein datasets are to be collected from are identified. In process 1404, the multiple machines in the system that generate the datasets to be collected are configured to send the datasets to the data source. In process 1406, an arrival location where dataset that is collected is to be logged is identified.
  • In process 1408, configurations for the multiple machines are specified simultaneously by accessing a master through a web page and specifying the data sources for agent nodes. The master can be used to configure and/or dynamically reconfigure the agent nodes. In addition, statuses of the agent nodes are visible via the master.
  • In process 1410, a sink is specified for each of the agent node as the arrival location. In one embodiment, the sink is configured to be a fan-out sink that sends incoming datasets to child sinks coupled to the sink for data replication or for splitting off a CPU intensive tap. In addition, the sink can be configured as a failover sink that specifies alternate collectors in case the collector fails. For example, the failover sink can specify a local disk to store data until the sink that has failed recovers.
  • In process 1412, the arrival location is specified as a collector source of a collector node. In process 1414, a distributed file system is specified as a collector sink of the collector node.
  • FIG. 15 depicts a flowchart of an example process for dynamically processing an event including a dataset that is streamed from a source to a sink via nodes.
  • Events or messages are transmitted in a flexible and extensible data model. For example, events can be formatted as unstructured blobs and can be annotated with key-value attribute pairs which can be arbitrary. This allows analytic features include feature extraction, filtering, and transformation to be performed as the system forwards datasets downstream. The general pattern can utilize use a feature extractor to add key-value attributes, and then use a analysis module that reads key-value attributes and acts on it. There are feature extraction modules in the system, that can scan raw data and pull out structured data (e.g., numbers). This data can be written as a key-value attribute. For example raw data can take upon the form: “[31 Mar. 2010, 12:06 AM] Jon wrote this example”. The system can use a regular expression to pull out the data and turn it into a single number representation (unix time). The system can add a metadata attribute called ‘eventTime’ with the numerical representation as its value.
  • Downstream, there could be a module that looks at this value and compares it against a threshold predicate. For example, a threshold predicate can be: “was ‘eventtime’ before 31 Mar?” Alternately, a histogramming module can use that value and determine the number of events that were on 31 Mar (and 30 Mar, and 1 Apr, etc). The example graphs shown in the examples of FIGS. 10E and 10F can be generated using this pattern.
  • In process 1502, attributes of the event is specified in a data model. An example of the data model is illustrated in FIG. 9. In one embodiment, the data model is extensible to add properties to the event as the dataset is streamed from the source to the sink. In addition, additional data fields can be appended to the data model in response to additional sources being added to the event. The attributes can include a priority, which can be specified by a logging system. If a machine is heavily loaded, some events can be prioritize over others, using the ‘priority field’. For example, FATAL (highest priority) messages may skip ahead of line and be sent before lesser prioritized messages (e.g., ERROR or WARN priority messages). Also, if the system is, for example, running out of disk space, the DEBUG priority data can be erased before any of the INFO data gets erased, and INFO before WARN, etc. In addition, the attributes can include a source machine, a body, a time stamp, a metadata table, etc. Events will generally include each of these attributes but the body may have zero length and the metadata table may be empty.
  • In one embodiment, the timestamp can be a UNIX timestamp which is the time stamp from the source machine. The UNIX timestamp can be measured in milliseconds, for example. The timestamp may be a nanosecond timestamp, or any other monotonically increasing identifier which is a machine specific counter from the source machine. The priority field of the message can take on example values: TRACE, DEGBUG, INFO, WARN, ERROR, OR FATAL. In general, a logging system (e.g., syslog or log4j) provides the priority values. The source host can include a machine name or IP address. The body can include the raw log entry body. By default, the body can be truncated to a max of 32 KB per event. The max size can be reconfigured.
  • In one embodiment, event/message output to directories or files can be controlled based on the values of data model representing the event. For example, the system or any analytics being performed uses the timestamp to bucket the dataset based on time or date, specified by day of week, month, date and time, day of month, hour, minute, locale's equivalent of AM or PM, seconds, seconds since, year, etc. For example, when monitoring a web server, logs can be bucketed based on time, page hit, or the browser that is used, using extraction mechanisms. When tracking nodes, logs can be bucketed based on time and data node name, for example. In addition, the data group can be grouped based on a project identifier or person, for example, when tracking a feed of JIRA tickets from an apache feed.
  • In process 1504, the event described by the data model is annotated with key-value attribute pairs in a metadata table to enable analytic features to be performed on the dataset. In one embodiment, the metadata table includes a map from a string attribute name to an array of bytes. The metadata table can be used to specify routing of the event to multiple or different storage locations. In addition, the metadata table can be used to specify outputting of the dataset at the sink in multiple or different formats. The acktags, rolltags, and checksum tags/values are examples of values that use the metadata table. Metrics information generated by nodes can use the metadata table to store individual metrics. Any additional features can also utilize tag/values and the metadata table to allow new fields without rebuilding the program).
  • In process 1506, an event including a dataset that is streamed from a source to a sink via nodes is dynamically processed.
  • FIG. 16 depicts a flowchart of an example process for collecting and aggregating datasets for storage in a file system with fault tolerance.
  • In process 1602, datasets are collected from a data source on a machine where the datasets are generated. The data can be collected from an agent node on the machine. In process 1604, a batch ID comprising multiple messages is generated from the datasets. In one embodiment, the agent node generates the batch ID. In process 1606, a tag is assigned to the batch ID and a checksum is computed for the batch, for example, also by the agent node.
  • In process 1608, the batch ID and messages are written to an entry in a write-ahead-log (WAL) in storage. The writing can be performed by the agent node. In one embodiment, the storage for the WAL is on the machine where the datasets are generated. In process 1610, the datasets are sent to a receiving location (e.g., a collector node). The data set on the agent can be sent when the size or time elapsed reaches a certain threshold.
  • In process 1612, a file is written to the destination location, for example, by the collector node. In process 1614, the tag is added to a map, in response to verifying the checksum of the batch of multiple messages at a known location (e.g., the master). The map can be associated with multiple tags assigned to multiple batches of messages from the datasets. In process 1616, tags associated with the batches in the file that have been written to the destination location are identified in the map. The process continues in the flow chart of FIG. 17.
  • FIG. 17 depicts a flowchart of an example process for deleting the WAL in response to acknowledgement of receipt by the master.
  • In process 1702, an acknowledgement message that the batch has been written to the destination location is queried for. In process 1704, it is determined whether the batch has been written in the destination location, which can be a storage tier comprising a distributed file system. If so, in process 1706, the entry is deleted from the storage of the machine. If not, in process 1708, the batch is resent.
  • FIG. 18 depicts a flowchart of an example process for collecting and aggregating datasets with fault tolerance using a store on failure mechanism.
  • In process 1802, a dataset is collected from a data source on a machine where the dataset is generated.
  • In process 1804, the dataset is sent to a receiving location which aggregates the dataset. In one embodiment, the receiving location is a collector in a collector tier on the machine where the dataset is generated or another machine. The receiving location may also be a file system in a storage tier comprised in one or more storage devices.
  • In process 1806, the dataset is stored in persistent storage of the machine until the receiving location has been repaired or until another destination is identified in response to determining that receiving location which is mapped to receive the dataset has failed.
  • FIG. 19 shows a diagrammatic representation of a machine 1900 in the example form of a computer system within which a set of instructions, for causing the machine to perform any one or more of the methodologies discussed herein, may be executed.
  • In the example of FIG. 19, the computer system 900 includes a processor, memory, non-volatile memory, and an interface device. Various common components (e.g., cache memory) are omitted for illustrative simplicity. The computer system 900 is intended to illustrate a hardware device on which any of the components depicted in the example of FIG. 1 (and any other components described in this specification) can be implemented. The computer system 900 can be of any applicable known or convenient type. The components of the computer system 900 can be coupled together via a bus or through some other known or convenient device.
  • The processor may be, for example, a conventional microprocessor such as an Intel Pentium microprocessor or Motorola power PC microprocessor. One of skill in the relevant art will recognize that the terms “machine-readable (storage) medium” or “computer-readable (storage) medium” include any type of device that is accessible by the processor.
  • The memory is coupled to the processor by, for example, a bus. The memory can include, by way of example but not limitation, random access memory (RAM), such as dynamic RAM (DRAM) and static RAM (SRAM). The memory can be local, remote, or distributed.
  • The bus also couples the processor to the non-volatile memory and drive unit. The non-volatile memory is often a magnetic floppy or hard disk, a magnetic-optical disk, an optical disk, a read-only memory (ROM), such as a CD-ROM, EPROM, or EEPROM, a magnetic or optical card, or another form of storage for large amounts of data. Some of this data is often written, by a direct memory access process, into memory during execution of software in the computer 900. The non-volatile storage can be local, remote, or distributed. The non-volatile memory is optional because systems can be created with all applicable data available in memory. A typical computer system will usually include at least a processor, memory, and a device (e.g., a bus) coupling the memory to the processor.
  • Software is typically stored in the non-volatile memory and/or the drive unit. Indeed, for large programs, it may not even be possible to store the entire program in the memory. Nevertheless, it should be understood that for software to run, if necessary, it is moved to a computer readable location appropriate for processing, and for illustrative purposes, that location is referred to as the memory in this paper. Even when software is moved to the memory for execution, the processor will typically make use of hardware registers to store values associated with the software, and local cache that, ideally, serves to speed up execution. As used herein, a software program is assumed to be stored at any known or convenient location (from non-volatile storage to hardware registers) when the software program is referred to as “implemented in a computer-readable medium.” A processor is considered to be “configured to execute a program” when at least one value associated with the program is stored in a register readable by the processor.
  • The bus also couples the processor to the network interface device. The interface can include one or more of a modem or network interface. It will be appreciated that a modem or network interface can be considered to be part of the computer system 1900. The interface can include an analog modem, isdn modem, cable modem, token ring interface, satellite transmission interface (e.g. “direct PC”), or other interfaces for coupling a computer system to other computer systems. The interface can include one or more input and/or output devices. The I/O devices can include, by way of example but not limitation, a keyboard, a mouse or other pointing device, disk drives, printers, a scanner, and other input and/or output devices, including a display device. The display device can include, by way of example but not limitation, a cathode ray tube (CRT), liquid crystal display (LCD), or some other applicable known or convenient display device. For simplicity, it is assumed that controllers of any devices not depicted in the example of FIG. 19 reside in the interface.
  • In operation, the computer system 1900 can be controlled by operating system software that includes a file management system, such as a disk operating system. One example of operating system software with associated file management system software is the family of operating systems known as Windows® from Microsoft Corporation of Redmond, Wash., and their associated file management systems. Another example of operating system software with its associated file management system software is the Linux operating system and its associated file management system. The file management system is typically stored in the non-volatile memory and/or drive unit and causes the processor to execute the various acts required by the operating system to input and output data and to store data in the memory, including storing files on the non-volatile memory and/or drive unit.
  • Some portions of the detailed description may be presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is here, and generally, conceived to be a self-consistent sequence of operations leading to a desired result. The operations are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. It has proven convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like.
  • It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically stated otherwise as apparent from the following discussion, it is appreciated that throughout the description, discussions utilizing terms such as “processing” or “computing” or “calculating” or “determining” or “displaying” or the like, refer to the action and processes of a computer system, or similar electronic computing device, that manipulates and transforms data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage, transmission or display devices.
  • The algorithms and displays presented herein are not inherently related to any particular computer or other apparatus. Various general purpose systems may be used with programs in accordance with the teachings herein, or it may prove convenient to construct more specialized apparatus to perform the methods of some embodiments. The required structure for a variety of these systems will appear from the description below. In addition, the techniques are not described with reference to any particular programming language, and various embodiments may thus be implemented using a variety of programming languages.
  • In alternative embodiments, the machine operates as a standalone device or may be connected (e.g., networked) to other machines. In a networked deployment, the machine may operate in the capacity of a server or a client machine in a client-server network environment, or as a peer machine in a peer-to-peer (or distributed) network environment.
  • The machine may be a server computer, a client computer, a personal computer (PC), a tablet PC, a laptop computer, a set-top box (STB), a personal digital assistant (PDA), a cellular telephone, an iPhone, a Blackberry, a processor, a telephone, a web appliance, a network router, switch or bridge, or any machine capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by that machine.
  • While the machine-readable medium or machine-readable storage medium is shown in an exemplary embodiment to be a single medium, the term “machine-readable medium” and “machine-readable storage medium” should be taken to include a single medium or multiple media (e.g., a centralized or distributed database, and/or associated caches and servers) that store the one or more sets of instructions. The term “machine-readable medium” and “machine-readable storage medium” shall also be taken to include any medium that is capable of storing, encoding or carrying a set of instructions for execution by the machine and that cause the machine to perform any one or more of the methodologies of the presently disclosed technique and innovation.
  • In general, the routines executed to implement the embodiments of the disclosure, may be implemented as part of an operating system or a specific application, component, program, object, module or sequence of instructions referred to as “computer programs.” The computer programs typically comprise one or more instructions set at various times in various memory and storage devices in a computer, and that, when read and executed by one or more processing units or processors in a computer, cause the computer to perform operations to execute elements involving the various aspects of the disclosure.
  • Moreover, while embodiments have been described in the context of fully functioning computers and computer systems, those skilled in the art will appreciate that the various embodiments are capable of being distributed as a program product in a variety of forms, and that the disclosure applies equally regardless of the particular type of machine or computer-readable media used to actually effect the distribution.
  • Further examples of machine-readable storage media, machine-readable media, or computer-readable (storage) media include but are not limited to recordable type media such as volatile and non-volatile memory devices, floppy and other removable disks, hard disk drives, optical disks (e.g., Compact Disk Read-Only Memory (CD ROMS), Digital Versatile Disks, (DVDs), etc.), among others, and transmission type media such as digital and analog communication links.
  • Unless the context clearly requires otherwise, throughout the description and the claims, the words “comprise,” “comprising,” and the like are to be construed in an inclusive sense, as opposed to an exclusive or exhaustive sense; that is to say, in the sense of “including, but not limited to.” As used herein, the terms “connected,” “coupled,” or any variant thereof, means any connection or coupling, either direct or indirect, between two or more elements; the coupling of connection between the elements can be physical, logical, or a combination thereof. Additionally, the words “herein,” “above,” “below,” and words of similar import, when used in this application, shall refer to this application as a whole and not to any particular portions of this application. Where the context permits, words in the above Detailed Description using the singular or plural number may also include the plural or singular number respectively. The word “or,” in reference to a list of two or more items, covers all of the following interpretations of the word: any of the items in the list, all of the items in the list, and any combination of the items in the list.
  • The above detailed description of embodiments of the disclosure is not intended to be exhaustive or to limit the teachings to the precise form disclosed above. While specific embodiments of, and examples for, the disclosure are described above for illustrative purposes, various equivalent modifications are possible within the scope of the disclosure, as those skilled in the relevant art will recognize. For example, while processes or blocks are presented in a given order, alternative embodiments may perform routines having steps, or employ systems having blocks, in a different order, and some processes or blocks may be deleted, moved, added, subdivided, combined, and/or modified to provide alternative or subcombinations. Each of these processes or blocks may be implemented in a variety of different ways. Also, while processes or blocks are at times shown as being performed in series, these processes or blocks may instead be performed in parallel, or may be performed at different times. Further any specific numbers noted herein are only examples: alternative implementations may employ differing values or ranges.
  • The teachings of the disclosure provided herein can be applied to other systems, not necessarily the system described above. The elements and acts of the various embodiments described above can be combined to provide further embodiments.
  • Any patents and applications and other references noted above, including any that may be listed in accompanying filing papers, are incorporated herein by reference. Aspects of the disclosure can be modified, if necessary, to employ the systems, functions, and concepts of the various references described above to provide yet further embodiments of the disclosure.
  • These and other changes can be made to the disclosure in light of the above Detailed Description. While the above description describes certain embodiments of the disclosure, and describes the best mode contemplated, no matter how detailed the above appears in text, the teachings can be practiced in many ways. Details of the system may vary considerably in its implementation details, while still being encompassed by the subject matter disclosed herein. As noted above, particular terminology used when describing certain features or aspects of the disclosure should not be taken to imply that the terminology is being redefined herein to be restricted to any specific characteristics, features, or aspects of the disclosure with which that terminology is associated. In general, the terms used in the following claims should not be construed to limit the disclosure to the specific embodiments disclosed in the specification, unless the above Detailed Description section explicitly defines such terms. Accordingly, the actual scope of the disclosure encompasses not only the disclosed embodiments, but also all equivalent ways of practicing or implementing the disclosure under the claims.
  • While certain aspects of the disclosure are presented below in certain claim forms, the inventors contemplate the various aspects of the disclosure in any number of claim forms. For example, while only one aspect of the disclosure is recited as a means-plus-function claim under 35 U.S.C. §112, ¶113, other aspects may likewise be embodied as a means-plus-function claim, or in other forms, such as being embodied in a computer-readable medium. (Any claims intended to be treated under 35 U.S.C. §112, ¶113 will begin with the words “means for”.) Accordingly, the applicant reserves the right to add additional claims after filing the application to pursue such additional claim forms for other aspects of the disclosure.

Claims (26)

1. A method for configuring a system to collect and aggregate datasets, the method, comprising:
identifying a data source in the system from where dataset is to be collected;
configuring a machine in the system that generates the dataset to be collected, to send the dataset to the data source;
identifying an arrival location where the dataset that is collected is to be aggregated or written;
configuring an agent node by specifying a source for the agent node as the data source in the system and specifying a sink for the agent node as the arrival location.
2. The method of claim 1, wherein, the system includes multiple machines which generate datasets that are to be collected.
3. The method of claim 1, further comprising:
specifying the arrival location as a collector source of a collector node;
specifying a distributed file system as a collector sink of the collector node;
wherein, the distributed file system is the Hadoop distributed file system (HDFS)
4. The method of claim 1, wherein, the agent node is configurable to perform analytics on the dataset.
5. The method of claim 1, further comprising, launching a web page on a machine to access a master for the system using the master's universal resource locator (URL).
6. The method of claim 5, further comprising, scaling the system by implementing multiple masters on multiple machines.
7. The method of claim 6, further comprising, maintaining consistency among the multiple masters using a consensus protocol for order critical state changes.
8. The method of claim 7, wherein, the order critical state changes include data flow configurations and node-to-machine mappings.
9. The method of claim 6, further comprising, maintaining consistency among the multiple masters using a gossip protocol.
10. The method of claim 9, wherein, the gossip protocol is used to maintain weak consistency among data including but not limited to node status updates, reporting information, and metrics information.
11. The method of claim 5, wherein, configuration of the agent node is performed by accessing a link on the web page for the master.
12. The method of claim 1, wherein, the data source includes, one or more of, a syslog port, an incoming network connection, and an IRC channel.
13. The method of claim 1, wherein, the data source includes, a static file source.
14. The method of claim 1, wherein, the data source includes, one or more of, a status update and a feed.
15. The method of claim 1, wherein, the data source includes, output from execution of a program.
16. The method of claim 1, wherein, the data sink includes a network connection.
17. The method of claim 1, wherein, the data sink includes a physical disk.
18. The method of claim 1, wherein, the data sink includes a console.
19. A method for configuring a system having multiple machines to collect datasets from the multiple machines and to perform analytics on the datasets, the method, comprising:
identifying data sources on the multiple machines wherein datasets are to be collected from;
configuring the multiple machines in the system that generate the datasets are to be collected, to send the datasets to the data source;
identifying an arrival location where dataset that is collected is to be logged;
specifying configurations for the multiple machines simultaneously by accessing a master through a web page and specifying the data sources for agent nodes;
wherein, each agent node is associated with each of the multiple machines;
specifying a sink for each of the agent node as the arrival location;
specifying the arrival location as a collector source of a collector node;
specifying a distributed file system as a collector sink of the collector node;
wherein, the agent node or the collector node performs analytics on the datasets.
20. The method of claim 19, wherein, the master is used to configure and dynamically reconfigure the agent nodes.
21. The method of claim 19, wherein, statuses of the agent nodes are visible via the master.
22. The method of claim 19, wherein, the distributed file system is the Hadoop distributed file system (HDFS).
23. The method of claim 19,
further comprising, launching a web page on a machine to access the master using the master's universal resource locator (URL);
wherein, configuration of the agent nodes for the multiple machines is performed by accessing a link on the web page for the master.
24. The method of claim 19, further comprising, configuring the sink to be a fan-out sink that sends incoming datasets to child sinks coupled to the sink for data replication or for splitting off a CPU intensive tap.
25. The method of claim 19, further comprising, configuring the sink as a failover sink that specifies alternate collectors in case the collector fails.
26. The method of claim 25,
wherein, the sink is a failover sink that specifies a local disk to store data until the sink that has failed recovers.
US12/877,902 2010-03-31 2010-09-08 Configuring a system to collect and aggregate datasets Active 2032-01-14 US9317572B2 (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
US12/877,902 US9317572B2 (en) 2010-03-31 2010-09-08 Configuring a system to collect and aggregate datasets
US15/098,198 US10187461B2 (en) 2010-03-31 2016-04-13 Configuring a system to collect and aggregate datasets

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US31981610P 2010-03-31 2010-03-31
US12/877,902 US9317572B2 (en) 2010-03-31 2010-09-08 Configuring a system to collect and aggregate datasets

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US15/098,198 Continuation US10187461B2 (en) 2010-03-31 2016-04-13 Configuring a system to collect and aggregate datasets

Publications (2)

Publication Number Publication Date
US20110246816A1 true US20110246816A1 (en) 2011-10-06
US9317572B2 US9317572B2 (en) 2016-04-19

Family

ID=44711031

Family Applications (2)

Application Number Title Priority Date Filing Date
US12/877,902 Active 2032-01-14 US9317572B2 (en) 2010-03-31 2010-09-08 Configuring a system to collect and aggregate datasets
US15/098,198 Active 2031-04-18 US10187461B2 (en) 2010-03-31 2016-04-13 Configuring a system to collect and aggregate datasets

Family Applications After (1)

Application Number Title Priority Date Filing Date
US15/098,198 Active 2031-04-18 US10187461B2 (en) 2010-03-31 2016-04-13 Configuring a system to collect and aggregate datasets

Country Status (1)

Country Link
US (2) US9317572B2 (en)

Cited By (45)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20120233212A1 (en) * 2011-03-10 2012-09-13 Salesforce.Com, Inc. Methods and systems for processing social media data
US20130091094A1 (en) * 2011-10-06 2013-04-11 International Business Machines Corporation Accelerating data profiling process
US20130204948A1 (en) * 2012-02-07 2013-08-08 Cloudera, Inc. Centralized configuration and monitoring of a distributed computing cluster
US8510164B2 (en) 2008-01-24 2013-08-13 Salesforce.Com, Inc. Method and system for targeted advertising based on topical memes
US20130297771A1 (en) * 2012-05-04 2013-11-07 Itron, Inc. Coordinated collection of metering data
US8762870B2 (en) 2011-07-19 2014-06-24 Salesforce.Com, Inc. Multifunction drag-and-drop selection tool for selection of data objects in a social network application
US20140215007A1 (en) * 2013-01-31 2014-07-31 Facebook, Inc. Multi-level data staging for low latency data access
US8818940B2 (en) 2011-03-29 2014-08-26 Salesforce.Com, Inc. Systems and methods for performing record actions in a multi-tenant database and application system
EP2778955A1 (en) * 2013-03-15 2014-09-17 Ricoh Company Ltd. Method for processing data and data warehouse for processing data
US20140307588A1 (en) * 2013-04-16 2014-10-16 Dell Products L.P. System and method for symmetric switching fabrics
US8874526B2 (en) 2010-03-31 2014-10-28 Cloudera, Inc. Dynamically processing an event using an extensible data model
US8880592B2 (en) 2011-03-31 2014-11-04 Cloudera, Inc. User interface implementation for partial display update
US20150067410A1 (en) * 2013-08-27 2015-03-05 Tata Consultancy Services Limited Hardware failure prediction system
CN104836686A (en) * 2015-05-11 2015-08-12 北京明略软件系统有限公司 Method and device for generating configuration files of servers
US9128949B2 (en) 2012-01-18 2015-09-08 Cloudera, Inc. Memory allocation buffer for reduction of heap fragmentation
US20160005197A1 (en) * 2014-07-01 2016-01-07 Aol Inc. Computerized systems and methods for graph data modeling
US9245252B2 (en) 2008-05-07 2016-01-26 Salesforce.Com, Inc. Method and system for determining on-line influence in social media
EP2980707A1 (en) * 2014-07-31 2016-02-03 Deutsche Telekom AG Method for creating a database clone of a distributed database, system for creating a database clone of a distributed database, program and computer program product
US20160065493A1 (en) * 2014-08-29 2016-03-03 Cynny Spa Systems and methods to distribute computing tasks among multiple computers
US20160094420A1 (en) * 2014-09-29 2016-03-31 Cisco Technology, Inc. Network embedded framework for distributed network analytics
US9317572B2 (en) 2010-03-31 2016-04-19 Cloudera, Inc. Configuring a system to collect and aggregate datasets
US9338008B1 (en) 2012-04-02 2016-05-10 Cloudera, Inc. System and method for secure release of secret information over a network
US9342557B2 (en) 2013-03-13 2016-05-17 Cloudera, Inc. Low latency query engine for Apache Hadoop
WO2016054469A3 (en) * 2014-10-02 2016-05-26 Netapp, Inc. Techniques for error handling in parallel splitting of storage commands
US9361203B2 (en) 2010-03-31 2016-06-07 Cloudera, Inc. Collecting and aggregating log data with fault tolerance
US9372766B2 (en) 2014-02-11 2016-06-21 Saudi Arabian Oil Company Circumventing load imbalance in parallel simulations caused by faulty hardware nodes
US20160197799A1 (en) * 2015-01-05 2016-07-07 Cisco Technology, Inc. Distributed and adaptive computer network analytics
US20160204998A1 (en) * 2015-01-09 2016-07-14 Lg Cns Co., Ltd. Method of constructing data collector, server performing the same and storage medium for the same
US9405692B2 (en) 2012-03-21 2016-08-02 Cloudera, Inc. Data processing performance enhancement in a distributed file system
US9477731B2 (en) 2013-10-01 2016-10-25 Cloudera, Inc. Background format optimization for enhanced SQL-like queries in Hadoop
US9690671B2 (en) 2013-11-01 2017-06-27 Cloudera, Inc. Manifest-based snapshots in distributed computing environments
US9747333B2 (en) 2014-10-08 2017-08-29 Cloudera, Inc. Querying operating system state on multiple machines declaratively
US9753954B2 (en) 2012-09-14 2017-09-05 Cloudera, Inc. Data node fencing in a distributed file system
US9842126B2 (en) 2012-04-20 2017-12-12 Cloudera, Inc. Automatic repair of corrupt HBases
US9934382B2 (en) 2013-10-28 2018-04-03 Cloudera, Inc. Virtual machine image encryption
CN107943647A (en) * 2017-11-21 2018-04-20 北京小度互娱科技有限公司 A kind of reliable distributed information log collection method and system
US10223431B2 (en) 2013-01-31 2019-03-05 Facebook, Inc. Data stream splitting for low-latency data access
US10235649B1 (en) 2014-03-14 2019-03-19 Walmart Apollo, Llc Customer analytics data model
US10235687B1 (en) 2014-03-14 2019-03-19 Walmart Apollo, Llc Shortest distance to store
US10346769B1 (en) 2014-03-14 2019-07-09 Walmart Apollo, Llc System and method for dynamic attribute table
US10565538B1 (en) 2014-03-14 2020-02-18 Walmart Apollo, Llc Customer attribute exemption
US10733555B1 (en) 2014-03-14 2020-08-04 Walmart Apollo, Llc Workflow coordinator
US10877943B1 (en) * 2015-12-28 2020-12-29 Veritas Technologies Llc System and method of near-constant time recovery of continuously changing synthesized large datasets
US10986112B2 (en) * 2017-11-27 2021-04-20 Korea Internet & Security Agency Method for collecting cyber threat intelligence data and system thereof
US11128724B1 (en) * 2020-03-09 2021-09-21 Adobe Inc. Real-time interactive event analytics

Families Citing this family (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2014006713A1 (en) * 2012-07-04 2014-01-09 富士通株式会社 System, information processing apparatus, acquisition method, and acquisition program
JP2016201775A (en) * 2015-04-14 2016-12-01 富士通株式会社 Radio communication system, radio communication apparatus and radio communication method
US10650621B1 (en) 2016-09-13 2020-05-12 Iocurrents, Inc. Interfacing with a vehicular controller area network
US20180097845A1 (en) * 2016-10-05 2018-04-05 Rapid Focus Security, Llc Self-Managed Intelligent Network Devices that Protect and Monitor a Distributed Network
US9942872B1 (en) * 2017-06-09 2018-04-10 Rapid Focus Security, Llc Method and apparatus for wireless device location determination using signal strength
US10992602B2 (en) * 2019-08-19 2021-04-27 Landis+Gyr Innovations, Inc. Sequential storage of collected data from heterogeneous intervals

Citations (42)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20020055989A1 (en) * 2000-11-08 2002-05-09 Stringer-Calvert David W.J. Methods and apparatus for scalable, distributed management of virtual private networks
US20020138762A1 (en) * 2000-12-01 2002-09-26 Horne Donald R. Management of log archival and reporting for data network security systems
US20020174194A1 (en) * 2001-05-18 2002-11-21 Eoin Mooney Providing access to a plurality of message accounts from a single web-based interface
US20030051036A1 (en) * 2000-09-04 2003-03-13 International Business Machines Corporation Computer network system, computer system, method for communication between computer systems, method for measuring computer system performance, and storage medium
US20030055868A1 (en) * 2001-09-19 2003-03-20 International Business Machines Corporation Building distributed software services as aggregations of other services
US6542930B1 (en) * 2000-03-08 2003-04-01 International Business Machines Corporation Distributed file system with automated file management achieved by decoupling data analysis and movement operations
US6651242B1 (en) * 1999-12-14 2003-11-18 Novell, Inc. High performance computing system for distributed applications over a computer
US6687847B1 (en) * 1999-04-21 2004-02-03 Cornell Research Foundation, Inc. Failure detector with consensus protocol
US20040049563A1 (en) * 2002-09-09 2004-03-11 Orcutt Edward K. System and method for connecting dynamic networks with limited resources
US20040059728A1 (en) * 2000-04-27 2004-03-25 Todd Miller Method and system for retrieving search results from multiple disparate databases
US20040103166A1 (en) * 2002-11-27 2004-05-27 International Business Machines Corporation Semi-hierarchical system and method for administration of clusters of computer resources
US20040172421A1 (en) * 2002-12-09 2004-09-02 Yasushi Saito Namespace consistency for a wide-area file system
US20050071708A1 (en) * 2003-09-29 2005-03-31 International Business Machines (Ibm) Corporation Method, system, and program for recovery from a failure in an asynchronous data copying system
US20050138111A1 (en) * 2003-10-15 2005-06-23 Microsoft Corporation On-line service/application monitoring and reporting system
US20060020854A1 (en) * 2004-07-22 2006-01-26 International Business Machines Corporation Method and apparatus for high-speed network adapter failover
US20060080417A1 (en) * 2004-10-12 2006-04-13 International Business Machines Corporation Method, system and program product for automated topology formation in dynamic distributed environments
US7031981B1 (en) * 2001-12-21 2006-04-18 Unisys Corporation Tool supporting system log file reporting
US7107323B2 (en) * 2003-03-17 2006-09-12 Hitachi, Ltd. System and method of file distribution for a computer system in which partial files are arranged according to various allocation rules
US20070100913A1 (en) * 2005-10-12 2007-05-03 Sumner Gary S Method and system for data backup
US20070282988A1 (en) * 2006-05-31 2007-12-06 Christof Bornhoevd Device registration in a hierarchical monitor service
US20090013029A1 (en) * 2007-07-03 2009-01-08 Childress Rhonda L Device, system and method of operating a plurality of virtual logical sites
US7487228B1 (en) * 2003-01-30 2009-02-03 Red Hat, Inc. Metadata structures and related locking techniques to improve performance and scalability in a cluster file system
US20090171999A1 (en) * 2007-12-27 2009-07-02 Cloudscale Inc. System and Methodology for Parallel Stream Processing
US20090300765A1 (en) * 2008-05-27 2009-12-03 Deutsche Telekom Ag Unknown malcode detection using classifiers with optimal training sets
US7631034B1 (en) * 2008-09-18 2009-12-08 International Business Machines Corporation Optimizing node selection when handling client requests for a distributed file system (DFS) based on a dynamically determined performance index
US7653668B1 (en) * 2005-11-23 2010-01-26 Symantec Operating Corporation Fault tolerant multi-stage data replication with relaxed coherency guarantees
US7685109B1 (en) * 2005-12-29 2010-03-23 Amazon Technologies, Inc. Method and apparatus for data partitioning and replication in a searchable data service
US20100179855A1 (en) * 2009-01-09 2010-07-15 Ye Chen Large-Scale Behavioral Targeting for Advertising over a Network
US20100188689A1 (en) * 2009-01-29 2010-07-29 Xerox Corporation Method and system for a distributed file system based on user behaviors and user locales
US7831991B1 (en) * 1999-02-19 2010-11-09 Chaincast, Inc. Method and system for ensuring continuous data flow between re-transmitters within a chaincast communication system
US20100299326A1 (en) * 2007-10-26 2010-11-25 Scott Germaise Apparatuses, Methods and Systems For A Forum Ferreting System
US7864834B1 (en) * 2006-10-27 2011-01-04 Xilinx, Inc. Estimating digital frequency synthesizer jitter
US20110044354A1 (en) * 2009-08-18 2011-02-24 Facebook Inc. Adaptive Packaging of Network Resources
US7937482B1 (en) * 2008-03-27 2011-05-03 Amazon Technologies, Inc. Scalable consensus protocol
US20110119328A1 (en) * 2009-11-18 2011-05-19 Microsoft Corporation Load Balancing In A Distributed Computing Environment
US20110307534A1 (en) * 2009-03-25 2011-12-15 Zte Corporation Distributed file system supporting data block dispatching and file processing method thereof
US8108771B2 (en) * 2007-08-27 2012-01-31 Yahoo! Inc. Secure inter-module communication mechanism
US8108338B2 (en) * 2005-06-10 2012-01-31 International Business Machines Corporation Method and system for model-based replication of data
US20120102072A1 (en) * 2009-07-07 2012-04-26 Zte Corporation Distributed management monitoring system, monitoring method and creating method thereof
US8484716B1 (en) * 2009-08-07 2013-07-09 Adobe Systems Incorporated Hosting a server application on multiple network tiers
US8560639B2 (en) * 2009-04-24 2013-10-15 Microsoft Corporation Dynamic placement of replica data
US9081888B2 (en) * 2010-03-31 2015-07-14 Cloudera, Inc. Collecting and aggregating log data with fault tolerance

Family Cites Families (98)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US1017916A (en) 1910-10-31 1912-02-20 Electromagnetic Tool Company Power-hammer.
US4908746A (en) 1986-10-15 1990-03-13 United States Data Corporation Industrial control system
US5825877A (en) 1996-06-11 1998-10-20 International Business Machines Corporation Support for portable trusted software
US6553476B1 (en) 1997-02-10 2003-04-22 Matsushita Electric Industrial Co., Ltd. Storage management based on predicted I/O execution times
US6910082B1 (en) 1999-11-18 2005-06-21 International Business Machines Corporation Method, system and program products for reducing data movement within a computing environment by bypassing copying data between file system and non-file system buffers in a server
US7243299B1 (en) 2000-04-21 2007-07-10 Microsoft Corporation Methods and apparatus for displaying multiple contexts in electronic documents
US6970891B1 (en) 2000-11-27 2005-11-29 Microsoft Corporation Smart card with volatile memory file subsystem
KR100811419B1 (en) 2000-12-07 2008-03-07 주식회사 케이티 Countermeasure Against Denial-of-Service Attack in Authentication Protocols Using Public-Key Encryption
US7640512B1 (en) 2000-12-22 2009-12-29 Automated Logic Corporation Updating objects contained within a webpage
US7827136B1 (en) 2001-09-20 2010-11-02 Emc Corporation Management for replication of data stored in a data storage environment including a system and method for failover protection of software agents operating in the environment
US6728855B2 (en) 2001-09-27 2004-04-27 Seagate Technology Llc Method and system for data path verification
WO2003043157A1 (en) 2001-11-13 2003-05-22 Prometric, A Division Of Thomson Learning, Inc. Method and system for computer based testing using plugins to expand functionality of a test driver
US7392421B1 (en) 2002-03-18 2008-06-24 Symantec Operating Corporation Framework for managing clustering and replication
JP2004015665A (en) 2002-06-10 2004-01-15 Takeshi Sakamura Authentication method and ic card in electronic ticket distribution system
AU2003244310A1 (en) 2002-06-19 2004-03-11 Advanced Computer Systems, Inc. Inter-authentication method and device
US6889341B2 (en) 2002-06-28 2005-05-03 Hewlett-Packard Development Company, L.P. Method and apparatus for maintaining data integrity using a system management processor
US6678828B1 (en) 2002-07-22 2004-01-13 Vormetric, Inc. Secure network file access control system
US6931530B2 (en) 2002-07-22 2005-08-16 Vormetric, Inc. Secure network file access controller implementing access control and auditing
US7069497B1 (en) 2002-09-10 2006-06-27 Oracle International Corp. System and method for applying a partial page change
AU2003275181A1 (en) 2002-09-18 2004-04-08 Netezza Corporation Programmable streaming data processor for database appliance having multiple processing unit groups
US7143288B2 (en) 2002-10-16 2006-11-28 Vormetric, Inc. Secure file system server architecture and methods
US20040186832A1 (en) 2003-01-16 2004-09-23 Jardin Cary A. System and method for controlling processing in a distributed system
JP3788438B2 (en) 2003-03-24 2006-06-21 ソニー株式会社 Information recording medium, information processing apparatus, information processing method, and computer program
US7234072B2 (en) 2003-04-17 2007-06-19 Computer Associates Think, Inc. Method and system for making an application highly available
US7194589B2 (en) 2003-08-22 2007-03-20 Oracle International Corporation Reducing disk IO by full-cache write-merging
EP1510952B1 (en) 2003-08-28 2007-02-14 Accenture Global Services GmbH Capture, aggregation and/or visualisation of structural data of architectures
US9183256B2 (en) 2003-09-19 2015-11-10 Ibm International Group B.V. Performing sequence analysis as a relational join
US8060743B2 (en) 2003-11-14 2011-11-15 Certicom Corp. Cryptographic method and apparatus
JP2005259113A (en) 2004-02-12 2005-09-22 Ricoh Co Ltd Process editing apparatus, process management apparatus, process editing program, process management program, recording medium, process editing method and process management method
US7490323B2 (en) 2004-02-13 2009-02-10 International Business Machines Corporation Method and system for monitoring distributed applications on-demand
JP2006080642A (en) 2004-09-07 2006-03-23 Sony Corp Information processing apparatus and method, program, and recording medium
US8024560B1 (en) 2004-10-12 2011-09-20 Alten Alex I Systems and methods for securing multimedia transmissions over the internet
US7788671B2 (en) 2004-11-01 2010-08-31 International Business Machines Corporation On-demand application resource allocation through dynamic reconfiguration of application cluster size and placement
US7647643B2 (en) 2004-12-30 2010-01-12 Cisco Technology, Inc. Template access control lists
US7810153B2 (en) 2005-01-28 2010-10-05 Microsoft Corporation Controlling execution of computer applications
KR100643325B1 (en) 2005-02-18 2006-11-10 삼성전자주식회사 Network and creating method of domain thereof
CN100449326C (en) 2005-03-16 2009-01-07 西门子(中国)有限公司 Recording method and system of monitoring journal
JP2006285809A (en) 2005-04-04 2006-10-19 Hitachi Ltd Storage device performing performance guarantee for streaming
US8069267B2 (en) 2005-06-28 2011-11-29 Microsoft Corporation Advertisement refresh rules for network applications
US7818313B1 (en) 2005-07-18 2010-10-19 Sybase, Inc. Method for distributing processing of queries over a cluster of servers in a continuous processing system
EP1920393A2 (en) 2005-07-22 2008-05-14 Yogesh Chunilal Rathod Universal knowledge management and desktop search system
US7698321B2 (en) 2005-09-30 2010-04-13 At&T Intellectual Property I, L.P. Methods, systems, and products for updating web content
US20070237133A1 (en) * 2005-10-07 2007-10-11 Steven Woods System and method for providing content, applications, services and digital media to users in a peer-to-peer network
US20070113188A1 (en) 2005-11-17 2007-05-17 Bales Christopher E System and method for providing dynamic content in a communities framework
JP4435076B2 (en) 2005-11-18 2010-03-17 フェリカネットワークス株式会社 Mobile terminal, data communication method, and computer program
US7818432B2 (en) 2005-12-08 2010-10-19 International Business Machines Corporation Seamless reflection of model updates in a visual page for a visual channel in a composite services delivery system
JP2007206739A (en) 2006-01-30 2007-08-16 Toshiba Corp Information processor and authentication control method
CN101043319B (en) 2006-03-22 2011-02-02 鸿富锦精密工业(深圳)有限公司 Digital content protective system and method
JP2007279890A (en) 2006-04-04 2007-10-25 Hitachi Ltd Backup system and method
US9094194B2 (en) 2006-04-18 2015-07-28 International Business Machines Corporation Method and system for automating the recovery of a credential store when a user has forgotten their password using a temporary key pair created based on a new password provided by the user
WO2007125911A1 (en) 2006-04-24 2007-11-08 Panasonic Corporation Data processing device, method, program, integrated circuit, and program generating device
JP2008071154A (en) 2006-09-14 2008-03-27 Ricoh Co Ltd Program and layout plan forming device
US20080104579A1 (en) 2006-10-31 2008-05-01 Sap Ag Systems and methods of transforming XML schemas
JP4997950B2 (en) 2006-12-11 2012-08-15 富士通株式会社 Network management system, network management program, and network management method
US9015301B2 (en) 2007-01-05 2015-04-21 Digital Doors, Inc. Information infrastructure management tools with extractor, secure storage, content analysis and classification and method therefor
US8655939B2 (en) 2007-01-05 2014-02-18 Digital Doors, Inc. Electromagnetic pulse (EMP) hardened information infrastructure with extractor, cloud dispersal, secure storage, content analysis and classification and method therefor
US8468244B2 (en) 2007-01-05 2013-06-18 Digital Doors, Inc. Digital information infrastructure and method for security designated data and with granular data stores
JP2008234370A (en) 2007-03-21 2008-10-02 Just Syst Corp Document processor and document processing method
US7802128B2 (en) 2007-03-26 2010-09-21 Oracle International Corporation Method to avoid continuous application failovers in a cluster
WO2008117556A1 (en) 2007-03-28 2008-10-02 Nec Corporation Log acquisition system, log collection terminal, log acquisition terminal, and log acquisition method and program using the same system and terminals
KR101366332B1 (en) 2007-04-19 2014-02-21 엘지전자 주식회사 A method of automatic repeat request(ARQ) in communication system
US20080263006A1 (en) 2007-04-20 2008-10-23 Sap Ag Concurrent searching of structured and unstructured data
US7734961B2 (en) 2007-05-01 2010-06-08 International Business Machines Corporation Distributed logging apparatus system and method
US7631214B2 (en) 2007-05-31 2009-12-08 International Business Machines Corporation Failover processing in multi-tier distributed data-handling systems
US8185551B2 (en) 2007-06-08 2012-05-22 Massachusetts Institute Of Technology Disk-resident streaming dictionary
US7904809B2 (en) 2007-06-29 2011-03-08 Microsoft Corporation Model-based editors for dynamic validation
US7984043B1 (en) 2007-07-24 2011-07-19 Amazon Technologies, Inc. System and method for distributed query processing using configuration-independent query plans
US7961878B2 (en) 2007-10-15 2011-06-14 Adobe Systems Incorporated Imparting cryptographic information in network communications
US20090177697A1 (en) 2008-01-08 2009-07-09 International Business Machines Corporation Correlation and parallelism aware materialized view recommendation for heterogeneous, distributed database systems
WO2009088513A1 (en) * 2008-01-10 2009-07-16 Hewlett-Packard Development Company, L.P. Multiway peer-to-peer media streaming
US8037295B2 (en) 2008-04-15 2011-10-11 Authenex, Inc. Hardware-bonded credential manager method and system
US10296941B2 (en) * 2008-05-02 2019-05-21 Level 3 Communications Llc System and method for optimizing content distribution
MX353765B (en) 2008-05-26 2018-01-24 Nxp Bv Decoupling of measuring the response time of a transponder and its authentication.
US11461785B2 (en) 2008-07-10 2022-10-04 Ron M. Redlich System and method to identify, classify and monetize information as an intangible asset and a production model based thereon
JP2010021888A (en) 2008-07-11 2010-01-28 Toshiba Corp Communication apparatus, key server, and management server
US8918488B2 (en) 2009-02-04 2014-12-23 Citrix Systems, Inc. Methods and systems for automated management of virtual resources in a cloud computing environment
US20100332373A1 (en) 2009-02-26 2010-12-30 Jason Crabtree System and method for participation in energy-related markets
US9178935B2 (en) 2009-03-05 2015-11-03 Paypal, Inc. Distributed steam processing
US8782086B2 (en) 2009-08-27 2014-07-15 Cleversafe, Inc. Updating dispersed storage network access control information
US8631403B2 (en) 2010-01-04 2014-01-14 Vmware, Inc. Method and system for managing tasks by dynamically scaling centralized virtual center in virtual infrastructure
US8799418B2 (en) 2010-01-13 2014-08-05 Vmware, Inc. Cluster configuration
US20110179160A1 (en) 2010-01-21 2011-07-21 Microsoft Corporation Activity Graph for Parallel Programs in Distributed System Environment
US8874961B2 (en) 2010-03-22 2014-10-28 Infosys Limited Method and system for automatic failover of distributed query processing using distributed shared memory
US8874526B2 (en) 2010-03-31 2014-10-28 Cloudera, Inc. Dynamically processing an event using an extensible data model
US9317572B2 (en) 2010-03-31 2016-04-19 Cloudera, Inc. Configuring a system to collect and aggregate datasets
US9665868B2 (en) 2010-05-10 2017-05-30 Ca, Inc. One-time use password systems and methods
US20120036146A1 (en) 2010-10-26 2012-02-09 ParElastic Corporation Apparatus for elastic database processing with heterogeneous data
US8601265B2 (en) 2010-11-22 2013-12-03 Netapp, Inc. Method and system for improving storage security in a cloud computing environment
US8676710B2 (en) 2010-11-22 2014-03-18 Netapp, Inc. Providing security in a cloud storage environment
US8667267B1 (en) 2011-01-31 2014-03-04 Gazzang, Inc. System and method for communicating with a key management system
US20130031240A1 (en) 2011-07-29 2013-01-31 Cisco Technology, Inc. Capacity Evaluation of Computer Network Capabilities
US8533231B2 (en) 2011-08-12 2013-09-10 Nexenta Systems, Inc. Cloud storage system with distributed metadata
US8543836B2 (en) 2011-08-23 2013-09-24 International Business Machines Corporation Lightweight document access control using access control lists in the cloud storage or on the local file system
US9172608B2 (en) 2012-02-07 2015-10-27 Cloudera, Inc. Centralized configuration and monitoring of a distributed computing cluster
US9613104B2 (en) 2012-02-17 2017-04-04 Netflix, Inc. System and method for building a point-in-time snapshot of an eventually-consistent data store
US8732674B1 (en) 2012-11-14 2014-05-20 Microsoft Corporation Revertable managed execution image instrumentation
US9311354B2 (en) 2012-12-29 2016-04-12 Futurewei Technologies, Inc. Method for two-stage query optimization in massively parallel processing database clusters
US9081826B2 (en) 2013-01-07 2015-07-14 Facebook, Inc. System and method for distributed database query engines

Patent Citations (46)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7831991B1 (en) * 1999-02-19 2010-11-09 Chaincast, Inc. Method and system for ensuring continuous data flow between re-transmitters within a chaincast communication system
US6687847B1 (en) * 1999-04-21 2004-02-03 Cornell Research Foundation, Inc. Failure detector with consensus protocol
US6651242B1 (en) * 1999-12-14 2003-11-18 Novell, Inc. High performance computing system for distributed applications over a computer
US6542930B1 (en) * 2000-03-08 2003-04-01 International Business Machines Corporation Distributed file system with automated file management achieved by decoupling data analysis and movement operations
US20040059728A1 (en) * 2000-04-27 2004-03-25 Todd Miller Method and system for retrieving search results from multiple disparate databases
US20030051036A1 (en) * 2000-09-04 2003-03-13 International Business Machines Corporation Computer network system, computer system, method for communication between computer systems, method for measuring computer system performance, and storage medium
US20020055989A1 (en) * 2000-11-08 2002-05-09 Stringer-Calvert David W.J. Methods and apparatus for scalable, distributed management of virtual private networks
US20020138762A1 (en) * 2000-12-01 2002-09-26 Horne Donald R. Management of log archival and reporting for data network security systems
US20020174194A1 (en) * 2001-05-18 2002-11-21 Eoin Mooney Providing access to a plurality of message accounts from a single web-based interface
US20030055868A1 (en) * 2001-09-19 2003-03-20 International Business Machines Corporation Building distributed software services as aggregations of other services
US7031981B1 (en) * 2001-12-21 2006-04-18 Unisys Corporation Tool supporting system log file reporting
US20040049563A1 (en) * 2002-09-09 2004-03-11 Orcutt Edward K. System and method for connecting dynamic networks with limited resources
US20040103166A1 (en) * 2002-11-27 2004-05-27 International Business Machines Corporation Semi-hierarchical system and method for administration of clusters of computer resources
US20040172421A1 (en) * 2002-12-09 2004-09-02 Yasushi Saito Namespace consistency for a wide-area file system
US8311980B2 (en) * 2002-12-09 2012-11-13 Hewlett-Packard Development Company, L.P. Namespace consistency for a wide-area file system
US7487228B1 (en) * 2003-01-30 2009-02-03 Red Hat, Inc. Metadata structures and related locking techniques to improve performance and scalability in a cluster file system
US7107323B2 (en) * 2003-03-17 2006-09-12 Hitachi, Ltd. System and method of file distribution for a computer system in which partial files are arranged according to various allocation rules
US7325041B2 (en) * 2003-03-17 2008-01-29 Hitachi, Ltd. File distribution system in which partial files are arranged according to various allocation rules associated with a plurality of file types
US7620698B2 (en) * 2003-03-17 2009-11-17 Hitachi, Ltd. File distribution system in which partial files are arranged according to various allocation rules associated with a plurality of file types
US20050071708A1 (en) * 2003-09-29 2005-03-31 International Business Machines (Ibm) Corporation Method, system, and program for recovery from a failure in an asynchronous data copying system
US20050138111A1 (en) * 2003-10-15 2005-06-23 Microsoft Corporation On-line service/application monitoring and reporting system
US20060020854A1 (en) * 2004-07-22 2006-01-26 International Business Machines Corporation Method and apparatus for high-speed network adapter failover
US20060080417A1 (en) * 2004-10-12 2006-04-13 International Business Machines Corporation Method, system and program product for automated topology formation in dynamic distributed environments
US8108338B2 (en) * 2005-06-10 2012-01-31 International Business Machines Corporation Method and system for model-based replication of data
US20070100913A1 (en) * 2005-10-12 2007-05-03 Sumner Gary S Method and system for data backup
US7653668B1 (en) * 2005-11-23 2010-01-26 Symantec Operating Corporation Fault tolerant multi-stage data replication with relaxed coherency guarantees
US7685109B1 (en) * 2005-12-29 2010-03-23 Amazon Technologies, Inc. Method and apparatus for data partitioning and replication in a searchable data service
US20070282988A1 (en) * 2006-05-31 2007-12-06 Christof Bornhoevd Device registration in a hierarchical monitor service
US7864834B1 (en) * 2006-10-27 2011-01-04 Xilinx, Inc. Estimating digital frequency synthesizer jitter
US20090013029A1 (en) * 2007-07-03 2009-01-08 Childress Rhonda L Device, system and method of operating a plurality of virtual logical sites
US8108771B2 (en) * 2007-08-27 2012-01-31 Yahoo! Inc. Secure inter-module communication mechanism
US20100299326A1 (en) * 2007-10-26 2010-11-25 Scott Germaise Apparatuses, Methods and Systems For A Forum Ferreting System
US20090171999A1 (en) * 2007-12-27 2009-07-02 Cloudscale Inc. System and Methodology for Parallel Stream Processing
US7937482B1 (en) * 2008-03-27 2011-05-03 Amazon Technologies, Inc. Scalable consensus protocol
US20090300765A1 (en) * 2008-05-27 2009-12-03 Deutsche Telekom Ag Unknown malcode detection using classifiers with optimal training sets
US7631034B1 (en) * 2008-09-18 2009-12-08 International Business Machines Corporation Optimizing node selection when handling client requests for a distributed file system (DFS) based on a dynamically determined performance index
US20100179855A1 (en) * 2009-01-09 2010-07-15 Ye Chen Large-Scale Behavioral Targeting for Advertising over a Network
US20100188689A1 (en) * 2009-01-29 2010-07-29 Xerox Corporation Method and system for a distributed file system based on user behaviors and user locales
US20110307534A1 (en) * 2009-03-25 2011-12-15 Zte Corporation Distributed file system supporting data block dispatching and file processing method thereof
US8560639B2 (en) * 2009-04-24 2013-10-15 Microsoft Corporation Dynamic placement of replica data
US20120102072A1 (en) * 2009-07-07 2012-04-26 Zte Corporation Distributed management monitoring system, monitoring method and creating method thereof
US8484716B1 (en) * 2009-08-07 2013-07-09 Adobe Systems Incorporated Hosting a server application on multiple network tiers
US20110044354A1 (en) * 2009-08-18 2011-02-24 Facebook Inc. Adaptive Packaging of Network Resources
US7970861B2 (en) * 2009-11-18 2011-06-28 Microsoft Corporation Load balancing in a distributed computing environment
US20110119328A1 (en) * 2009-11-18 2011-05-19 Microsoft Corporation Load Balancing In A Distributed Computing Environment
US9081888B2 (en) * 2010-03-31 2015-07-14 Cloudera, Inc. Collecting and aggregating log data with fault tolerance

Cited By (72)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8510164B2 (en) 2008-01-24 2013-08-13 Salesforce.Com, Inc. Method and system for targeted advertising based on topical memes
US9245252B2 (en) 2008-05-07 2016-01-26 Salesforce.Com, Inc. Method and system for determining on-line influence in social media
US8874526B2 (en) 2010-03-31 2014-10-28 Cloudera, Inc. Dynamically processing an event using an extensible data model
US9817859B2 (en) 2010-03-31 2017-11-14 Cloudera, Inc. Collecting and aggregating log data with fault tolerance
US9317572B2 (en) 2010-03-31 2016-04-19 Cloudera, Inc. Configuring a system to collect and aggregate datasets
US9817867B2 (en) 2010-03-31 2017-11-14 Cloudera, Inc. Dynamically processing an event using an extensible data model
US9361203B2 (en) 2010-03-31 2016-06-07 Cloudera, Inc. Collecting and aggregating log data with fault tolerance
US10187461B2 (en) 2010-03-31 2019-01-22 Cloudera, Inc. Configuring a system to collect and aggregate datasets
US9201910B2 (en) 2010-03-31 2015-12-01 Cloudera, Inc. Dynamically processing an event using an extensible data model
US20120233212A1 (en) * 2011-03-10 2012-09-13 Salesforce.Com, Inc. Methods and systems for processing social media data
US8949270B2 (en) * 2011-03-10 2015-02-03 Salesforce.Com, Inc. Methods and systems for processing social media data
US8818940B2 (en) 2011-03-29 2014-08-26 Salesforce.Com, Inc. Systems and methods for performing record actions in a multi-tenant database and application system
US8880592B2 (en) 2011-03-31 2014-11-04 Cloudera, Inc. User interface implementation for partial display update
US8762870B2 (en) 2011-07-19 2014-06-24 Salesforce.Com, Inc. Multifunction drag-and-drop selection tool for selection of data objects in a social network application
US8719271B2 (en) * 2011-10-06 2014-05-06 International Business Machines Corporation Accelerating data profiling process
US20130091094A1 (en) * 2011-10-06 2013-04-11 International Business Machines Corporation Accelerating data profiling process
US9128949B2 (en) 2012-01-18 2015-09-08 Cloudera, Inc. Memory allocation buffer for reduction of heap fragmentation
US20150039735A1 (en) * 2012-02-07 2015-02-05 Cloudera, Inc. Centralized configuration of a distributed computing cluster
US9172608B2 (en) * 2012-02-07 2015-10-27 Cloudera, Inc. Centralized configuration and monitoring of a distributed computing cluster
US9716624B2 (en) * 2012-02-07 2017-07-25 Cloudera, Inc. Centralized configuration of a distributed computing cluster
US20130204948A1 (en) * 2012-02-07 2013-08-08 Cloudera, Inc. Centralized configuration and monitoring of a distributed computing cluster
US9405692B2 (en) 2012-03-21 2016-08-02 Cloudera, Inc. Data processing performance enhancement in a distributed file system
US9338008B1 (en) 2012-04-02 2016-05-10 Cloudera, Inc. System and method for secure release of secret information over a network
US9842126B2 (en) 2012-04-20 2017-12-12 Cloudera, Inc. Automatic repair of corrupt HBases
EP2661093B1 (en) 2012-05-04 2017-10-04 Itron, Inc. Coordinated collection of metering data
US20130297771A1 (en) * 2012-05-04 2013-11-07 Itron, Inc. Coordinated collection of metering data
US9674589B2 (en) * 2012-05-04 2017-06-06 Itron, Inc. Coordinated collection of metering data
US9753954B2 (en) 2012-09-14 2017-09-05 Cloudera, Inc. Data node fencing in a distributed file system
US10223431B2 (en) 2013-01-31 2019-03-05 Facebook, Inc. Data stream splitting for low-latency data access
US9609050B2 (en) * 2013-01-31 2017-03-28 Facebook, Inc. Multi-level data staging for low latency data access
US20140215007A1 (en) * 2013-01-31 2014-07-31 Facebook, Inc. Multi-level data staging for low latency data access
US10581957B2 (en) * 2013-01-31 2020-03-03 Facebook, Inc. Multi-level data staging for low latency data access
US9342557B2 (en) 2013-03-13 2016-05-17 Cloudera, Inc. Low latency query engine for Apache Hadoop
EP2778955A1 (en) * 2013-03-15 2014-09-17 Ricoh Company Ltd. Method for processing data and data warehouse for processing data
US9590846B2 (en) 2013-04-16 2017-03-07 Dell Products, L.P. System and method for symmetric switching fabrics
US9240957B2 (en) * 2013-04-16 2016-01-19 Dell Products L.P. System and method for symmetric switching fabrics
US20140307588A1 (en) * 2013-04-16 2014-10-16 Dell Products L.P. System and method for symmetric switching fabrics
US20150067410A1 (en) * 2013-08-27 2015-03-05 Tata Consultancy Services Limited Hardware failure prediction system
US9477731B2 (en) 2013-10-01 2016-10-25 Cloudera, Inc. Background format optimization for enhanced SQL-like queries in Hadoop
US9934382B2 (en) 2013-10-28 2018-04-03 Cloudera, Inc. Virtual machine image encryption
US9690671B2 (en) 2013-11-01 2017-06-27 Cloudera, Inc. Manifest-based snapshots in distributed computing environments
US9372766B2 (en) 2014-02-11 2016-06-21 Saudi Arabian Oil Company Circumventing load imbalance in parallel simulations caused by faulty hardware nodes
US10346769B1 (en) 2014-03-14 2019-07-09 Walmart Apollo, Llc System and method for dynamic attribute table
US10565538B1 (en) 2014-03-14 2020-02-18 Walmart Apollo, Llc Customer attribute exemption
US10235687B1 (en) 2014-03-14 2019-03-19 Walmart Apollo, Llc Shortest distance to store
US10235649B1 (en) 2014-03-14 2019-03-19 Walmart Apollo, Llc Customer analytics data model
US10733555B1 (en) 2014-03-14 2020-08-04 Walmart Apollo, Llc Workflow coordinator
US10290126B2 (en) * 2014-07-01 2019-05-14 Oath Inc. Computerized systems and methods for graph data modeling
US11080904B2 (en) 2014-07-01 2021-08-03 Verizon Media Inc. Computerized systems and methods for graph data modeling
US11798208B2 (en) 2014-07-01 2023-10-24 Yahoo Assets Llc Computerized systems and methods for graph data modeling
US20160005197A1 (en) * 2014-07-01 2016-01-07 Aol Inc. Computerized systems and methods for graph data modeling
EP2980707A1 (en) * 2014-07-31 2016-02-03 Deutsche Telekom AG Method for creating a database clone of a distributed database, system for creating a database clone of a distributed database, program and computer program product
US20160065493A1 (en) * 2014-08-29 2016-03-03 Cynny Spa Systems and methods to distribute computing tasks among multiple computers
US9928149B2 (en) 2014-08-29 2018-03-27 Cynny Space Srl Systems and methods to maintain data integrity and redundancy in a computing system having multiple computers
US10565074B2 (en) * 2014-08-29 2020-02-18 Cynny Space Srl Systems and methods to distribute computing tasks among multiple computers
US9823985B2 (en) 2014-08-29 2017-11-21 Cynny Space Srl Systems and methods to organize a computing system having multiple computers
US11695657B2 (en) * 2014-09-29 2023-07-04 Cisco Technology, Inc. Network embedded framework for distributed network analytics
US20160094420A1 (en) * 2014-09-29 2016-03-31 Cisco Technology, Inc. Network embedded framework for distributed network analytics
US10256947B2 (en) 2014-10-02 2019-04-09 Netapp Inc. Techniques for error handling in parallel splitting of storage commands
WO2016054469A3 (en) * 2014-10-02 2016-05-26 Netapp, Inc. Techniques for error handling in parallel splitting of storage commands
US9489275B2 (en) 2014-10-02 2016-11-08 Netapp, Inc. Techniques for error handling in parallel splitting of storage commands
US9747333B2 (en) 2014-10-08 2017-08-29 Cloudera, Inc. Querying operating system state on multiple machines declaratively
US10305759B2 (en) * 2015-01-05 2019-05-28 Cisco Technology, Inc. Distributed and adaptive computer network analytics
CN107210928A (en) * 2015-01-05 2017-09-26 思科技术公司 Distributed and self-adaptive computer network analysis
US20160197799A1 (en) * 2015-01-05 2016-07-07 Cisco Technology, Inc. Distributed and adaptive computer network analytics
US20160204998A1 (en) * 2015-01-09 2016-07-14 Lg Cns Co., Ltd. Method of constructing data collector, server performing the same and storage medium for the same
US9998341B2 (en) * 2015-01-09 2018-06-12 Lg Cns Co., Ltd. Method of constructing data collector, server performing the same and storage medium for the same
CN104836686A (en) * 2015-05-11 2015-08-12 北京明略软件系统有限公司 Method and device for generating configuration files of servers
US10877943B1 (en) * 2015-12-28 2020-12-29 Veritas Technologies Llc System and method of near-constant time recovery of continuously changing synthesized large datasets
CN107943647A (en) * 2017-11-21 2018-04-20 北京小度互娱科技有限公司 A kind of reliable distributed information log collection method and system
US10986112B2 (en) * 2017-11-27 2021-04-20 Korea Internet & Security Agency Method for collecting cyber threat intelligence data and system thereof
US11128724B1 (en) * 2020-03-09 2021-09-21 Adobe Inc. Real-time interactive event analytics

Also Published As

Publication number Publication date
US9317572B2 (en) 2016-04-19
US20160226968A1 (en) 2016-08-04
US10187461B2 (en) 2019-01-22

Similar Documents

Publication Publication Date Title
US9817859B2 (en) Collecting and aggregating log data with fault tolerance
US10187461B2 (en) Configuring a system to collect and aggregate datasets
US9817867B2 (en) Dynamically processing an event using an extensible data model
US9082127B2 (en) Collecting and aggregating datasets for analysis
US11586692B2 (en) Streaming data processing
US11615084B1 (en) Unified data processing across streaming and indexed data sets
US20220215028A1 (en) Data Pipeline for Scalable Analytics and Management
US11886440B1 (en) Guided creation interface for streaming data processing pipelines
US11093518B1 (en) Information technology networked entity monitoring with dynamic metric and threshold selection
US11250056B1 (en) Updating a location marker of an ingestion buffer based on storing buckets in a shared storage system
US11163758B2 (en) External dataset capability compensation
US11416528B2 (en) Query acceleration data store
US20180089258A1 (en) Resource allocation for multiple datasets
US20180089262A1 (en) Dynamic resource allocation for common storage query
US11615082B1 (en) Using a data store and message queue to ingest data for a data intake and query system
US20220121708A1 (en) Dynamic data enrichment
US11449371B1 (en) Indexing data at a data intake and query system based on a node capacity threshold
US11573971B1 (en) Search and data analysis collaboration system
US11715051B1 (en) Service provider instance recommendations using machine-learned classifications and reconciliation
US11574242B1 (en) Guided workflows for machine learning-based data analyses
US11681707B1 (en) Analytics query response transmission
US20180248772A1 (en) Managing intelligent microservices in a data streaming ecosystem
WO2019157399A1 (en) Data pipeline for scalable analytics and management
US11687487B1 (en) Text files updates to an active processing pipeline
US11860940B1 (en) Identifying buckets for query execution using a catalog of buckets

Legal Events

Date Code Title Description
AS Assignment

Owner name: CLOUDERA, INC., CALIFORNIA

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:HSIEH, JONATHAN MING-CYN;ROBINSON, HENRY NOEL;SIGNING DATES FROM 20101104 TO 20101114;REEL/FRAME:025435/0145

STCF Information on status: patent grant

Free format text: PATENTED CASE

MAFP Maintenance fee payment

Free format text: PAYMENT OF MAINTENANCE FEE, 4TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1551); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

Year of fee payment: 4

AS Assignment

Owner name: CITIBANK, N.A., AS COLLATERAL AGENT, NEW YORK

Free format text: SECURITY INTEREST;ASSIGNORS:CLOUDERA, INC.;HORTONWORKS, INC.;REEL/FRAME:054832/0559

Effective date: 20201222

AS Assignment

Owner name: JPMORGAN CHASE BANK, N.A., ILLINOIS

Free format text: SECOND LIEN NOTICE AND CONFIRMATION OF GRANT OF SECURITY INTEREST IN PATENTS;ASSIGNORS:CLOUDERA, INC.;HORTONWORKS, INC.;REEL/FRAME:057776/0284

Effective date: 20211008

Owner name: JPMORGAN CHASE BANK, N.A., ILLINOIS

Free format text: FIRST LIEN NOTICE AND CONFIRMATION OF GRANT OF SECURITY INTEREST IN PATENTS;ASSIGNORS:CLOUDERA, INC.;HORTONWORKS, INC.;REEL/FRAME:057776/0185

Effective date: 20211008

AS Assignment

Owner name: HORTONWORKS, INC., CALIFORNIA

Free format text: RELEASE OF SECURITY INTERESTS IN PATENTS;ASSIGNOR:CITIBANK, N.A.;REEL/FRAME:057804/0355

Effective date: 20211008

Owner name: CLOUDERA, INC., CALIFORNIA

Free format text: RELEASE OF SECURITY INTERESTS IN PATENTS;ASSIGNOR:CITIBANK, N.A.;REEL/FRAME:057804/0355

Effective date: 20211008

MAFP Maintenance fee payment

Free format text: PAYMENT OF MAINTENANCE FEE, 8TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1552); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

Year of fee payment: 8