WO2016122653A1

WO2016122653A1 - Common search distances in data partitions

Info

Publication number: WO2016122653A1
Application number: PCT/US2015/013952
Authority: WO
Inventors: Maria T. GONZALEZ DIAZ; Jun Li; Krishnamurthy Viswanathan; Mijung Kim
Original assignee: Hewlett Packard Enterprise Development Lp
Priority date: 2015-01-30
Filing date: 2015-01-30
Publication date: 2016-08-04

Abstract

Methods, systems, and techniques for using common search distances in data partitions of a data set stored in a multicore machine are discussed herein. For example, a first data point and a second data point from a data set are respectively assigned to a first data partition stored in a first processing node and second data partition stored in a second processing node. The assignments of the data points to the data partitions may be based on hash values derived from identifiers assigned the respective data points. A first set of hash indexes are stored in the first processing node. The first set of hash indexes may individually correspond to common search distances. A second set of hash indexes are stored in the second processing node. The second set of hash indexes may individually correspond to the common search distances.

Description

COMMON SEARCH DISTANCES IN DATA PARTITIONS

Background

[0001] Quite often, a relatively large volume of data is searched for purposes of identifying and retrieving the closest matches to a search query. For example, the data may be time series data that may be, for example, acquired by a sensor.

Issues with the sensor may be identified by searching for certain patterns in the time series data. The time series data may be searched for patterns for various other purposes, such as classification, pattern detection, modeling and anomaly detection, as examples.

Brief Description of the Drawings

[0002] FIG.1 is a schematic diagram of a muiticore machine according to an example implementation.

[0003] FIG. 2 is a flowchart illustrating a method for assigning data points of a data set to multiple processing nodes of a muiticore system, according to an example.

[0004] FIG. 3 is a block diagram illustrating an example operation of a shuffle- based configuration module that may be used to partition a dataset onto multiple processing nodes in a muiticore machine, according to an example.

[0005] FIG. 4 is a flow chart illustrating a method for generating common search distances based on data points stored in data partitions, according to an example.

[0006] FIG. 5 is a sequence diagram illustrating a method for handling query requests, according to an example.

[0007] FIG. 6 is a diagram illustrating a worker thread of a processing node capable of processing concurrent local query requests, according to an example.

Detailed Description

[0008] Systems and techniques are described herein for purposes of processing a search query using a multiple core, or "multicore" machine, in this context, a "multicore" machine refers to a physical machine that contains a computing component with at least two independent central processing units (CPUs). Each CPU can contain multiple processing cores. A given processing core is a unit that is constructed to read and execute machine executable instructions. In accordance with example implementations, the multicore machine may contain one or multiple CPU semiconductor packages, where each package contains multiple processing cores. Further, multiple processing cores may access or otherwise share the same local memory. As used herein, the term "processing node" may refer to a unit that includes multiple processing cores that share the same local memory to access instructions and/or data.

[0009] The use of a multicore machine to process a search query, as disclosed herein, allows relatively time efficient searching of a relative large dataset for purposes of identifying and retrieving matches to the search query. Techniques and systems that are disclosed herein may be applied to relatively large volume data, such as high dimensional data (multimedia image or video data, as examples), time series data, or any other suitable data. Identifying and retrieving high volume data that are similar to a data item specified by a search query may be useful for such purposes as classification, pattern detection, modeling, fault diagnosis, and anomaly detection, as well as for other purposes. Performing a relatively time efficient search may allow the construction of better models, better pattern detection, faster fault analysis, more rapid classifications, and more timely detection of anomalies. Other and different advantages may be achieved in accordance with further

implementations.

[0010] Approaches for using a multicore machine to identify and retrieve matches in a data set that are similar to a search query may involve partitioning the data set across multiple processing nodes in the multicore machine. One approach for partitioning the data set may be to read a block of data from the original data set and store that block of data in partition A, read another block of data from the original data set and store that block of data in partition B, and so on. Thus, under this approach, if two data points are stored in the same block of data in the original data set, those two data points are likely to be stored in the same data partition in the muiticore machine.

[0011] However, in some cases, data points stored in the same block of data may exhibit spatial locality and temporal locality. Accordingly, if the data set is partitioned based on how the data points are stored in the original data set, such spatial and temporal locality will be preserved in individual data partitions. Thus, it's possible that different partitions will be statistically dissimilar. Because the data partitions can be statistically dissimilar, it may be difficult to create a common search model for the different data partitions. Having different search models may make it difficult to determine whether the K closest points to a given query have been retrieved at any given point.

[0012] Examples discussed herein may use a partitioning approach that allows for the different partitions to use common search distances when building hash indexes on the processing nodes assigned to the data partitions. To better understand the term "common search distance," locality sensitive hashing (LSH) is now discussed. LSH may be a technique for building data structures that identify similar data points, as may be used in performing proximity searches. Here, proximity refers not to the storage location of two data points but instead a distance between data points in a multidimensional space. Thus, a proximity search using an input data point may return data points that located close to the input data point in a multidimensional space.

[0013] Conceptually, LSH involves hashing data points to entries of a hash table such that data points that are similar to each are likely to map to the same entry. In some cases, similarity may be measured as a function of a hash parameter referred to as a "search distance" (also referred to as "R"). Thus, in LSH, an input data point is hashed to generate a key that maps to an entry of a hash table that specifies other data points that are likely to be proximate to the input data point.

[0014] In some cases, a key may be a vector of hash values generated by multiple hash functions. The data structure that includes a hash table may be referred to as a hash index. To increase the probability that a hash index will return the top K closest data points to an input data point with a given confidence interval, the hash index can include multiple hash tables that each return data points that are likely to be near the input data point as a function of the search distance.

[0015] Constructing a hash index with a search distance that works well over a large search space is often difficult. Such may be the case because a search distance that is too large may return too many data points. Having too many data points can result in a computationally expensive exhaustive search over a very large search space to determine the top-K data points within the entry of the hash table. Conversely, having too few data points may result in results that are incomplete as the entry may have less than the K even though the original data set has more than K.

[0016] To compensate for the difficulty in selecting a single search distance, some implementations of LSH-based search systems may use multiple hash indexes where each hash index is built using a different search distance (e.g., R0, R1 ,..RX). In processing a search query, a search system may then start with the hash index associated with the shortest search distance. If sufficient results are returned (e.g., a result set that exceeds the result size threshold), the search system returns those results and the search is done. Otherwise, the search system uses the hash index associated with the next largest search distance. This process of determining whether a hash index produces a sufficient result size and advancing if not continues until a sufficient search result is obtained.

[0017] Thus, one aspect discussed herein, may involve assigning a first data point from a data set to a first data partition stored in a first processing node of a mu!ticore machine. The assignment of the first data point to the first data partition may be based on a first hash value derived from a first identifier (e.g., a file name, a sequence identifier, or any other suitable data usable to uniquely reference between different data points) assigned to the first data point. A second data point from the data set may be assigned to a second data partition stored in a second processing node of the multicore machine. The assignment of the second data point to the second data partition may be based on a second hash value derived from a second identifier assigned to the second data point. A first set of hash indexes may be stored in the first processing node. The first set of hash indexes may individually correspond to common search distances. The second set of hash indexes may be stored in the second processing node. The second set of hash indexes may individually correspond to the common search distances.

[0018] In another aspect, a machine-readable storage device may include instructions that, when executed, cause a processor to distribute a data set across a first processing node and a second processing node according to a distribution pattern that maintains a similar distribution pattern between data points from the data set stored in a first local memory unit of the first processing node and data points from the data set stored in a second local memory of the second processing node. The instructions, when executed, may further cause the processor to calculate common search distances based on an analysis of the distributed data set. The shuffle-based configuration module may also cause the first processing node and the second processing node to create hash indexes with the common search distances.

[0019] In yet another aspect, an apparatus may include a first processing node, a second processing node, and a search coordinator module. The first processing node may include a first set of processing cores with access to a first local memory unit. The first local memory unit stores a first data partition and a first hash index constructed with common search distances. The second processing node that includes a second set of processing cores with access to a second local memory unit. The second local memory unit stores a second data partition and a second hash index constructed with the common search distances. The search coordinator module may initiate local searches in the first processing node and the second processing node. The local searches perform searches on the first data partition and the second data partitions based on progressively increasing the common search distances. The search coordinator module is to notify a client device when cumulative search results from local search results exceed a search result size threshold. A client device may be a computer device or computer system that sends requests to the search coordinator module. The requests may be query requests that requests a search of a data set for data points that may be similar to a data item specified by the request. The "search result size threshold" may be a parameter of the search that specifies a determinable number of data points that are to be returned from the request, in some cases, the search result size threshold may represent a number such that the results return a number of data points that are the closest data points in the data set.

[0020] The figures are now discussed in greater detail. FIG. 1 is a schematic diagram of a muiticore machine 100 according to an example implementation. The muiticore machine 100 is a physical machine that is constructed from actual machine executable instructions, or "software," and actual hardware. The hardware of the muiticore machine 100 includes processing nodes 110 (processing nodes 1 10-1 .110-S, being depicted as examples in FIG. 1 ), which may be part of, for example, a particular muiticore central processing unit (CPU) package or multiple muiticore CPU packages. As depicted in Fig. 1 , each processing node 110 contains CPU processing cores 1 12 (processing cores 112-1 , 112-2. . .112-Q, being depicted in Fig. 1 for each node 1 10) and a local memory 114.

[0021] The processing cores 112 experience relatively rapid access times to the associated local memory 1 14 of the same processing node 1 10, as compared to, for example, the times to access the memory 1 14 of another processing node 110. In this manner, access to a memory 114 of another processing node 110 occurs through a memory hub 120 of the machine 100 or another interface, which introduces delays. In accordance with example implementations, each processing node 110 contains a memory controller (not shown). In accordance with example implementations, the muiticore machine 100 may have a non-uniform memory access architecture (NUMA); and the processing nodes 110 may be NUMA nodes. Fig. 1 aiso depicts a persistent memory 130 (a non-voiatiie memory, such as flash memory, for example) that may be accessed by the processing cores 112 via the memory hub 120. The persistent memory 130 may store a data set. The data set may store a large volume of data. In some cases, the data stored in the data set may be represented as multidimensional data points (or simply "data points"). In some cases, the data set 160 is organized as files in a distributed file system.

[0022] In accordance with an example illustrated in FIG. 1 , the multicore machine 100 may also include a search system 140 comprising of, among other things, a shuffle-based configuration module 142 and a search coordinator module 144. The shuffle-based configuration module 142 may distribute the data set 160 across the processing nodes according to a distribution pattern that maintains a similar distribution pattern between data points from the data set stored in the different locai memory units. As described further herein, once the shuffle-based configuration module 142 distributes the data set to the processing nodes, the processing nodes may generate, and store in iocal memory hash indexes derived from common search distances. As described below, the common search distances may be supplied by the shuffle-based configuration module 142. In accordance with some example implementations, the shuffle-based configuration module 142 may be configured to issue commands to the processing nodes 110 that initiate index building on the processing nodes 110. In some cases, these commands may include common search distances derived from an analysis of the data distribution in the data partitions.

[0023] The search coordinator module 144 may receive query request, which can specify a given muitidimensional data point to be searched as part of a search query. The search coordinator module 144 can then return query output data (also referred to as a query response) that represents a top K number of similarity matches (also herein referred to as the "search size threshold" or the "top K results") to the search query. In accordance with some example implementations, bi-directional

communication occurs between the processing nodes 110 and the search

coordinator module 144. For example, the search coordinator module 144 may issue commands that initiate local searches in the processing nodes. The local searches may execute by progressively performing searches on the data partitions based on progressively increasing the common search distances. The search coordinator module 144 may notify the client device when cumulative search results from local search results exceed a search result size threshold.

[0024] In accordance with example implementations, the shuff!e-based configuration module 142 and the search coordinator module 144 may be

components that are formed by the execution of machine executable instructions by one or more of the processing cores 112, or by externa! systems and devices communicativeiy coupled to the multicore machine 100.

[0025] Example operational aspects of a multicore machine are now discussed in greater detail.

[0026] FIG. 2 is a flowchart illustrating a method 200 for assigning data points of a data set to multiple processing nodes of a multicore system, according to an exampie. The method 200 may be performed by the modules, components, systems shown in FIG. 1 , and, accordingly, is described herein merely by way of reference thereto, it will be appreciated that the method 200 may, however, be performed on any suitable hardware.

[0027] The method 200 may begin when, at operation 202, a shuffle-based configuration module assigns a first data point from a data set to a data partition stored in a first processing node of a multicore machine. This assignment may be based on a first hash value the shuffle-based configuration module derives from a first identifier assigned to the first data point, in some cases, the data set may be obtained from a persistent storage device of the multicore machine or received from an external storage device or a communication input/output device.

[0028] At operation 204, the shuffle-based configuration module may assign a second point from the data set to a second data partition stored in a second processing node of the multicore machine based on a second hash value derived from a second identifier assigned to the second data point. [0029] Examples of further implementations of operations 202, 204 are described in the foregoing, with reference to FIG. 3.

[0030] With continued reference to FIG. 2, at operation 206, the first processing node may store a first set of hash indexes in the first processing node. The first set of hash indexes may individually correspond to common search distances. For example, in a LSH scheme, each index may correspond to a common search distance (e.g., R1 , R2, ... RN).

[0031] At operation 208, the second processing node may store a second set of hash indexes in the second processing node. Similar to the first set of hash indexes, the second set of hash indexes may individually correspond to the common search distances. Accordingly, the first set of hash indexes and the second set of hash indexes may correspond to the same search distances. Example implementations for generating common search distances is described in the foregoing with reference to FIG. 4.

[0032] FIG. 3 is a block diagram illustrating an example operation of a shuffle- based configuration module 300 that may be used to partition a dataset onto multiple processing nodes in a multicore machine, according to an example. The shuffle- based configuration module 300 shown in FIG. 3 includes readers 302a-m, a hashing module 306, and writers 304a-n. in some cases the shuffle-based configuration module 300 may be an implementation of the shuffle-based

configuration module 142 shown in FIG. 1.

[0033] The readers 302a-m may be threads of execution executing on a processing node of a multicore machine or a machine communicatively coupled to the multicore machine. FIG. 3 shows that there are m readers. Each reader may be configured to read a portion of the data set 160 to which the reader is assigned. A portion can be a block of data if the data set is stored in distributed file system, such as a Hadoop File System. In the context of the Hadoop File System, which has a default file system block size of 64 MB, the number of readers (e.g., m) can determined by the total number of the blocks occupied by the data set in the Hadoop File System. [0034] The writers 304a-n, similar to the readers 302a-m, may be threads of execution executing on a processing node of a mu!ticore machine or a machine communicatively coupled to the multicore machine. FIG. 3 shows that there are n writers, which can be different from m. The number n can be determined by the total data size of the data set, over the designed partition size. For example, a partition size can be 4 million data points. Correspondingly, a partition size can be 1 GB, 4 GB, 12 GB, or any other suitable data size, which can be significantly different from the distributed file system's block size. In the context of a Hadoop File System, the writers 304a-n write to the files stored in a distributed file system. In this case, a partition can be a file.

[0035] The hashing module 306 may also be threads of execution executing on a processing node of a multicore machine or a machine communicatively coupled to the multicore machine. The hashing module 306 may be configured to map data points from the readers 302a-m to the writers 304a-n. For example, a reader may read a data point stored in the data set. The data point then may be communicated (via pull or push methods) to the hashing module 306. Hashing module 306 can then determine which destination writer is to process the data point, in some implementations, the data point can have an identifier, such as a file name, a unique sequence identifier in the whole data set, or any other data that uniquely references the data point. Accordingly, the hashing module 306 can map the data point to a writer according to:

Hash_Function (ID) % N.

[0036] The Hash_Function may be a hash function. It is to be appreciated that the Hash_Function is different from the LSH hashing scheme discussed above in that Hash_Function (and the hashing module 306 in general) is used to map data points to data partitions rather than generating hash-indexes. There can be any number of suitable hash functions, depending on implementation, such as, for example, the MD5 hash function, the SHA-1 hash function, and the like. ID may be an identifier for the data point. N may be the number of writers. [0037] Eventually the destination writer receives the data point from the hashing module 306. The writer then writes or otherwise stores the data point to a corresponding data partition 310. A "data partition," such as the data partition 310, may be a unit of data storage used to store a subset of the data points that are distributed from the data source 160. With momentary reference to FIG. 1 , local memory 114 of a processing node 110 may store the data partition 310, as, for example, a file. In one implementation, the operations of the shuffle-based configuration module 300 can be implemented by a MapReduce framework.

[0038] Mapping data points to data partitions using techniques similar to techniques shown in FIG. 3 may have many practical applications. For example, in large data sets, data points that are adjacent to each other are likely obtained from the same data source (spatial locality) or get collected at time instants that are close to each other (temporal locality). If example implementations simply partition the data set based on how the data set is stored on the file system, such spatial and temporal locality will be preserved on individual partitions and, as a result, different partitions will be statistically dissimilar. However, mapping the data points to partitions using a hashing function, for example, may produce partitions that are statistically similar to the original data set. Having statistically similar data partitions can, in some cases, allow for common search distances to be used when generating the indexes for each partition. Using common search distances on statistically similar partitions can lead to advantages when performing a similarity search on the data partitions. For example, from the perspective of the design of the coordinator, it is much easier to determine when the search results obtained at any point are sufficient to determine the K closest points to a given query if all the data partitions use identical search parameters, particularly the search distances.

[0039] As just described, partitioning a data set into multiple data partitions using a hash-based approach may allow a shuffle-based configuration module to generate common search distances. FIG. 4 is a flow chart illustrating a method 400 for generating common search distances based on data points stored in data partitions, according to an example. [0040] FIG. 4 shows that the method 400 may begin at operation 402 when the shuffle-based configuration module selects a determinable number of data points from each data partition. The selection performed at operation may be done at random, in some cases. These data points may be referred to as a representative query set.

[0041] At operation 404, for each data point in the representative query set, the shuffle-based configuration module may: (a) retrieve the K closest points in each of the data partitions; and (b) compute the distances of the closest points in each of the data partitions.

[0042] At operation 406, the shuffle based configuration module may estimate the cumulative distribution function for the distance of the K-th closest point based on the K closest distances returned by the partitions for the representative queries.

[0043] At operation 408, the common search distances can be computed. For example, given the cumulative distribution function, the R values can be computed in different ways to optimize various objective functions. One such choice is described below. Let F(x) be the estimated cumulative distribution function of the distance of the K-th closest point to a random query. Then a set of p R values R1 , R2,...Rp, are selected to be the values that satisfy F(Ri) = i/p for i=1 , 2, ...p. This choice makes the probability that the query is answered by the index corresponding to Ri uniform over i=1.2, ... p. More generally, instead of a uniform distribution, if a certain distribution {q1 , q2, ... qp) is desired, then one can choose the distance to be the ones that satisfy F(Ri) = (q1 +q2...+qi).

[0044] Once the shuffle-based configuration module has calculated the common search distances, the shuffle-based configuration module may send a command to the processing nodes to generate indexes based on the common search distances.

[0045] Once the various processing nodes have generated hash indexes, the processing nodes may begin handling query requests. FIG. 5 is a sequence diagram illustrating a method 500 for handling query requests, according to an example. [0046] The method 500 may begin at operation 510 when the search coordinator module receives a query search request 510 from a client device 502. Responsive to receiving the query request, the search coordinator module 504 may distribute local query requests 512 to the processing nodes 506a-n, each of which may correspond to a different data partition. After the search coordinator module 504 distributes the local query requests 512, the search coordinator module 504 may block 514 to receive local query results from the processing nodes 506a-n.

[0047] When a processing node receives a local query request, the processing node can perform a search on the corresponding data partition. FIG. 5 shows that the processing nodes 506a-n may perform local searches 516a-n, in parallel, on the data partitions stored in local memory of the processing node, in some

implementations, the processing nodes may perform the local searches using a LSH scheme. In this case, the local searches 516a-n may use a first search distance value (e.g., R0).

[0048] When a processing node completes a local search, the processing node sends a local query response back to the search coordinator module. In some cases, the processing nodes 506a-n send local query responses 515 independent of each other, it is possible that there are no top-K search results to be found at the given search distance used to perform the search (e.g., R0) for a processing node. In that case, some implementations of the processing node can send a NULL search result to the search coordinator module 504. After the processing node sends the local query response, the processing node may then proceed by performing searches 522a using the next search distance, say, in this case, R1.

[0049] As just described, the search coordinator module blocks processing of the query request (but may, in some cases, continue processing other query requests) until receiving local query responses from each of the processing nodes. Once the local query responses are received, the search coordinator module evaluates the search results found in the local query responses to determine whether the results (e.g., the number of data points within the search results) exceeds the result size threshold. In the context of LSH, the local query responses 515 may relate to a search distance RO, and, accordingly, the search coordinator module may determine whether there are at least K points within distance RO across the different local query responses. If not, the search coordinator module 504 blocks 520 for the next set of local query responses 523. If, on the other hand, the local query responses exceed the result size threshold (as may be determined at 524), the search coordinator module04 will send a query result response 526 to the client device 502. At the same time, the search coordinator module 504 may send a local query abandonment command 525 to the processing nodes 506a-n.

[0050] Once a processing node receives a local query abandonment command, and if the local search is still in progress, then the processing node can terminate the search. In some cases, the processing node terminates the local search after the current iteration of the local search completes processing, in order to maintain internal states that are tracked.

[0051] A note about possible execution implementations for the processing nodes. Once a processing node sends a local query response for a given search distance (e.g., say R0), that processing node may immediately advance to perform a local search using the next search distance (e.g., say R1 ). in some cases, before starting the local search using the next search distance, the processing node can check whether an early abandonment command for the local query request has been received. If so, the processing node may abandon the local search; otherwise, the processing node may advance to the next search distance. Therefore, the search coordinator module does not restart the processing node for each search distance in a synchronized manner.

[0052] The early abandonment techniques described herein may be used to more efficiently manage computational resources of a multicore machine. For example, causing a processing node to abandon execution of local searches allows the processing node to use computational resources that would otherwise be occupied for processing a local query request, Q1 , to be given away to a different local query request, Q2. This computational resource transfer can happen earlier than in the case where the processing node terminates itself based on its local search criteria, namely, K search results are found for the local processing node. Therefore, early abandonment is able to speed up concurrent query processing in these cases.

[0053] FIG. 5 also shows that when a processing node returns a local query response for a search distance (e.g., R0) to the search coordinator module, the processing node may immediately advance to the next search distance (e.g., R1 ), if the processing node discovers less than K points within distance R0 from the query in its partition. However, globally, the search coordinator module may be able to discover at least K such points by combining the results from other processing nodes, and thus the query request can be completed early. But this global decision wiil not be known to the processing nodes until ail of the local query results related to R0, from all the processing nodes, have been gathered by the search coordinator module and the resultant decision notified to the processing nodes. Instead of the processing nodes potentially wasting CPU resources by advancing to the next search distance, some implementations of the processing nodes can process the next global query request, Q2. This might mean processing the appropriate search distance value for Q2 completely independent of the progress Q1 is making.

[0054] FIG. 6 is a diagram illustrating a worker thread 600 of a processing node capable of processing concurrent local query requests, according to an example. The worker thread 600 may be configured to execute on a processing node of a muiticore machine. In some cases, a processing nodes may execute multiple worker threads that are each assigned to different incoming local query requests received from a search coordinator module. As FIG. 6 shows, the worker thread 600 may include a local query queue 602 to store local query requests that are in progress, an early abandonment queue 604 to store eariy abandonment commands associated with iocal queries being processed by the worker thread 600, and a iocal execution engine 606 to perform a distance based similarity search on the data partition that is Iocal to the processing node executing the worker thread 600.

[0055] Operationally, data or logic (e.g., a listener thread) executing on a processing node may route a local query request to the worker thread 600. In turn, the worker thread may store the iocal query request in the locai query queue 602 as a local query record, such as one of the Iocal query records 610a-n. A iocal query record may include, among other things, a query identifier {e.g., Q) to uniquely identify a query from other Iocal queries and a current search distance (e.g., R) to specify the search distance to which the iocal query should be executed. The locai query record may include other data, such as the iocal top-K search results that have been identified so far, for example.

[0056] Independently, data or logic (e.g., another listener thread or the same) executing on the processing node may route early abandonment commands to the worker thread 600. in turn, the worker thread may store the early abandonment commands in the abandonment queue 604 as an abandonment record, such as abandonment record 612. An abandonment record may include, among other things, a query identifier to uniquely identify a locai query from other locai queries.

[0057] The Iocal execution engine 606 operates by retrieving a Iocal query record from the Iocal query queue and then checks whether an abandonment record matches the iocal query record (e.g., the locai query record and the abandonment record include the same query identifier). If there is a match, the iocal execution engine will drop the local query record and remove the corresponding abandonment record. If not, the Iocal execution executes a proximity search using the parameters recorded in the locai query record. Once the proximity search completes, the locai query response is returned to the search coordinator module and the Iocal query record is pushed to the end of the Iocal query queue 602 after incrementing or otherwise advancing the current search distance of that Iocal query record. In some cases, the Iocal execution engine 606 may evaluate the iocal query response includes a number of resuits that exceed the result size threshoid. If so, the local execution engine 606 may drop the Iocal query record from the Iocal query queue 602 instead of pushing the record to the back of the iocal query queue 602.

[0058] Once the Iocal execution engine 606 reaches the end of the Iocal query queue 602, which constitutes a pass, the Iocal execution engine 606 returns to the beginning of the Iocal query queue 602 and starts the next pass, in some cases, before performing the next pass, the iocal execution engine 606 may re-prioritize the local query queue 602 based on the current search distances of the local query records 610a-n. For example, in some cases, the local execution engine 606 may re-order the local query queue 602 in non-descending order based on the values of the current search distances of the records.

[0059] Thus, in some cases, instead of being limited to processing just one query at a time, some processing nodes can be implemented to handle multiple queries concurrently.

[0060] While the present invention has been described with respect to a limited number of embodiments, those skilled in the art, having the benefit of this disciosure, will appreciate numerous modifications and variations therefrom. It is intended that the appended claims cover ail such modifications and variations as fall within the true spirit and scope of this present invention.

Claims

Claims What is ciaimed is:

1. A method comprising:

assigning a first data point from a data set to a first data partition stored in a first processing node of a multicore machine based on a first hash vaiue derived from a first identifier assigned to the first data point;

assigning a second data point from the data set to a second data partition stored in a second processing node of the multicore machine based on a second hash vaiue derived from a second identifier assigned to the second data point;

storing a first set of hash indexes in the first processing node, the first set of hash indexes individually corresponding to common search distances; and

storing a second set of hash indexes in the second processing node, the second set of hash indexes individuaily corresponding to the common search distances.

2. The method of claim 1 , further comprising:

calculating the common search distances based on:

generating a representative query set based on a random number of data points selected from both the first data partition and the second data partition;

for each query in the represented query set:

retrieve K closest data points from the first data partition and the second data partition, and

compute distances from the each query and the K closest data points; and

estimate a cumulative distribution function for a distance of a K- th closest data point based on the distances, wherein the common search distances are calculated based on the cumulative distribution function.

3. The method of claim 1 , further comprising, prior to storing the first set of hash indexes and the second set of hash indexes, calculating the common search distances based on a combination of distribution properties shared by the first data partition and the second data partition.

4. The method of claim 3, further comprising, after calculating the common search distances, generating, at the first processing node, the first set of hash indexes based on using hash functions parameterized in accordance to the common search distances.

5. The method of claim 1 , wherein the first set of hash indexes are populated according to a locality sensitivity hashing (LSH) scheme.

6. The method of claim 1 , wherein the first data partition and the second data partition are files stored in a distributed file system accessible from the first processing node and the second processing node.

7. The method of claim 1 , further comprising:

receiving a query request, the query request including a query data point; sending local query requests to both the first processing node and the second processing node, the local query requests each including the query data point; receiving a first local query response from the first processing node and a second local query response from the second processing node;

merging results of the first local query response and the second local query response into a first candidate search result set; and

responsive to a determination that the first candidate search result set satisfies a result size threshold, causing the first processing node and the second processing node to terminate execution of the local query requests.

8. The method of claim 7, wherein the first local query response and the second local query response correspond to a first search distance.

9. The method of claim 8, further comprising, before receiving the first local query response and the second local query response:

receiving a third local query response from the first processing node and a fourth local query response from the second processing node, the third local query response and the fourth local query response corresponding to a second search distance;

merging results of the third local query response and the fourth local query response into a second candidate search result set; and

responsive to a determination that the second candidate search result set fails to exceed the result size threshold, advancing a current search distance to be the first search distance.

10. The method of claim 7, further comprising, before making the determination, pruning the first candidate search result of data points that exceed the first search distance.

11. A machine-readable storage device comprising instructions that, when executed, cause a processor to:

distribute a data set across a first processing node and a second processing node according to a distribution pattern that maintains a similar distribution pattern between data points from the data set stored in a first local memory unit of the first processing node and data points from the data set stored in a second local memory of the second processing node,

calculate common search distances based on an analysis of the distributed data set; and

cause the first processing node and the second processing node to create hash indexes with the common search distances.

12. The machine-readable storage device of claim 11 , wherein the instructions further to cause, when executed, the processor to distribute the data set by storing data points in different files in a distributed file system.

13. An apparatus comprising:

a first processing node that includes a first set of processing cores with access to a first local memory unit, the first local memory unit storing a fist data partition and a first hash index constructed with common search distances;

a second processing node that includes a second set of processing cores with access to a second local memory unit, the second local memory unit storing a second data partition and a second hash index constructed with the common search distances; and

a search coordinator to:

initiate local searches in the first processing node and the second processing node, the local searches progressively performing searches on the first data partition and the second data partitions based on progressively increasing the common search distances, and

notify a client device when cumulative search results from local search results exceed a search result size threshold.

14. The apparatus of claim 13, wherein the first hash index is constructed according to a locality sensitivity hashing (LSH) scheme.

15. The apparatus of claim 14, wherein, when the cumulative search results from the local search results exceed the search result size threshold, the search coordinator to further send abandonment commands to the first processing node and the second processing node.