WO2009051809A1 - Ranking and providing search results based in part on a number of click-through features - Google Patents
Ranking and providing search results based in part on a number of click-through features Download PDFInfo
- Publication number
- WO2009051809A1 WO2009051809A1 PCT/US2008/011894 US2008011894W WO2009051809A1 WO 2009051809 A1 WO2009051809 A1 WO 2009051809A1 US 2008011894 W US2008011894 W US 2008011894W WO 2009051809 A1 WO2009051809 A1 WO 2009051809A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- ranking
- search
- search result
- query
- click
- Prior art date
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/951—Indexing; Web crawling techniques
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
Embodiments are configured to provide information based on a user query. In an embodiment, a system includes a search component having a ranking component that can be used to rank search results as part of a query response. In one embodiment, the ranking component includes a ranking algorithm that can use one or more click-through features to rank search results which may be returned in response to a query. Other embodiments are available.
Description
RANKING AND PROVIDING SEARCH RESULTS BASED IN PART ON A NLMBER OF CLICK-THROUGH FEATURES
RELATED APPLICATIONS [0001] This application is being filed on 17 October 2008, as a PCT International Patent application in the name of Microsoft Corporation, a U.S. national corporation, applicant for the designation of all countries except the U.S., and Dmitriy Meyerzon, a citizen of the U.S., Yauhen Shnitko, a citizen of Belarus, and Michael J. Taylor, a citizen of Great Britain, applicants for the designation of the U.S. only, and claims priority to U.S. Utility Patent Application Serial No. 11/874, 579 filed on 18 October 2008.
BACKGROUND
[0002] Computer users have different ways to locate information that may be locally or remotely stored. For example, search engines can be used to locate documents and other files using keywords. Search engines can also be used to perform web-based queries. A search engine attempts to return relevant results based on a query.
SUMMARY
[0003] This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended as an aid in determining the scope of the claimed subject matter.
[0004] Embodiments are configured to provide information including using one or more ranking features when providing search results. In an embodiment, a system includes a search engine that includes a ranking algorithm that can be configured to use one or more click-through ranking features to rank and provide search results based on a query.
[0005] These and other features and advantages will be apparent from a reading of the following detailed description and a review of the associated drawings. It is to be understood that both the foregoing general description and the following detailed description are explanatory only and are not restrictive of the invention as claimed.
BRIEF DESCRIPTION OF THE DRAWINGS
[0006] FIGURE 1 depicts a block diagram of an example system configured to manage information.
[0007] FIGURE 2 is a flow diagram depicting an example of a ranking and query process.
[0008] FIGURE 3 is a flow diagram depicting an example of a ranking and query process.
[0009] FIGURE 4 is a block diagram illustrating a computing environment for implementation of various embodiments described herein.
DETAILED DESCRIPTION
[0010] Embodiments are configured to provide information including using one or more ranking features when providing search results. In an embodiment, a system includes a search engine that includes a ranking algorithm that can be configured to use one or more click-through ranking features to rank and provide search results based on a query. In one embodiment, a system includes a ranking component that can use a click parameter, a skip parameter, and one or more stream parameters to rank and provide a search result. [0011] In one embodiment, a system includes a search component which comprises a searching application that can be included as part of a computer- readable storage medium. The searching application can be used to provide search results based in part on a user query and other user action and/or inaction. For example, a user can input keywords to the search application and the search application can use the keywords to return relevant search results. The user may or may not click on a search result for more information. As described below, the search application can use prior action and prior inaction based information when ranking and returning search results. Correspondingly, the search application can use user interactions based on a search result to provide additional focus when returning relevant search results. For example, the search application can use click- through information when ranking search results and returning the ranked search results based on a user query.
[0012] FIGURE 1 is a block diagram of a system 100 which includes indexing, searching, and other functionality. For example, the system 100 can include indexing, searching, and other applications that can be used to index information as
part of an indexed data structure and search for relevant data using the indexed data structure. As described below, components of the system 100 can be used to rank and return search results based at least in part on a query. For example, components of the system 100 can be configured to provide web-based search engine functionality that can be used to return search results to a user browser, based in part on a submitted query which may consist of one or more keywords, phrases, and other search items. A user can submit queries to the search component 102 using a user interface 103, such as a browser or search window for example. [0013] As shown in FIGURE 1, the system 100 includes a search component 102, such as a search engine for example, that can be configured to return results based in part on a query input. For example, the search component 102 can operate to use a word, words, phrases, concepts, and other data to locate relevant files, documents, web pages, and other information. The search component 102 can operate to locate information and can be used by an operating system (OS), file system, web-based system, or other system. The search component 102 can also be included as an add-in component, wherein the searching functionality can be used by a host system or application.
[0014] The search component 102 can be configured to provide search results (uniform resource locaters (URLs) for example) that may be associated with files, such as documents for example, file content, virtual content, web-based content, and other information. For example, the search component 102 may use text, property information, and/or metadata when returning search results associated with local files, remotely networked files, combinations of local and remote files, etc. In one embodiment, the search component 102 can interact with a file system, virtual web, network, or other information source when providing search results.
[0015] The search component 102 includes a ranking component 104 that can be configured to rank search results based at least in part on a ranking algorithm 106 and one or more ranking features 108. In one embodiment, the ranking algorithm 106 can be configured to provide a number or other variable that can be used for sorting purposes by the search component 102. The ranking features 108 can be described as basic inputs or raw numbers that can be used when identifying relevance of a search result. The ranking features 108 can be collected, stored, and maintained in a database component 110.
[0016] For example, the click-through ranking features can be stored and maintained using a number of query logging tables which can also contain query information associated with user queries. In an alternative embodiment, the ranking features 108 can be stored and maintained in a dedicated store, including local, remote, and other storage mediums. One or more of the ranking features 108 can be input to the ranking algorithm 106, and the ranking algorithm 106 can operate to rank search results as part of a ranking determination. As described below, in one embodiment, the ranking component 104 can manipulate one or more ranking features 108 as part of the ranking determination. [0017] Correspondingly, the search component 102 can use the ranking component 104 and associated ranking algorithm 106 when using one or more of the ranking features 108 as part of a ranking determination to provide search results. Search results can be provided based on a relevance ranking or some other ranking. For example, the search component 102 can render the search results from most relevant to least relevant based at least in part on the relevance determination providing by the ranking component 104 using one or more of the ranking features 108.
[0018] With continuing reference to FIGURE 1, the system 100 also includes an index component 112 that can be used to index information. The index component 112 can be used to index and catalog information to be stored in the database component 110. Moreover, the index component 102 can use the metadata, content, and/or other information when indexing against a number of disparate information sources. For example, the index component 112 can be used to build an inverted index data structure that maps keywords to documents, including URLs associated with documents.
[0019] The search component 102 can use the indexed information when returning relevant search results according to the ranking provided by the ranking component 104. In an embodiment, as part of a search, the search component 102 can be configured to identify a set of candidate results, such as a number of candidate documents for example, that contain a portion or all of a user's query information, such as keywords and phrases for example. For example, query information may be located in a document's body or metadata, or additional metadata associated with a document that can be stored in other documents or data stores (such as anchor text for example). As described below, rather than returning
an entire set of search results if the set is large, the search component 102 can use the ranking component 104 to rank the candidates with respect to relevance or some other criteria, and return a subset of the entire set based at least in part on the ranking determination. However, if the set of candidates is not too large, the search component 102 can operate to return the entire set.
[0020] In an embodiment, the ranking component 104 can use the ranking algorithm 106 to predict a degree of relevance of a candidate associated with a particular query. For example, the ranking algorithm 106 can calculate a rank value associated with a candidate search result, wherein a higher rank value corresponds with a more relevant candidate. Multiple features, including one or more ranking features 108, can be input into the ranking algorithm 106 which can then compute an output that enables the search component 102 to sort candidates by a rank or some other criteria. The search component 102 can use the ranking algorithm 106 to prevent the user from having to inspect an entire set of candidates, such as large volume internet candidates and enterprise URL collections for example, by limiting a set of candidates according to rank.
[0021] In one embodiment, the search component 102 can monitor and collect action-based and/or inaction-based ranking features. The action-based and inaction- based ranking features can be stored in the database component 110 and updated as necessary. For example, click-through information, can be monitored and stored in the database component 110 as one or more ranking features 108 when a user interacts with, such as by clicking, a search result. The information can also be used to track when a user does not interact with a search result. For example, a user may skip over and not click on one or more search results. In an alternative embodiment, a separate component, such as an input detector or other recording component for example, can be used to monitor user interactions associated with a search result or results.
[0022] The search component 102 can use a select number of the collected action-based and inaction-based ranking features as part of a relevance determination when returning search results. In one embodiment, the search component 102 can collect and use a number of click-based interaction parameters as part of a relevance determination when returning search results based on a query. For example, assume that a user clicks on a search result (e.g., a document) that was not returned at the top of the results for whatever reason. As described below, the
search component 102 can record and use the click feature to boost the rank of the clicked result the next time some user issues the same or a similar query. The search component 102 can also collect and use other interactive features and/or parameters, such as a touch input, pen input, and other affirmative user inputs. [0023] In one embodiment, the search component 102 can use one or more click-through ranking features, wherein the one or more click-through ranking features can be derived from implicit user feedback. The click-through ranking features can be collected and stored, including updated features, in a number of query logging tables of the database component 110. For example, the search component 102 can use the functionality of an integrated server platform, such as MICROSOFT OFFICE SHAREPOINT SERVER® system, to collect, store, and update interaction-based features that can be used as part of a ranking determination. The functionality of the server platform can include web content management, enterprise content services, enterprise search, shared business processes, business intelligence services, and other services.
[0024] According to this embodiment, the search component 102 can use one or more click-through ranking features as part of a ranking determination when returning search results. The search component 102 can use prior click- through information when compiling the click-through ranking features which it can use to bias ranking orderings as part of a relevance determination. As described below, the one or more click-through ranking features can be used to provide a self-tunable ranking functionality by utilizing the implicit feedback a search result receives when a user interacts or does not interact with the search result. For example, a number of search results may be provided by the search component 102 listed by relevance on a search result page, and parameters can be collected based on whether the user clicks on a search result or skips a search result.
[0025] The search component 102 can use information in the database component 110, including stored action and/or inaction based features, when ranking and providing search results. The search component 102 can use query records and information associated with prior user actions or inactions associated with a query result when providing a current list of relevant results to a requestor. For example, the search component 102 can use information associated with how other users have responded to prior search results (e.g., files, documents, feeds, etc.)
in response to the same or similar queries when providing a current list of references based on an issued user query.
[0026] In one embodiment, the search component 102 can be used in conjunction with the functionality of a serving system, such as the MICROSOFT OFFICE SHAREPOrNT SERVER® system, operating to record and use queries and/or query strings, record and use user actions and/or inactions associated with search results, and to record and use other information associated with a relevance determination. For example, the search component 102 can be used in conjunction with the functionality of the MICROSOFT OFFICE SHAREPOINT SERVER® system, to record and use issued queries along with a search result URL that may have been clicked for a particular query. The MICROSOFT OFFICE SHAREPOINT SERVER® system can also record a list of URLs that were shown or presented with a clicked URL, such as a number of URLs that were shown above a clicked URL for example,. Additionally, the MICROSOFT OFFICE SHAREPOINT SERVER® system can operate to record a search result URL that was not clicked based on a particular query. The click-through ranking features can be aggregated and used when making a relevance determination, described below. [0027] In one embodiment, a number of click-through ranking features can be aggregated and defined as follows: [0028] 1) a click parameter, Nc, which corresponds with a number of times (across all queries) that a search result (e.g., a document, file, URL, etc.) was clicked.
[0029] 2) a skip parameter, Ns, which corresponds with a number of times (across all queries) that a search result was skipped. That is, the search result was included with other search results, may have been observed by a user, but not clicked. For example, an observed or skipped search result may refer to a search result having a higher rank than a clicked result. In one embodiment, the search component 102 can use an assumption that a user scans search results from top to bottom when interacting with search results. [0030] 3) a first stream parameter, Pc, which can be represented as a text stream corresponding to a union of all query strings associated with a clicked search result. In one embodiment, the union includes all query strings for which a result was returned and clicked. Duplicates of the query strings are possible (i.e., every individual query can be used in the union operation).
[0031] 4) a second stream parameter, Ps, which can be represented as a text stream corresponding to a union of all query strings associated with a skipped search result. In one embodiment, the union includes all query strings for which a result was returned and skipped. Duplicates of the query strings are possible (i.e., every individual query can be used in the union operation).
[0032] The above-listed click-through ranking features can be collected at a desired time, such as by one or more crawling systems on some periodic basis for example, and associated with each search result. For example, one or more of the click-through ranking features can be associated with a document which was returned by the search component 102 based on a user query. Thereafter, one or more of the click-through ranking features can be input to the ranking component 104 and used with the ranking algorithm 106 as part of the ranking and relevance determination. In some cases, some search results (e.g., documents, URLs, etc.) may not include click-through information. For search results with missing click- through information, certain text properties (e.g., Pc and/or Ps streams) may be left empty and certain static parameters (e.g., Nc and Ns) may have zero values. [0033] In one embodiment, one or more of the click-through ranking features can be used with the ranking algorithm 106 which first requires collecting one or more click-through aggregates during a crawl, including full and/or incremental crawls. For example, the search component 102 can employ a crawler which can operate to crawl a file system, web-based collection, or other repository when collecting information associated with click-through ranking features and other data. One or more crawlers can be implemented for a crawl or crawls depending on the crawl target or targets and particular implementation. [0034] The search component 102 can use the collected information, including any click-through ranking features, to update query independent stores, such as a number of query logging tables for example, with one or more features that can be used when ranking search results. For example, the search component 102 can update a number of query logging tables with the click (Nc) parameter and/or the skip (Ns) parameter for each search result that includes updated click-through information. Information associated with the updated query independent stores can be also used by various components, including the index component 102 when performing indexing operations.
[0035] Accordingly, the index component 112 can periodically obtain any changes or updates from one or more independent stores. Moreover, the index component 112 can periodically update one or more indexes which can include one or more dynamic and other features. In one embodiment, the system 100 can include two indexes, a main index and a secondary index for example, that the search component 102 can use to serve a query. The first (main) index can be used to index keywords from document bodies and/or metadata associated with web sites, file servers, and other information repositories. The secondary index can be used to index additional textual and static features that may not be directly obtained from a document. For example, additional textual and static features may include anchor text, click distance, click data, etc.
[0036] The secondary index also allows for separate update schedules. For example, when a new document is clicked, to index the associated data only requires partially rebuilding the secondary index. Thus, the main index can remain unchanged and the entire document does not require re-crawling. The main index structure can be structures as an inverted index and can be used to map keywords to document IDs, but is not so limited. For example, the index component 112 can update a secondary index using the first stream parameter Pc and/or the second stream parameter Ps for each search result that includes updated click-through information. Thereafter, one or more of the click-through ranking features and associated parameters can be applied and used by the search component 102, such as one or more inputs to the ranking algorithm 106 as part of a relevance determination associated with a query execution. [0037] As described below, a two layer neural network can be used as part of a relevance determination. In one embodiment, the implementation of the two layer neural network includes a training phase and a ranking phase as part of a forward propagation process using the two layer neural network. A lambda ranking model can be used as a training algorithm (see C. Burges, R. Ragno, Q. V. Le, "Learning To Rank With Nonsmooth Cost Functions" in Schόlkopf, Platt and Hofmann (Ed.) Advances in Neural Information Processing Systems 19, Proceedings of the 2006 Conference, (MIT Press, 2006) during the training phase, and a neural net forward propagation model can be used as part of the ranking determination. For example, a standard neural net forward propagation model can be used as part of the ranking phase. One or more of the click-through ranking features can be used in conjunction
with the two layer neural network as part of a relevance determination when returning query results based on a user query.
[0038] In an embodiment, the ranking component 104 utilizes a ranking algorithm 106 which comprises a two layer neural network scoring function, hereinafter "scoring function," which includes:
[0039] wherein,
A, = tanhl (!>( - w 1 + t 1 (Ia)
[0040] wherein, [0041] hj is an output of hidden nodey,
[0042] x, is an input value from input node i, such as one or more ranking feature inputs,
[0043] w2j is a weight to be applied to a hidden node output,
[0044] w,j is a weight to be applied to input value x, by hidden nodey, [0045] tj is the threshold value for hidden nodey,
[0047] In an alternative embodiment, other functions having similar properties and characteristics as the tanh function can be used above, hi one embodiment, the variable x, can represent one or more click-through parameters. A λ-rank training algorithm can be used to train the two layer neural network scoring function before ranking as part of a relevance determination. Moreover, new features and parameters can be added to the scoring function without significantly affecting a training accuracy or training speed. [0048] One or more ranking features 108 can be input and used by the ranking algorithm 106, the two layer neural network scoring function for this embodiment, when making a relevance determination when returning search results based on a user query, hi one embodiment, one or more click-through ranking parameters (Nc, Ns, Pc, and/or Ps can be input and used by the ranking algorithm 106 when making a relevance determination as part of returning search results based on a user query.
[0049] The Nc parameter can be used to produce an additional input to the two layer neural net scoring function. In one embodiment, the input value associated with the Nc parameter can be calculated according to the following formula: input =
(2)
[0050] wherein,
[0051] in one embodiment, the Nc parameter corresponds with a raw parameter value associated with a number of times (across all queries and all users) that a search result was clicked.
[0052] KNC is a tunable parameter (e.g., greater than or equal to zero).
[0053] MNC and SN0 are mean and standard deviation parameters or normalization constants associated with training data, and,
[0054] /Nc corresponds with an index of an input mode. [0055] The Ns parameter can be used to produce an additional input to the two layer neural net scoring function. In one embodiment, the input value associated with the Ns parameter can be calculated according to the following formula: input =
[0057] in one embodiment, the Ns parameter corresponds with a raw parameter value associated with a number of times (across all queries and all users) that a search result was skipped.
[0058] KM is a tunable parameter (e.g., greater than or equal to zero),. [0059] MNS and SNS are mean and standard deviation parameters or normalization constants associated with training data, and,. [0060] iNs corresponds with an index of an input node.
[0061] The Pc parameter can be incorporated into the formula (4) below which can be used to produce a content dependent input to the two layer neural net scoring function. input =
(4)
[0062] The formula for TF', can be calculated as follows:
[0063] wherein,
[0064] Q is a query string, [0065] t is an individual query term (e.g., word),
[0066] D is a result (e.g., document) being scored,
[0067] p is an individual property of a result (e.g., document) (for example, title, body, anchor text, author, etc. and any other textual property to be used for ranking, [0068] N is a total number of results (e.g., documents) in a search domain,
[0069] nt is a number of results (e.g., documents) containing term t,
[0070] DLp is a length of the property p,
[0071] A VDLp is an average length of the property/?,
[0072] TF tp is a term t frequency in the property/?, [0073] TFtpc corresponds to a number of times that a given term appears in the parameter Pc,
[0074] DLpc corresponds with a length of the parameter Pc (e.g., the number of terms included),
[0075] A VDLpc corresponds with an average length of the parameter Pc, [0076] Wpc and bpc correspond with tunable parameters,
[0077] D\Pc corresponds with a set of properties of a document D excluding property Pc (item for Pc is taken outside of the sum sign only for clarity),
[0078] iBM25main is an index of an input node, and,
[0079] M and S represent mean and standard deviation normalization constants.
[0080] The Ps parameter can be incorporated into the formula (6) below which can be used to produce an additional input to the two layer neural net scoring function. input =
^iP1 = BM25Gsecondary (Q. D) = ^ * <- J-
(6) [0081] where, l + bps TFt = TFt,Λ • wPl ■ DLps
\AVDLPS + bj>s) (7)
[0082] and, [0083] TFtps represent a number of times that a given term is associated with the Ps parameter,
[0084] DLps represents a length of the Ps parameter (e.g., a number of terms), [0085] A VDLps represents an average length of the Ps parameter, [0086] N represents a number of search results (e.g., documents) in a corpus, [0087] nt represents a number of search results (e.g., documents) containing a given query term,
[0088] k[, Wp3, bPs represent tunable parameters, and,
[0089] M and S represent mean and standard deviation normalization constants. [0090] Once one or more of the inputs have been calculated as shown above, one or more of the inputs can be input into (1), and a score or ranking can be output which can then be used when ranking search results as part of the relevance determination. As an example, X1 can be used to represent the calculated input associated with the Nc parameter, X2 can be used to represent the calculated input associated with the Ns parameter, X3 can be used to represent the calculated input associated with the Pc parameter, and, x4 can be used to represent the calculated input associated with the Ps parameter. As described above, streams can also include body, title, author, URL, anchor text, generated title, and/or Pc. Accordingly, one or more inputs, e.g., X1, X2, X3, and/or X4 can be input into the scoring function (1) when ranking search results as part of the relevance determination. Correspondingly, the search component 102 can provide ranked search results to a user based on an issued query and one or more ranking inputs.
For example, the search component 102 can return a set of URLs, wherein URLs within the set can be presented to the user based on a ranking order (e.g., high relevance value to low relevance value).
[0091] Other features can also be used when ranking and providing search results. In an embodiment, click distance (CD), URL depth (UD), file type or type prior (T), language or language prior (L), and/or other ranking features can be used to rank and provide search results. One or more of the additional ranking features can be used as part of a linear ranking determination, neural net determination, or other ranking determination. For example, one or more static ranking features can be used in conjunction with one or more dynamic ranking features as part of a linear ranking determination, neural net determination, or other ranking determination. [0092] Accordingly, CD represents click distance, wherein CD can be described as a query-independent ranking feature that measures a number of "clicks" required to reach a given target, such as a page or document for example, from a reference location. CD takes advantage of a hierarchical structure of a system which may follow a tree structure, with a root node (e.g., the homepage) and subsequent branches extending to other nodes from that root. Viewing the tree as a graph, CD may be represented as the shortest path between the root, as reference location, and the given page. UD represents URL depth, wherein UD can be used to represent a count of the number of slashes ("/") in a URL. T represents type prior, and, L represents language prior.
[0093] The T and L features can be used to represent enumerated data types. Examples of such a data type include file type and language type. As an example, for any given search domain, there may be a finite set of file types present and/or supported by the associated search engine. For example an enterprise intranet may contain word processing documents, spreadsheets, HTML web pages, and other documents. Each of these file types may have a different impact on the relevance of the associated document. An exemplary transformation can convert a file type value into a set of binary flags, one for each supported file type. Each of these flags can be used by a neural network individually so that each may be given a separate weight and processed separately. Language (in which the document is written) can be handled in a similar manner, with a single discrete binary flag used to indicate whether or not a document is written in a certain language. The sum of the term
frequencies may also include body, title, author, anchor text, URL display name, extracted title, etc.
[0094] Ultimately, user satisfaction is one of surest measures of the operation of the search component 102. A user would prefer that the search component 102 quickly return the most relevant results, so that the user is not required to invest much time investigating a resulting set of candidates. For example, a metric evaluation can be used to determine a level of user satisfaction. In one embodiment, a metric evaluation can be improved by varying inputs to the ranking algorithm 106 or aspects of the ranking algorithm 106. A metric evaluation can be computed over some representative or random set of queries. For example, a representative set of queries can be selected based on a random sampling of queries contained in query logs stored in the database component 110. Relevance labels can be assigned to or associated with each result returned by the search component 102 for each of the metric evaluation queries. [0095] For example, a metric evaluation may comprise an average count of relevant documents in the query at top N (1, 5, 10, etc.) results (also referred to as precision @1, 5, 10, etc.). As another example, a more complicated measure can be used to evaluate search results, such as an average precision or Normalized Discounted Cumulative Gain (NDCG). The NDCG can be described as a cumulative metric that allows multi-level judgments and penalizes the search component 102 for returning less relevant documents higher in the rank, and more relevant documents lower in the rank. A metric can be averaged over a query set to determine an overall accuracy quantification. [0096] Continuing the NDCG example, for a given query "Qi" the NDCG can be computed as:
[0097] where N is typically 3 or 10. The metric can be averaged over a query set to determine an overall accuracy number.
[0098] Below are some experimental results obtained based on using the Nc, Ns, and Pc click-through parameters with the scoring function (1). Experiments were conducted on 10-splits query set (744 queries, -130K documents), 5-fold cross-validation run. For each fold, 6 splits were used for training, 2 for validation, and 2 for testing. A standard version of a λ-rank algorithm was used (see above).
[0099] Accordingly, aggregated results using 2-layer neural net scoring function with 4 hidden nodes resulted in the following as shown in Table 1 below:
Table 1
[00100] The aggregated results using 2-layer neural net scoring function with 6 hidden nodes resulted in the following as shown in Table 2 below:
Table 2
[00101] FIGURE 2 is a flow diagram illustrating a process of providing information based in part on a user query, in accordance with an embodiment. Components of FIGURE 1 are used in the description of FIGURE 2, but the embodiment is not so limited. At 200, the search component 102 receives query data associated with a user query. For example, a user using a web-based browser can submit a text string consisting of a number of keywords which defines the user query. At 202, the search component 102 can communicate with the database component 110 to retrieve any ranking features 108 associated with the user query. For example, the search component 102 can retrieve one or more click-through ranking features from a number of query tables, wherein the one or more click- through ranking features are associated with previously issued queries having similar or identical keywords.
[00102] At 204, the search component 102 can use the user query to locate one or more search results. For example, the search component 102 can use a text string to locate documents, files, and other data structures associated with a file system, database, web-based collection, or some other information repository. At 206, the search component 102 uses one or more of the ranking features 108 to rank the search results. For example, the search component 102 can input one or more click-
through ranking parameters to the scoring function (1) which can provide an output associated with a ranking for each search result.
[00103] At 208, the search component 102 can use the rankings to provide the search results to a user in a ranked order. For example, the search component 102 can provide a number of retrieved documents to a user, wherein the retrieved documents can be presented to the user according to a numerical ranking order (e.g., a descending order, ascending order, etc.). At 210, the search component 102 can use a user action or inaction associated with a search result to update one or more ranking features 108 which may be stored in the database component 110. For example, if a user clicked or skipped a URL search result, the search component 102 can push the click-through data (click data or skip data) to a number of query logging tables of the database component 110. Thereafter, the index component 112 can operate to use the updated ranking features for various indexing operations, including indexing operations associated with updating an indexed catalog of information.
[00104] FIGURE 3 is a flow diagram illustrating a process of providing information based in part on a user query, in accordance with an embodiment. Again, components of FIGURE 1 are used in the description of FIGURE 3, but the embodiment is not so limited. The process of FIGURE 3 is subsequent to the search component 102 receiving a user query issued from the user interface 103, wherein the search component 102 has located a number of documents which satisfy the user query. For example, the search component 102 can use a number of submitted keywords to locate documents as part of a web-based search. [00105] At 300, the search component 102 obtains a next document which satisfied the user query. If all documents have been located by the search component 102 at 302, the flow proceeds to 316, wherein the search component 102 can sort the located documents according to rank. If all documents have not been located at 302, the flow proceeds to 304 and the search component 102 retrieves any click-through features from the database component 110, wherein the retrieved click-through features are associated with the current document located by the search component 102.
[00106] At 306, the search component 102 can compute an input associated with the Pc parameter for use by the scoring function (1) as part of a ranking determination. For example, the search component 102 can input the Pc parameter
into the formula (4) to compute an input associated with the Pc parameter. At 308, the search component 102 can compute a second input associated with the Nc parameter for use by the scoring function (1) as part of a ranking determination. For example, the search component 102 can input the Nc parameter into the formula (2) to compute an input associated with the Nc parameter.
[00107] At 310, the search component 102 can compute a third input associated with the Ns parameter for use by the scoring function (1) as part of a ranking determination. For example, the search component 102 can input the Ns parameter into the formula (3) to compute an input associated with the Ns parameter. At 312, the search component 102 can compute a fourth input associated with the Ps parameter for use by the scoring function (1) as part of a ranking determination. For example, the search component 102 can input the Ps parameter into the formula (6) to compute an input associated with the Ps parameter. [00108] At 314, the search component 102 operates to input one or more of the calculated inputs into the scoring function (1) to compute a rank for the current document. In alternative embodiments, the search component 102 may instead calculate input values associated with select parameters, rather than calculating inputs for each click-through parameter. If there are no remaining documents to rank, at 316 the search component 102 sorts the documents by rank. For example, the search component 102 may sort the documents according to a descending rank order, starting with a document having a highest rank value and ending with a document having a lowest rank value. The search component 102 can also use the ranking as a cutoff to limit the number of results presented to the user. For example, the search component 102 may only present documents having a rank greater than X, when providing search results. Thereafter, the search component 102 can provide the sorted documents to a user for further action or inaction. While a certain order is described with respect to FIGURES 2 and 3, the order can be changed according to a desired implementation. [00109] The embodiments and examples described herein are not intended to be limiting and other embodiments are available. Moreover, the components described above can be implemented as part of networked, distributed, or other computer- implemented environment. The components can communicate via a wired, wireless, and/or a combination of communication networks. A number of client computing
devices, including desktop computers, laptops, handhelds, or other smart devices can interact with and/or be included as part of the system 100.
[00110] In alternative embodiments, the various components can be combined and/or configured according to a desired implementation. For example, the index component 112 can be included with the search component 102 as a single component for providing indexing and searching functionality. As additional example, neural networks can be implemented either in hardware or software. While certain embodiments include software implementations, they are not so limited and they encompass hardware, or mixed hardware/software solutions. Other embodiments and configurations are available. Exemplary Operating Environment
[00111] Referring now to FIGURE 4, the following discussion is intended to provide a brief, general description of a suitable computing environment in which embodiments of the invention may be implemented. While the invention will be described in the general context of program modules that execute in conjunction with program modules that run on an operating system on a personal computer, those skilled in the art will recognize that the invention may also be implemented in combination with other types of computer systems and program modules. [00112] Generally, program modules include routines, programs, components, data structures, and other types of structures that perform particular tasks or implement particular abstract data types. Moreover, those skilled in the art will appreciate that the invention may be practiced with other computer system configurations, including hand-held devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, and the like. The invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices. [00113] Referring now to FIGURE 4, an illustrative operating environment for embodiments of the invention will be described. As shown in FIGURE 4, computer 2 comprises a general purpose desktop, laptop, handheld, or other type of computer capable of executing one or more application programs. The computer 2 includes at least one central processing unit 8 ("CPU"), a system memory 12, including a
random access memory 18 ("RAM") and a read-only memory ("ROM") 20, and a system bus 10 that couples the memory to the CPU 8. A basic input/output system containing the basic routines that help to transfer information between elements within the computer, such as during startup, is stored in the ROM 20. The computer 2 further includes a mass storage device 14 for storing an operating system 32, application programs, and other program modules.
[00114] The mass storage device 14 is connected to the CPU 8 through a mass storage controller (not shown) connected to the bus 10. The mass storage device 14 and its associated computer-readable media provide non-volatile storage for the computer 2. Although the description of computer-readable media contained herein refers to a mass storage device, such as a hard disk or CD-ROM drive, it should be appreciated by those skilled in the art that computer-readable media can be any available media that can be accessed or utilized by the computer 2. [00115] By way of example, and not limitation, computer-readable media may comprise computer storage media and communication media. Computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, RAM, ROM, EPROM, EEPROM, flash memory or other solid state memory technology, CD-ROM, digital versatile disks ("DVD"), or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by the computer 2. [00116] According to various embodiments of the invention, the computer 2 may operate in a networked environment using logical connections to remote computers through a network 4, such as a local network, the Internet, etc. for example. The computer 2 may connect to the network 4 through a network interface unit 16 connected to the bus 10. It should be appreciated that the network interface unit 16 may also be utilized to connect to other types of networks and remote computing systems. The computer 2 may also include an input/output controller 22 for receiving and processing input from a number of other devices, including a keyboard, mouse, etc. (not shown). Similarly, an input/output controller 22 may provide output to a display screen, a printer, or other type of output device.
[00117] As mentioned briefly above, a number of program modules and data files may be stored in the mass storage device 14 and RAM 18 of the computer 2, including an operating system 32 suitable for controlling the operation of a networked personal computer, such as the WINDOWS operating systems from MICROSOFT CORPORATION of Redmond, Washington. The mass storage device 14 and RAM 18 may also store one or more program modules. In particular, the mass storage device 14 and the RAM 18 may store application programs, such as a search application 24, word processing application 28, a spreadsheet application 30, e-mail application 34, drawing application, etc. [00118] It should be appreciated that various embodiments of the present invention can be implemented (1) as a sequence of computer implemented acts or program modules running on a computing system and/or (2) as interconnected machine logic circuits or circuit modules within the computing system. The implementation is a matter of choice dependent on the performance requirements of the computing system implementing the invention. Accordingly, logical operations including related algorithms can be referred to variously as operations, structural devices, acts or modules. It will be recognized by one skilled in the art that these operations, structural devices, acts and modules may be implemented in software, firmware, special purpose digital logic, and any combination thereof without deviating from the spirit and scope of the present invention as recited within the claims set forth herein.
[00119] Although the invention has been described in connection with various exemplary embodiments, those of ordinary skill in the art will understand that many modifications can be made thereto within the scope of the claims that follow. Accordingly, it is not intended that the scope of the invention in any way be limited by the above description, but instead be determined entirely by reference to the claims that follow.
Claims
1. A system for providing information comprising: a search component configured to locate a search result based on a query input; a database component configured to store information associated with the query input including one or more ranking features, wherein the one or more ranking features can be associated with a user action or user inaction associated with the search result which can be collected with respect to the search result for a same query or a similar query performed by prior users; and, a ranking component configured to rank the search result based at least in part on a ranking function and the one or more ranking features, including an action- based feature and an inaction-based feature, wherein the search component can use the rank of the search result when providing search results according to a ranking order.
2 The system of claim 1, further comprising an index component configured to use the one or more updated ranking features when performing index operations associated with a search index.
3. The system of claim 1, wherein the one or more ranking features comprise one or more dynamic ranking features selected from a group consisting of body, title, author, generated title, an anchor text, and a URL.
4. The system of claim 1, wherein the one or more ranking features comprise one or more static ranking features selected from a group consisting of click distance, URL depth, file type, and language.
5. The system of claim 1, wherein the ranking function further comprises a scoring function defined as:
Score(xλ ,- -,xn ) = V∑hj - Wl X wherein,
Xi represents one or more inputs to the scoring function, w2j represent weights of hidden nodes,
Wjj represent weights of the inputs, tj represent a number of thresholds, and, tanh is a hyperbolic tangent function.
6. The system of claim 1, wherein the ranking component can use the one or more click-through parameters when ranking the search result, wherein the one or more click-through parameters further comprise one or more of the following: a click parameter associated with a number of times that the search result has been clicked; a skip parameter associated with a number of times that the search result has been skipped; a first stream parameter corresponding to a union of query strings associated with a clicked search result; and, a second stream parameter corresponding to a union of query strings associated with a skipped search result.
7. The system of claim 6, wherein the search component is further configured to update one or more of the click-through parameters including using information associated with how a user interacted with the search result when updating the one or more of the click-through parameters.
8. The system of claim 7, wherein the search component is further configured to update the one or more click-through parameters, wherein the update of the one or more click-through parameters corresponds with a selected search result or a skipped search result by a user.
9. The system of claim 1, wherein the wherein the one or more ranking features comprise one or more dynamic ranking features selected from a group consisting of body, title, author, generated title, an anchor text, and a URL, and one or more static ranking features selected from a group consisting of click distance, URL depth, file type, and language.
10. The system of claim 6, wherein the ranking component is further configured to calculate an input value associated with the click parameter, wherein the calculated input is defined as: %')
11. The system of claim 6, wherein the search component is further configured to calculate an input value associated with the skip parameter, wherein the calculated input is defined as: {Έ^ΓΓ 1*"*)
12. The system of claim 6, wherein the search component is further configured to calculate an input value associated with the first stream parameter, wherein the calculated input is defined as:
TFr
(I ∑, tEQ U ' J. TF ' ■ ■<)) - My
and,
13. The system of claim 6, wherein the search component is further configured to calculate an input value associated with the second stream parameter, wherein the calculated input is defined as: and,
1 + 6 PS
TFr = TFtrPs ■ wP3 DL PS
\AVDL PS •+ »Λ)
14. A search engine configured to: receive information associated with a query; locate a search result associated with the query; calculate a first input associated with a click parameter and the search result; calculate a second input associated with a skip parameter and the search result; and, rank the search result using the first and second inputs.
15. The search engine of claim 14, further configured to: calculate a third input associated with a first stream parameter and the search result; calculate a fourth input associated with a second stream parameter and the search result; and, rank the search result using at least three of the first, second, third, and fourth inputs.
16. The search engine of claim 14, further configured to update a store with click parameter and skip parameter updates associated with user interactions with the search result.
17. The search engine of claim 14, further configured to update a store with stream parameter updates associated with user interactions with the search result.
18. A method of providing information comprising: receiving a query which includes one or more keywords; searching for a candidate based in part on the one or more keywords; finding query candidates based in part on the one or more keywords; determining a first input value associated with a prior user action and at least one of the query candidates; determining a second input value associated with a prior user inaction and at least one of the query candidates; and, ranking a set of the query candidates based in part on a scoring determination using a scoring function and one or more of the first and second input values.
19. The method of claim 18, further comprising: determining a third input value associated with a text stream and a user selection of at least one of the query candidates; and ranking the set of the query candidates based in part on a scoring determination using a scoring function and one or more of the first, second input, and third input values.
20. The method of claim 18, further comprising ranking a set of documents according to a numerical order.
Priority Applications (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN2008801124165A CN101828185B (en) | 2007-10-18 | 2008-10-17 | Ranking and providing search results based in part on a number of click-through features |
EP08840594A EP2212813A4 (en) | 2007-10-18 | 2008-10-17 | Ranking and providing search results based in part on a number of click-through features |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US11/874,579 | 2007-10-18 | ||
US11/874,579 US20090106221A1 (en) | 2007-10-18 | 2007-10-18 | Ranking and Providing Search Results Based In Part On A Number Of Click-Through Features |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2009051809A1 true WO2009051809A1 (en) | 2009-04-23 |
Family
ID=40564493
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/US2008/011894 WO2009051809A1 (en) | 2007-10-18 | 2008-10-17 | Ranking and providing search results based in part on a number of click-through features |
Country Status (4)
Country | Link |
---|---|
US (1) | US20090106221A1 (en) |
EP (1) | EP2212813A4 (en) |
CN (1) | CN101828185B (en) |
WO (1) | WO2009051809A1 (en) |
Families Citing this family (60)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7606793B2 (en) | 2004-09-27 | 2009-10-20 | Microsoft Corporation | System and method for scoping searches using index keys |
US7761448B2 (en) | 2004-09-30 | 2010-07-20 | Microsoft Corporation | System and method for ranking search results using click distance |
US7827181B2 (en) * | 2004-09-30 | 2010-11-02 | Microsoft Corporation | Click distance determination |
US7716198B2 (en) * | 2004-12-21 | 2010-05-11 | Microsoft Corporation | Ranking search results using feature extraction |
US7792833B2 (en) * | 2005-03-03 | 2010-09-07 | Microsoft Corporation | Ranking search results using language types |
US20060200460A1 (en) * | 2005-03-03 | 2006-09-07 | Microsoft Corporation | System and method for ranking search results using file types |
US8117197B1 (en) * | 2008-06-10 | 2012-02-14 | Surf Canyon, Inc. | Adaptive user interface for real-time search relevance feedback |
US8051072B2 (en) * | 2008-03-31 | 2011-11-01 | Yahoo! Inc. | Learning ranking functions incorporating boosted ranking in a regression framework for information retrieval and ranking |
US8812493B2 (en) | 2008-04-11 | 2014-08-19 | Microsoft Corporation | Search results ranking using editing distance and document information |
US8671093B2 (en) * | 2008-11-18 | 2014-03-11 | Yahoo! Inc. | Click model for search rankings |
US20100287152A1 (en) * | 2009-05-05 | 2010-11-11 | Paul A. Lipari | System, method and computer readable medium for web crawling |
US10303722B2 (en) | 2009-05-05 | 2019-05-28 | Oracle America, Inc. | System and method for content selection for web page indexing |
US20100287174A1 (en) * | 2009-05-11 | 2010-11-11 | Yahoo! Inc. | Identifying a level of desirability of hyperlinked information or other user selectable information |
CN101887437B (en) | 2009-05-12 | 2016-03-30 | 阿里巴巴集团控股有限公司 | A kind of Search Results generation method and information search system |
US8311792B1 (en) * | 2009-12-23 | 2012-11-13 | Intuit Inc. | System and method for ranking a posting |
US8150841B2 (en) * | 2010-01-20 | 2012-04-03 | Microsoft Corporation | Detecting spiking queries |
US8738635B2 (en) | 2010-06-01 | 2014-05-27 | Microsoft Corporation | Detection of junk in search result ranking |
US8265778B2 (en) | 2010-06-17 | 2012-09-11 | Microsoft Corporation | Event prediction using hierarchical event features |
CN102542474B (en) * | 2010-12-07 | 2015-10-21 | 阿里巴巴集团控股有限公司 | Result ranking method and device |
US10409851B2 (en) | 2011-01-31 | 2019-09-10 | Microsoft Technology Licensing, Llc | Gesture-based search |
US10444979B2 (en) | 2011-01-31 | 2019-10-15 | Microsoft Technology Licensing, Llc | Gesture-based search |
US8898156B2 (en) | 2011-03-03 | 2014-11-25 | Microsoft Corporation | Query expansion for web search |
US8326862B2 (en) * | 2011-05-01 | 2012-12-04 | Alan Mark Reznik | Systems and methods for facilitating enhancements to search engine results |
CN102841904B (en) * | 2011-06-24 | 2016-05-04 | 阿里巴巴集团控股有限公司 | A kind of searching method and equipment |
US9262513B2 (en) | 2011-06-24 | 2016-02-16 | Alibaba Group Holding Limited | Search method and apparatus |
US8965882B1 (en) | 2011-07-13 | 2015-02-24 | Google Inc. | Click or skip evaluation of synonym rules |
CN102956009B (en) | 2011-08-16 | 2017-03-01 | 阿里巴巴集团控股有限公司 | A kind of electronic commerce information based on user behavior recommends method and apparatus |
US20130097146A1 (en) * | 2011-10-05 | 2013-04-18 | Medio Systems, Inc. | Personalized ranking of categorized search results |
CN103092856B (en) * | 2011-10-31 | 2015-09-23 | 阿里巴巴集团控股有限公司 | Search result ordering method and equipment, searching method and equipment |
US8909627B1 (en) | 2011-11-30 | 2014-12-09 | Google Inc. | Fake skip evaluation of synonym rules |
CN103164804B (en) | 2011-12-16 | 2016-11-23 | 阿里巴巴集团控股有限公司 | The information-pushing method of a kind of personalization and device |
CN102521377B (en) * | 2011-12-19 | 2014-02-05 | 刘松涛 | Method and system for screening high-quality documents from document collection of document processing system |
US20130166525A1 (en) * | 2011-12-27 | 2013-06-27 | Microsoft Corporation | Providing application results based on user intent |
US9355095B2 (en) * | 2011-12-30 | 2016-05-31 | Microsoft Technology Licensing, Llc | Click noise characterization model |
US8965875B1 (en) | 2012-01-03 | 2015-02-24 | Google Inc. | Removing substitution rules based on user interactions |
US9152698B1 (en) | 2012-01-03 | 2015-10-06 | Google Inc. | Substitute term identification based on over-represented terms identification |
US9141672B1 (en) | 2012-01-25 | 2015-09-22 | Google Inc. | Click or skip evaluation of query term optionalization rule |
US9495462B2 (en) | 2012-01-27 | 2016-11-15 | Microsoft Technology Licensing, Llc | Re-ranking search results |
US10984337B2 (en) * | 2012-02-29 | 2021-04-20 | Microsoft Technology Licensing, Llc | Context-based search query formation |
WO2013169245A2 (en) * | 2012-05-09 | 2013-11-14 | Google Inc. | Generating application recommendations based on user feedback |
US8959103B1 (en) | 2012-05-25 | 2015-02-17 | Google Inc. | Click or skip evaluation of reordering rules |
US9304584B2 (en) | 2012-05-31 | 2016-04-05 | Ca, Inc. | System, apparatus, and method for identifying related content based on eye movements |
US9020927B1 (en) * | 2012-06-01 | 2015-04-28 | Google Inc. | Determining resource quality based on resource competition |
US9146966B1 (en) * | 2012-10-04 | 2015-09-29 | Google Inc. | Click or skip evaluation of proximity rules |
CN103838764B (en) * | 2012-11-26 | 2019-04-30 | 深圳市世纪光速信息技术有限公司 | A kind of search result relevance evaluating method and device |
CN103235796B (en) * | 2013-04-07 | 2019-12-24 | 北京百度网讯科技有限公司 | Search method and system based on user click behavior |
US9405803B2 (en) * | 2013-04-23 | 2016-08-02 | Google Inc. | Ranking signals in mixed corpora environments |
US9965549B2 (en) * | 2013-10-09 | 2018-05-08 | Foxwordy Inc. | Excerpted content |
US9721309B2 (en) | 2013-12-31 | 2017-08-01 | Microsoft Technology Licensing, Llc | Ranking of discussion threads in a question-and-answer forum |
US10762091B2 (en) * | 2014-09-08 | 2020-09-01 | Salesforce.Com, Inc. | Interactive feedback for changes in search relevancy parameters |
EP3096241A1 (en) * | 2015-05-18 | 2016-11-23 | Carsten Kraus | Method for searching a database of data sets |
US10007732B2 (en) | 2015-05-19 | 2018-06-26 | Microsoft Technology Licensing, Llc | Ranking content items based on preference scores |
US9720774B2 (en) * | 2015-06-29 | 2017-08-01 | Sap Se | Adaptive recovery for SCM-enabled databases |
RU2632138C2 (en) * | 2015-09-14 | 2017-10-02 | Общество С Ограниченной Ответственностью "Яндекс" | Method (options) and server of search results ranking based on utility parameter |
US20170109413A1 (en) * | 2015-10-14 | 2017-04-20 | Quixey, Inc. | Search System and Method for Updating a Scoring Model of Search Results based on a Normalized CTR |
US10496319B2 (en) | 2017-02-28 | 2019-12-03 | Sap Se | Lifecycle management for data in non-volatile memory including blocking creation of a database savepoint and associating non-volatile memory block identifiers with database column fragments |
US20190251422A1 (en) * | 2018-02-09 | 2019-08-15 | Microsoft Technology Licensing, Llc | Deep neural network architecture for search |
US11397924B1 (en) | 2019-03-27 | 2022-07-26 | Microsoft Technology Licensing, Llc | Debugging tool for recommendation systems |
US11790037B1 (en) * | 2019-03-27 | 2023-10-17 | Microsoft Technology Licensing, Llc | Down-sampling of negative signals used in training machine-learned model |
US11281640B2 (en) * | 2019-07-02 | 2022-03-22 | Walmart Apollo, Llc | Systems and methods for interleaving search results |
Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
KR20020015838A (en) * | 2000-08-23 | 2002-03-02 | 전홍건 | Method for re-adjusting ranking of document to use user's profile and entropy |
KR20030082109A (en) * | 2002-04-16 | 2003-10-22 | (주)메타웨이브 | Method and System for Providing Information and Retrieving Index Word using AND Operator |
KR20060116042A (en) * | 2005-05-09 | 2006-11-14 | 엔에이치엔(주) | Personalized search method using cookie information and system for enabling the method |
Family Cites Families (103)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US85716A (en) * | 1869-01-05 | Improvement in bee-hives | ||
US125392A (en) * | 1872-04-09 | Improvement in corn-shellers | ||
US224554A (en) * | 1880-02-17 | marbeau | ||
US6202058B1 (en) * | 1994-04-25 | 2001-03-13 | Apple Computer, Inc. | System for ranking the relevance of information objects accessed by computer users |
US5606609A (en) * | 1994-09-19 | 1997-02-25 | Scientific-Atlanta | Electronic document verification system and method |
US5594660A (en) * | 1994-09-30 | 1997-01-14 | Cirrus Logic, Inc. | Programmable audio-video synchronization method and apparatus for multimedia systems |
US5642502A (en) * | 1994-12-06 | 1997-06-24 | University Of Central Florida | Method and system for searching for relevant documents from a text database collection, using statistical ranking, relevancy feedback and small pieces of text |
US5729730A (en) * | 1995-03-28 | 1998-03-17 | Dex Information Systems, Inc. | Method and apparatus for improved information storage and retrieval system |
US5974455A (en) * | 1995-12-13 | 1999-10-26 | Digital Equipment Corporation | System for adding new entry to web page table upon receiving web page including link to another web page not having corresponding entry in web page table |
JP3113814B2 (en) * | 1996-04-17 | 2000-12-04 | インターナショナル・ビジネス・マシーンズ・コーポレ−ション | Information search method and information search device |
US6038610A (en) * | 1996-07-17 | 2000-03-14 | Microsoft Corporation | Storage of sitemaps at server sites for holding information regarding content |
EP0822502A1 (en) * | 1996-07-31 | 1998-02-04 | BRITISH TELECOMMUNICATIONS public limited company | Data access system |
US5870739A (en) * | 1996-09-20 | 1999-02-09 | Novell, Inc. | Hybrid query apparatus and method |
US5870740A (en) * | 1996-09-30 | 1999-02-09 | Apple Computer, Inc. | System and method for improving the ranking of information retrieval results for short queries |
US5893116A (en) * | 1996-09-30 | 1999-04-06 | Novell, Inc. | Accessing network resources using network resource replicator and captured login script for use when the computer is disconnected from the network |
GB2323003B (en) * | 1996-10-02 | 2001-07-04 | Nippon Telegraph & Telephone | Method and apparatus for graphically displaying hierarchical structure |
GB2331166B (en) * | 1997-11-06 | 2002-09-11 | Ibm | Database search engine |
US5890147A (en) * | 1997-03-07 | 1999-03-30 | Microsoft Corporation | Scope testing of documents in a search engine using document to folder mapping |
AUPO710597A0 (en) * | 1997-06-02 | 1997-06-26 | Knowledge Horizons Pty. Ltd. | Methods and systems for knowledge management |
US6029164A (en) * | 1997-06-16 | 2000-02-22 | Digital Equipment Corporation | Method and apparatus for organizing and accessing electronic mail messages using labels and full text and label indexing |
US6012053A (en) * | 1997-06-23 | 2000-01-04 | Lycos, Inc. | Computer system with user-controlled relevance ranking of search results |
US6182113B1 (en) * | 1997-09-16 | 2001-01-30 | International Business Machines Corporation | Dynamic multiplexing of hyperlinks and bookmarks |
US6999959B1 (en) * | 1997-10-10 | 2006-02-14 | Nec Laboratories America, Inc. | Meta search engine |
US6026398A (en) * | 1997-10-16 | 2000-02-15 | Imarket, Incorporated | System and methods for searching and matching databases |
US7010532B1 (en) * | 1997-12-31 | 2006-03-07 | International Business Machines Corporation | Low overhead methods and apparatus for shared access storage devices |
KR100285265B1 (en) * | 1998-02-25 | 2001-04-02 | 윤덕용 | Db management system and inverted index storage structure using sub-index and large-capacity object |
US6182085B1 (en) * | 1998-05-28 | 2001-01-30 | International Business Machines Corporation | Collaborative team crawling:Large scale information gathering over the internet |
US6208988B1 (en) * | 1998-06-01 | 2001-03-27 | Bigchalk.Com, Inc. | Method for identifying themes associated with a search query using metadata and for organizing documents responsive to the search query in accordance with the themes |
US6216123B1 (en) * | 1998-06-24 | 2001-04-10 | Novell, Inc. | Method and system for rapid retrieval in a full text indexing system |
JP3665480B2 (en) * | 1998-06-24 | 2005-06-29 | 富士通株式会社 | Document organizing apparatus and method |
US6199081B1 (en) * | 1998-06-30 | 2001-03-06 | Microsoft Corporation | Automatic tagging of documents and exclusion by content |
US6360215B1 (en) * | 1998-11-03 | 2002-03-19 | Inktomi Corporation | Method and apparatus for retrieving documents based on information other than document content |
US20030069873A1 (en) * | 1998-11-18 | 2003-04-10 | Kevin L. Fox | Multiple engine information retrieval and visualization system |
US6922699B2 (en) * | 1999-01-26 | 2005-07-26 | Xerox Corporation | System and method for quantitatively representing data objects in vector space |
US6862710B1 (en) * | 1999-03-23 | 2005-03-01 | Insightful Corporation | Internet navigation using soft hyperlinks |
US6336117B1 (en) * | 1999-04-30 | 2002-01-01 | International Business Machines Corporation | Content-indexing search system and method providing search results consistent with content filtering and blocking policies implemented in a blocking engine |
US6990628B1 (en) * | 1999-06-14 | 2006-01-24 | Yahoo! Inc. | Method and apparatus for measuring similarity among electronic documents |
US6873982B1 (en) * | 1999-07-16 | 2005-03-29 | International Business Machines Corporation | Ordering of database search results based on user feedback |
US6381597B1 (en) * | 1999-10-07 | 2002-04-30 | U-Know Software Corporation | Electronic shopping agent which is capable of operating with vendor sites which have disparate formats |
US7346604B1 (en) * | 1999-10-15 | 2008-03-18 | Hewlett-Packard Development Company, L.P. | Method for ranking hypertext search results by analysis of hyperlinks from expert documents and keyword scope |
US6351755B1 (en) * | 1999-11-02 | 2002-02-26 | Alta Vista Company | System and method for associating an extensible set of data with documents downloaded by a web crawler |
US6539376B1 (en) * | 1999-11-15 | 2003-03-25 | International Business Machines Corporation | System and method for the automatic mining of new relationships |
US6883135B1 (en) * | 2000-01-28 | 2005-04-19 | Microsoft Corporation | Proxy server using a statistical model |
US6516312B1 (en) * | 2000-04-04 | 2003-02-04 | International Business Machine Corporation | System and method for dynamically associating keywords with domain-specific search engine queries |
US6859800B1 (en) * | 2000-04-26 | 2005-02-22 | Global Information Research And Technologies Llc | System for fulfilling an information need |
US6772160B2 (en) * | 2000-06-08 | 2004-08-03 | Ingenuity Systems, Inc. | Techniques for facilitating information acquisition and storage |
JP3573688B2 (en) * | 2000-06-28 | 2004-10-06 | 松下電器産業株式会社 | Similar document search device and related keyword extraction device |
DE10033416C1 (en) * | 2000-07-08 | 2001-06-21 | Mtu Friedrichshafen Gmbh | Cover plate for crankcase of IC engine extends over complete crankcase and forms modular unit with heat exchangers, pumps, and filters |
US6678692B1 (en) * | 2000-07-10 | 2004-01-13 | Northrop Grumman Corporation | Hierarchy statistical analysis system and method |
US7072847B2 (en) * | 2000-08-25 | 2006-07-04 | Jonas Ulenas | Method and apparatus for obtaining consumer product preferences through product selection and evaluation |
NO313399B1 (en) * | 2000-09-14 | 2002-09-23 | Fast Search & Transfer Asa | Procedure for searching and analyzing information in computer networks |
US6560600B1 (en) * | 2000-10-25 | 2003-05-06 | Alta Vista Company | Method and apparatus for ranking Web page search results |
US6766316B2 (en) * | 2001-01-18 | 2004-07-20 | Science Applications International Corporation | Method and system of ranking and clustering for document indexing and retrieval |
US6526440B1 (en) * | 2001-01-30 | 2003-02-25 | Google, Inc. | Ranking search results by reranking the results based on local inter-connectivity |
US20040003028A1 (en) * | 2002-05-08 | 2004-01-01 | David Emmett | Automatic display of web content to smaller display devices: improved summarization and navigation |
DE60239742D1 (en) * | 2001-05-10 | 2011-05-26 | Amdocs Software Systems Ltd | SMART INTERNET WEBSITE WITH HIERARCHICAL MENU |
US6947920B2 (en) * | 2001-06-20 | 2005-09-20 | Oracle International Corporation | Method and system for response time optimization of data query rankings and retrieval |
US7039234B2 (en) * | 2001-07-19 | 2006-05-02 | Microsoft Corporation | Electronic ink as a software object |
US6868411B2 (en) * | 2001-08-13 | 2005-03-15 | Xerox Corporation | Fuzzy text categorizer |
US20030046389A1 (en) * | 2001-09-04 | 2003-03-06 | Thieme Laura M. | Method for monitoring a web site's keyword visibility in search engines and directories and resulting traffic from such keyword visibility |
US6970863B2 (en) * | 2001-09-18 | 2005-11-29 | International Business Machines Corporation | Front-end weight factor search criteria |
US20060004732A1 (en) * | 2002-02-26 | 2006-01-05 | Odom Paul S | Search engine methods and systems for generating relevant search results and advertisements |
US7693830B2 (en) * | 2005-08-10 | 2010-04-06 | Google Inc. | Programmable search engine |
US20040006559A1 (en) * | 2002-05-29 | 2004-01-08 | Gange David M. | System, apparatus, and method for user tunable and selectable searching of a database using a weigthted quantized feature vector |
CA2395905A1 (en) * | 2002-07-26 | 2004-01-26 | Teraxion Inc. | Multi-grating tunable chromatic dispersion compensator |
US7599911B2 (en) * | 2002-08-05 | 2009-10-06 | Yahoo! Inc. | Method and apparatus for search ranking using human input and automated ranking |
US7231379B2 (en) * | 2002-11-19 | 2007-06-12 | Noema, Inc. | Navigation in a hierarchical structured transaction processing system |
US7216123B2 (en) * | 2003-03-28 | 2007-05-08 | Board Of Trustees Of The Leland Stanford Junior University | Methods for ranking nodes in large directed graphs |
US7197497B2 (en) * | 2003-04-25 | 2007-03-27 | Overture Services, Inc. | Method and apparatus for machine learning a document relevance function |
US7283997B1 (en) * | 2003-05-14 | 2007-10-16 | Apple Inc. | System and method for ranking the relevance of documents retrieved by a query |
US7836391B2 (en) * | 2003-06-10 | 2010-11-16 | Google Inc. | Document search engine including highlighting of confident results |
US20050060186A1 (en) * | 2003-08-28 | 2005-03-17 | Blowers Paul A. | Prioritized presentation of medical device events |
US7505964B2 (en) * | 2003-09-12 | 2009-03-17 | Google Inc. | Methods and systems for improving a search ranking using related queries |
US7454417B2 (en) * | 2003-09-12 | 2008-11-18 | Google Inc. | Methods and systems for improving a search ranking using population information |
US7346839B2 (en) * | 2003-09-30 | 2008-03-18 | Google Inc. | Information retrieval based on historical data |
US20050071328A1 (en) * | 2003-09-30 | 2005-03-31 | Lawrence Stephen R. | Personalization of web search |
US7634472B2 (en) * | 2003-12-01 | 2009-12-15 | Yahoo! Inc. | Click-through re-ranking of images and other data |
US20060047649A1 (en) * | 2003-12-29 | 2006-03-02 | Ping Liang | Internet and computer information retrieval and mining with intelligent conceptual filtering, visualization and automation |
US7395260B2 (en) * | 2004-08-04 | 2008-07-01 | International Business Machines Corporation | Method for providing graphical representations of search results in multiple related histograms |
US7634461B2 (en) * | 2004-08-04 | 2009-12-15 | International Business Machines Corporation | System and method for enhancing keyword relevance by user's interest on the search result documents |
US20060036598A1 (en) * | 2004-08-09 | 2006-02-16 | Jie Wu | Computerized method for ranking linked information items in distributed sources |
US20060047643A1 (en) * | 2004-08-31 | 2006-03-02 | Chirag Chaman | Method and system for a personalized search engine |
RU2388050C2 (en) * | 2004-09-16 | 2010-04-27 | Теленор Аса | Method, system and computer software for searching, navigation and ranking documents in personal area network |
US20060064411A1 (en) * | 2004-09-22 | 2006-03-23 | William Gross | Search engine using user intent |
US7644107B2 (en) * | 2004-09-30 | 2010-01-05 | Microsoft Corporation | System and method for batched indexing of network documents |
US7827181B2 (en) * | 2004-09-30 | 2010-11-02 | Microsoft Corporation | Click distance determination |
US7698331B2 (en) * | 2005-01-18 | 2010-04-13 | Yahoo! Inc. | Matching and ranking of sponsored search listings incorporating web search technology and web content |
JP4648455B2 (en) * | 2005-05-06 | 2011-03-09 | エヌエイチエヌ コーポレーション | Personalized search method and personalized search system |
US7599917B2 (en) * | 2005-08-15 | 2009-10-06 | Microsoft Corporation | Ranking search results using biased click distance |
US7653617B2 (en) * | 2005-08-29 | 2010-01-26 | Google Inc. | Mobile sitemaps |
US7499919B2 (en) * | 2005-09-21 | 2009-03-03 | Microsoft Corporation | Ranking functions using document usage statistics |
US7716226B2 (en) * | 2005-09-27 | 2010-05-11 | Patentratings, Llc | Method and system for probabilistically quantifying and visualizing relevance between two or more citationally or contextually related data objects |
US7689531B1 (en) * | 2005-09-28 | 2010-03-30 | Trend Micro Incorporated | Automatic charset detection using support vector machines with charset grouping |
US7689559B2 (en) * | 2006-02-08 | 2010-03-30 | Telenor Asa | Document similarity scoring and ranking method, device and computer program product |
US20070276812A1 (en) * | 2006-05-23 | 2007-11-29 | Joshua Rosen | Search Result Ranking Based on Usage of Search Listing Collections |
US20080005068A1 (en) * | 2006-06-28 | 2008-01-03 | Microsoft Corporation | Context-based search, retrieval, and awareness |
US20080016053A1 (en) * | 2006-07-14 | 2008-01-17 | Bea Systems, Inc. | Administration Console to Select Rank Factors |
US7685084B2 (en) * | 2007-02-09 | 2010-03-23 | Yahoo! Inc. | Term expansion using associative matching of labeled term pairs |
US7996392B2 (en) * | 2007-06-27 | 2011-08-09 | Oracle International Corporation | Changing ranking algorithms based on customer settings |
US20090006358A1 (en) * | 2007-06-27 | 2009-01-01 | Microsoft Corporation | Search results |
US8122032B2 (en) * | 2007-07-20 | 2012-02-21 | Google Inc. | Identifying and linking similar passages in a digital text corpus |
US8201081B2 (en) * | 2007-09-07 | 2012-06-12 | Google Inc. | Systems and methods for processing inoperative document links |
US8370331B2 (en) * | 2010-07-02 | 2013-02-05 | Business Objects Software Limited | Dynamic visualization of search results on a graphical user interface |
-
2007
- 2007-10-18 US US11/874,579 patent/US20090106221A1/en not_active Abandoned
-
2008
- 2008-10-17 CN CN2008801124165A patent/CN101828185B/en not_active Expired - Fee Related
- 2008-10-17 EP EP08840594A patent/EP2212813A4/en not_active Withdrawn
- 2008-10-17 WO PCT/US2008/011894 patent/WO2009051809A1/en active Application Filing
Patent Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
KR20020015838A (en) * | 2000-08-23 | 2002-03-02 | 전홍건 | Method for re-adjusting ranking of document to use user's profile and entropy |
KR20030082109A (en) * | 2002-04-16 | 2003-10-22 | (주)메타웨이브 | Method and System for Providing Information and Retrieving Index Word using AND Operator |
KR20060116042A (en) * | 2005-05-09 | 2006-11-14 | 엔에이치엔(주) | Personalized search method using cookie information and system for enabling the method |
Non-Patent Citations (1)
Title |
---|
See also references of EP2212813A4 * |
Also Published As
Publication number | Publication date |
---|---|
CN101828185A (en) | 2010-09-08 |
US20090106221A1 (en) | 2009-04-23 |
EP2212813A4 (en) | 2011-02-23 |
CN101828185B (en) | 2012-11-28 |
EP2212813A1 (en) | 2010-08-04 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
AU2009290574B2 (en) | Document length as a static relevance feature for ranking search results | |
WO2009051809A1 (en) | Ranking and providing search results based in part on a number of click-through features | |
Lin et al. | A survey on expert finding techniques | |
Xue et al. | Optimizing web search using web click-through data | |
JP4996300B2 (en) | File system search ranking method and related search engine | |
EP1934823B1 (en) | Click distance determination | |
US20090106223A1 (en) | Enterprise relevancy ranking using a neural network | |
US8239372B2 (en) | Using link structure for suggesting related queries | |
US8099417B2 (en) | Semi-supervised part-of-speech tagging | |
RU2501078C2 (en) | Ranking search results using edit distance and document information | |
US7698294B2 (en) | Content object indexing using domain knowledge | |
US20100262610A1 (en) | Identifying Subject Matter Experts | |
US9275145B2 (en) | Electronic document retrieval system with links to external documents | |
Singh et al. | A comparative study of page ranking algorithms for information retrieval | |
Barrio et al. | Sampling strategies for information extraction over the deep web | |
Murugudu et al. | Efficiently harvesting deep web interfaces based on adaptive learning using two-phase data crawler framework | |
US20050080774A1 (en) | Ranking of business objects for search engines | |
Yan et al. | Research on PageRank and hyperlink-induced topic search in web structure mining | |
An et al. | Assessment for ontology-supported deep web search | |
Ahamed et al. | State of the art process in query processing ranking system | |
Johansson | Using clickstream data as implicit feedback in information retrieval systems | |
Cordeschi et al. | An information guided spidering: a domain specific case study | |
Pathak et al. | ANALYSIS OF VARIOUS LINK ALGORITHMS IN WEB MINING USING LINK STRUCTURE | |
Yu et al. | Study on Page Ranking Algorithms for Search Engine Optimization | |
Srivastava et al. | International Journal of Emerging Technologies in Computational and Applied Sciences (IJETCAS) www. iasir. net |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
WWE | Wipo information: entry into national phase |
Ref document number: 200880112416.5 Country of ref document: CN |
|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 08840594 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
WWE | Wipo information: entry into national phase |
Ref document number: 2008840594 Country of ref document: EP |