WO2010027916A1 - System and method for generating an approximation of a search engine ranking algorithm - Google Patents
System and method for generating an approximation of a search engine ranking algorithm Download PDFInfo
- Publication number
- WO2010027916A1 WO2010027916A1 PCT/US2009/055342 US2009055342W WO2010027916A1 WO 2010027916 A1 WO2010027916 A1 WO 2010027916A1 US 2009055342 W US2009055342 W US 2009055342W WO 2010027916 A1 WO2010027916 A1 WO 2010027916A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- features
- page
- label
- values
- search engine
- Prior art date
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/951—Indexing; Web crawling techniques
Definitions
- This disclosure relates to a system and method for generating an approximation of a search engine ranking algorithm.
- the World Wide Web is a distributed database including literally billions of pages accessible through the Internet. Searching and indexing these pages to produce useful results in response to user queries is constantly a challenge.
- a search engine is typically used to search the WWW.
- FIG. 1 A typical prior art search engine 20 is shown in Fig. 1.
- Pages from the Internet or other source 22 are accessed through the use of a crawler 24.
- Crawler 24 aggregates pages from source 22 to ensure that these pages are searchable.
- the pages retrieved by crawler 24 are stored in a database 36. Thereafter, these pages are indexed by an indexer 26.
- lndexer 26 builds a searchable index of the pages in a database 34. For example, each web page may be broken down into words and respective locations of each word on the page.
- a user 32 sends a search query to a dispatcher 30.
- Dispatcher 30 compiles a list of search nodes in cluster 28 to execute the query and forwards the query to those selected search nodes.
- the search nodes in search node cluster 28 search respective parts of the index 34 and return search results along with a document identifier to dispatcher 30.
- Dispatcher 30 merges the received results to produce a final result set displayed to user 32 sorted by ranking scores based on a ranking function.
- the ranking function is a function of the query itself and the type of page produced.
- Factors that are used for relevance include hundreds of features extracted, collected or identified for each page including: a static relevance score for the page such as link cardinality and page quality, superior parts of the page such as titles, metadata and page headers, authority of the page such as external references and the "level" of the references, the GOOGLE page rank algorithm, and page statistics such as query term frequency in the page, words on a page, global term frequency, term distances within the page, etc.
- search engines have become one of the most popular online activities with billions of searches being performed by users every month. Search engines are also a starting point for consumers for shopping and various day to day purchases and activities. With billions of dollars being spent by consumers online, it has become ever more important for web sites to organize and optimize their web pages in an effort to be more visible and accessible to users of a search engine.
- One embodiment of the invention is a method for approximating a ranking function of a search engine.
- the method comprises receiving training data, the training data including at least a first page, a first label, a second page and a second label; and receiving the first page at a feature extraction processor, the feature extraction processor effective to identify first features in the first page and calculate first values relating to the first features.
- the method further comprises receiving the second page at the feature extraction processor, the feature extraction processor effective to identify second features and calculate second values relating to the second features; receiving the first features, the first values, the first label, the second features, the second values, and the second label at a machine learning processor; and generating a ranking function at the machine learning processor based on first features, the first values, the first label, the second features, the second values, and the second label.
- Another embodiment of the invention is a system for approximating a ranking function of a search engine.
- the system comprises a first database including training data, the training data including at least a first page, a first label, a second page and a second label; and a feature extraction processor connected to the first database, the feature extraction processor effective to receive the first page and to identify first features and calculate first values relating to the first features, the feature extraction processor further effective to receive the second page and to identify second features and calculate second values relating to the second page.
- the system further comprises a machine learning processor connected to the first database and to the feature extraction server, the machine learning processor effective to receive the first features, the first values, the first label, the second features, the second values, and the second label, the machine learning processor further effective to generate a ranking function based on the first features, the first values, the first label, the second features, the second values, and the second label.
- Yet another embodiment of the invention is an approximation of a ranking function for a search engine produced by the process of receiving training data, the training data including at least a first page, a first label, a second page and ⁇ second label; and receiving the first page at a feature extraction processor, the feature extraction processor effective to identify first features in the first page and calculate first values relating to the first features.
- the process further includes receiving the second page at the feature extraction processor, the feature extraction processor effective to identify second features and calculate second values relating to the second features; receiving the first features, the first values, the first label, the second features, the second values, and the second label at a machine learning processor, and generating a ranking function at the machine learning processor based on first features, the first values, the first label, the second features, the second values, and the second label.
- Fig. 1 is a system drawing a search engine in accordance with the prior art.
- FIG. 2 is a system drawing of system for generating an approximation of a ranking algorithm in accordance with an embodiment of the invention.
- FIG. 3 is schematic drawing of a database structure in accordance with an embodiment of the invention.
- Fig. 4 is a flow chart of a process which could be used in accordance with an embodiment of the invention.
- Fig. 5 is a flow chart of a process which could be used in accordance with an embodiment of the invention.
- Fig, 6 is a flow chart of a process which could be used in accordance with an embodiment of the invention.
- search engines When applying a ranking function, search engines receive as input: 1 ) at least one keyword and 2) a plurality of web pages in a result set produced based on the keyword(s). With those inputs, the search engine produces as an output a ranking score for each web page.
- the inventors recognized this phenomenon and produced a system and algorithm to reverse engineer the function performed by search engines to produce that output. Stated another way. search engines perform the following ranking function to generate a ranking score for each page in a result set:
- the input is the search query in the form of keyword(s) and extracted features of the pages in the result set.
- training data may be sent to a machine learning system. Generating such training data is perhaps the most difficult and labor intensive part of any machine learning system.
- prior art techniques for generating training data include the use of teams ofhumans subjectively viewing selected portions of available data such as keywords and result sets. Even if collection of data may be automated, in the prior art. labeling of the data is performed manually. Such labeling techniques are often inaccurate as they are subject to human judgment oFa complex system such as a search engine. A human being typically cannot judge by intuition whether he has collected all kinds of different search results to ensure that the training data is diverse and it is generally not possible to manually track or generate a diverse set of data. A diverse training set is desired for a machine learning algorithm to work well, Moreover, human labeling in not accurate because it is generally not possible to judge a label value by intuition.
- System 80 includes a training data generator server 60.
- Training data generator server or processor 60 sends keywords 62 over a network 64 (such as the Internet) to a search engine server 66.
- Keywords 62 could be virtually any set of keywords that, when input to a search engine, yield web pages in a result set, It is desirable to generate a number of different sets of keywords. Many techniques could be used to generate such sets.
- Keywords 62 are sent by search engine server 66 to a search engine ' index 68.
- Search engine index 68 outputs web pages 70 that are responsive to a search query including keywords 62.
- Search engine server 66 receives web pages 70 and orders or ranks web pages 70 based on an unknown ranking algorithm to produce ranked web pages 76.
- Ranked web pages 76 are sent over network 64 and fed to training data generaior server 6D.
- Training data generator server 60 stores ranked web pages 76 and labels 82 for those pages in a training data storage 84, A label 82 is associated with each ranked web page 76 corresponding to the rank of the ranked web page 76 based on keyword 62.
- the inventors have determined that a linear distribution of the ranking scores is a good representaiion of those scores Consequently, if L ranked web pages 76 are considered, the highest ranked web page is given a label L, the second highest is given a label L-I , etc.
- a feature extraction engine 72 receives web pages 76 and labels 82 from training data storage 84, and extracts values for defined features in ranked web pages 76. Search engines generate a ranking score and rank pages based on values of certain features. Those features include, for example, features used in the GOOGLE page rank algorithm (such as the links pointing to the page and links in the page pointing somewhere else), the size of the web page, the number of matches between the web page and keywords 62. etc.
- the features may be derived from the content and structure of HTML documents.
- the kinds of features extracted may relate to: keyword frequency in Title tag of HTML documents, keyword frequency in metntags of HTML documents, keyword frequency in the body of HTML documents, keyword frequency in anchor text of in-links to a HTML document, number of back links of a HTML document, or distribution of back links of HTML documents.
- f5 and f6 would have vulues corresponding to those respective features. For example, if fl corresponds to the number of keyword matches, and if page 1 OO has 10 words that match keywords 62, fl may have a value of ] 0.
- training data structure 1 10 which may be stored in training data storage 84.
- training data structure 110 may include a label column 1 14 and a web page column 118.
- Label column 114 includes labels 1 16 for ranked web pages 76 (Fig. 2).
- the web pages themselves may be stored in web page column 1 18.
- the contents of training data structure 1 10 may be used as training data in machine learning server 74 (discussed below).
- Fig. 4 there is shown a flow chart of a process which could be used to implement a process in accordance with an embodiment of the invention, The process could be used with, for example, system 80 described with respect to Fig. 2.
- a search engine queries a search engine index using the keyword to produce a result set including web pages, or the process uses the keywords as input to produce an output.
- the search engine ranks the web pages, or the process ranks the outputs.
- the search engine or process forwards the inputs or keywords and ranked web pages or output to a training data server or processor.
- the training data server or processor assigns a label to each page or output based on the rank.
- the labels and pages or outputs are used as training data.
- a flow chart of a process which could be used in accordance with an embodiment of the invention.
- the process shown in Fig. 5 could be used with, for example, system 80 described above with reference to Fig. 2.
- a web page is received by a feature extraction server.
- features in the page are identified.
- values are calculated for each feature.
- a page vector is generated including the values for each feature.
- feature extraction server 72 stores page vectors 78 jn a feature storage 86.
- a machine learning server or processor 74 may receive page vectors 78 from feature storage 86 and labels 82 from training data storage 84, and use page vectors 78 and labels 86 as training data.
- Machine learning server 74 applies a machine learning algorithm to page vectors 78 and labels 82 to produce an approximated ranking function 88.
- Machine learning server 74 may use any known machine learning techniques on training data to produce an approximated ranking function 88 for a search engine of interest. Although a specific algorithm is disclosed to produce training data, any training data may be used. More specifically, as described above, machine learning server 74 receives page vectors 78 stored in feature storage 86 and labels 82 stored in training data storage 84. Machine learning server 74 then applies a known machine learning algorithm using, for example, boosted decision trees, support vector machines, bayesian learners, any other algorithm, or a combination of these techniques, to calculate an approximated ranking function 88 which is an approximation of the ranking algorithm used by search engine server 66 to score and rank web pages 70.
- a known machine learning algorithm using, for example, boosted decision trees, support vector machines, bayesian learners, any other algorithm, or a combination of these techniques, to calculate an approximated ranking function 88 which is an approximation of the ranking algorithm used by search engine server 66 to score and rank web pages 70.
- Machine learning server 74 may continually process page vector 78 and labels 82 using many different machine learning algorithms to calculate approximated ranking function 88.
- machine learning server 74 may calculate a set of three different possible approximated ranking functions, and in one particular function a combination of values v results in ranking scores R(P) for pages matching labels 82 of ranked web pages 76 80% of the time. That particular function may be assigned as the approximated ranking function 88.
- Approximated ranking function 88 need not be the same as the actual ranking algorithm used by search engine server 66 to be of significant value.
- the GOOGLE search engine may assign different weights to different features than those assigned by machine learning server 74.
- machine learning server 74 may identify new patterns and/or useful features not currently appreciated by known search engines.
- approximated ranking function 88 yields a valuable result.
- a machine learning server or processor receives a page vector including features and values.
- the machine learning server receives labels for each of the pages or outputs indicating how well each page corresponds to a particular keyword or input.
- a query is made as to whether there axe more web pages or outputs to process. If the answer to query S44 is yes, control branches back to step S40. If the answer to query S44 is no, control branches to step S46.
- the machine learning server uses a machine learning algorithm to generate an approximated ranking function based on the features, values and labels.
- Ranking function 88 may be used to evaluate how well a particular web page would rank in a result set based on a keyword in a search engine of interest.
- a system in accordance with that described above can be used to collect training data on any search engine. Moreover, the system can adapt automatically to changes in ranking functions of existing search engines and produce new training data accordingly. Prior art systems are significantly limited in that subjective, expensive human capita) is used to analyze only samples of available data. A system in accordance with the invention could analyze one page or thousands of pages easily and efficiently.
- the system and process described above is more accurate than human labeling because, in part, results of an unknown process such as search engine ranking are used.
- results of an unknown process such as search engine ranking are used.
- the system is automated, it is possible to easily collect large amounts of training data without manual intervention.
- the use of automated labeling further eliminates the need for manual ' intervention.
- Learning algorithms produced in accordance with the invention are change resistant. This is because training data is based on search results. If any search engine changes its ranking algorithm the results will change and the training data will change.
- Prior art systems based on intuition and prior knowledge of humans cannot adapt as easily.
- the system works with known and to be developed search engines and can easily be applied to specific sites such as, for example, TRAVELOCITY.COM.
Abstract
A system and method for generating a search ranking score for a web page The system comprises a training data processor to receive training data including at least a first page, a first label, a second page and a second label The system further comprises a feature extraction processor effective to receive the first page, identify first features in the first page, calculate first values and to receive the second page, identify second features, calculate second values A machine learning processor is effective to receive the first features, the first values, the first label, the second features, the second values, and the second label and generate a ranking function for a search engine A receiving processor receives a web page and a ranking processor is effective apply the ranking function to the web page to generate a score.
Description
SYSTEM AND METHOD FOR GENERATING AN APPROXIMATION OF A SEARCH ENGINE RANKING ALGORITHM
BACKGROUND OF THE INVENTION
[0001] This application claims priority to U.S. Patent application serial number 61/093,586 entitled "Techniques for Automated Search Rank Function, Approximation, Rank Improvement Recommendations and Predictions", filed September 2, 2008, the entirety of which is hereby incorporated by reference.
Field of the Invention
[0002] This disclosure relates to a system and method for generating an approximation of a search engine ranking algorithm.
Description of the Related Art
|0003] Referring to Fig. 1. the World Wide Web ("WWW") is a distributed database including literally billions of pages accessible through the Internet. Searching and indexing these pages to produce useful results in response to user queries is constantly a challenge. A search engine is typically used to search the WWW.
[0004] A typical prior art search engine 20 is shown in Fig. 1. Pages from the Internet or other source 22 are accessed through the use of a crawler 24. Crawler 24 aggregates pages from source 22 to ensure that these pages are searchable. Many algorithms exist for crawlers and in most cases these crawlers follow links in known hypertext documents to obtain other documents. The pages retrieved by crawler 24 are stored in a database 36. Thereafter, these pages are indexed by an indexer 26. lndexer 26 builds a searchable index of the pages in a database 34. For example, each web page may be broken down into words and respective locations of each word on the page. The paces are then indexed by the words and their respective locations,
[0005] In use, a user 32 sends a search query to a dispatcher 30. Dispatcher 30 compiles a list of search nodes in cluster 28 to execute the query and forwards the query to those selected search nodes. The search nodes in search node cluster 28 search respective parts of the index 34 and return search results along with a document identifier to dispatcher 30. Dispatcher 30 merges the received results to produce a final result set displayed to user 32 sorted by ranking scores based on a ranking function.
[0006] The ranking function is a function of the query itself and the type of page produced.
Factors that are used for relevance include hundreds of features extracted, collected or identified for each page including: a static relevance score for the page such as link cardinality and page quality, superior parts of the page such as titles, metadata and page headers, authority of the page such as external references and the "level" of the references, the GOOGLE page rank algorithm, and page statistics such as query term frequency in the page, words on a page, global term frequency, term distances within the page, etc.
[0007] The use of search engines has become one of the most popular online activities with billions of searches being performed by users every month. Search engines are also a starting point for consumers for shopping and various day to day purchases and activities. With billions of dollars being spent by consumers online, it has become ever more important for web sites to organize and optimize their web pages in an effort to be more visible and accessible to users of a search engine.
|0008] As discussed above, for each web page, hundreds of features are extracted and a ranking function is applied to those features to produce a ranking score. A merchant with a web page would like his page to be ranked higher in a result set based on relevant search keywords
compared with web pages of his competitor for the same keywords. For example, for a merchant selling telephones, that merchant would like his web page to acquire a higher ranking score, and appear higher in a result set produced by a search engine, based on the keyword query "telephone" than the ranking scores of web sites of his competitors for the same keyword. There are some prior art solutions available to guess the ranking algorithm used by a search engine and to provide recommendations about improvements that can be made to web pages so that the ranking score for a web page relating to particular keywords may improve. However, most of these systems use manual, human judgment and historical knowledge about search engines. Humans must be trained to perform this analysis. The basis for these judgments are mostly guesses or arrived at by tnal and error. Consequently, most prior art solutions are inaccurate, time consuming, and require expensive human capital, Moreover, these solutions are available only for specific search engines and are not immune to changes in search or ranking algorithms used by known search engines nor do they have the ability to adapt to new search engines.
SUMMARY OF THE INVENTION
[0009] One embodiment of the invention is a method for approximating a ranking function of a search engine. The method comprises receiving training data, the training data including at least a first page, a first label, a second page and a second label; and receiving the first page at a feature extraction processor, the feature extraction processor effective to identify first features in the first page and calculate first values relating to the first features. The method further comprises receiving the second page at the feature extraction processor, the feature extraction processor effective to identify second features and calculate second values relating to the second features; receiving the first features, the first values, the first label, the second features, the second values, and the second label at a machine learning processor; and generating a ranking function at the
machine learning processor based on first features, the first values, the first label, the second features, the second values, and the second label.
[0010] Another embodiment of the invention is a system for approximating a ranking function of a search engine. The system comprises a first database including training data, the training data including at least a first page, a first label, a second page and a second label; and a feature extraction processor connected to the first database, the feature extraction processor effective to receive the first page and to identify first features and calculate first values relating to the first features, the feature extraction processor further effective to receive the second page and to identify second features and calculate second values relating to the second page. The system further comprises a machine learning processor connected to the first database and to the feature extraction server, the machine learning processor effective to receive the first features, the first values, the first label, the second features, the second values, and the second label, the machine learning processor further effective to generate a ranking function based on the first features, the first values, the first label, the second features, the second values, and the second label.
[0011] Yet another embodiment of the invention is an approximation of a ranking function for a search engine produced by the process of receiving training data, the training data including at least a first page, a first label, a second page and ε second label; and receiving the first page at a feature extraction processor, the feature extraction processor effective to identify first features in the first page and calculate first values relating to the first features. The process further includes receiving the second page at the feature extraction processor, the feature extraction processor effective to identify second features and calculate second values relating to the second features; receiving the first features, the first values, the first label, the second features, the second values, and the second label at a machine learning processor, and generating a ranking function at the
machine learning processor based on first features, the first values, the first label, the second features, the second values, and the second label.
BRIEF DESCRIPTION OF THE DRAWINGS
{0012] The drawings constitute a part of the specification and include exemplary embodiments of the present invention and illustrate various objects and features thereof.
[0013| Fig. 1 is a system drawing a search engine in accordance with the prior art.
[0014] Fig. 2 is a system drawing of system for generating an approximation of a ranking algorithm in accordance with an embodiment of the invention.
[0015] Fig. 3 is schematic drawing of a database structure in accordance with an embodiment of the invention.
[0016] Fig. 4 is a flow chart of a process which could be used in accordance with an embodiment of the invention.
(0037] Fig. 5 is a flow chart of a process which could be used in accordance with an embodiment of the invention.
[0018] Fig, 6 is a flow chart of a process which could be used in accordance with an embodiment of the invention.
DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0019] Various embodiments of the invention are described hereinafter with reference to the figures. Elements of like structures or function are represented with like reference numerals throughout the figures. The figures are only intended to facilitate the description of the invention
or as a limitation on the scope of the invention. In addition, an aspect described in conjunction with a particular embodiment of the invention is not necessarily limited to that embodiment and can be practiced in conjunction with any other embodiments of the invention.
[0020] When applying a ranking function, search engines receive as input: 1 ) at least one keyword and 2) a plurality of web pages in a result set produced based on the keyword(s). With those inputs, the search engine produces as an output a ranking score for each web page. The inventors recognized this phenomenon and produced a system and algorithm to reverse engineer the function performed by search engines to produce that output. Stated another way. search engines perform the following ranking function to generate a ranking score for each page in a result set:
[0021] ranking score = F(input)
[0022] where the input is the search query in the form of keyword(s) and extracted features of the pages in the result set.
[0023] In order to approximate the ranking function, training data may be sent to a machine learning system. Generating such training data is perhaps the most difficult and labor intensive part of any machine learning system. As discussed above, prior art techniques for generating training data include the use of teams ofhumans subjectively viewing selected portions of available data such as keywords and result sets. Even if collection of data may be automated, in the prior art. labeling of the data is performed manually. Such labeling techniques are often inaccurate as they are subject to human judgment oFa complex system such as a search engine. A human being typically cannot judge by intuition whether he has collected all kinds of different search results to ensure that the training data is diverse and it is generally not possible to manually
track or generate a diverse set of data. A diverse training set is desired for a machine learning algorithm to work well, Moreover, human labeling in not accurate because it is generally not possible to judge a label value by intuition.
[0024] Referring to Fig. 2, there is shown a system 80 in accordance with an embodiment of the invention. System 80 includes a training data generator server 60. Training data generator server or processor 60 sends keywords 62 over a network 64 (such as the Internet) to a search engine server 66. Keywords 62 could be virtually any set of keywords that, when input to a search engine, yield web pages in a result set, It is desirable to generate a number of different sets of keywords. Many techniques could be used to generate such sets. For example, keyword tools provided by search engines such as the MSN Keyword tool, or the GOOGLE ADWORDs tool could be used, third party tools which monitor and collect keywords based on popularity usage and other metrics may be used, or statistical analysis may be used to determine important keywords from web pages and web logs. For example, by collecting the frequency distribution of keywords from web pages and web logs, it may be possible to identify important keywords from pages. Keywords 62 are sent by search engine server 66 to a search engine'index 68.
(0025] Search engine index 68 outputs web pages 70 that are responsive to a search query including keywords 62. Search engine server 66 receives web pages 70 and orders or ranks web pages 70 based on an unknown ranking algorithm to produce ranked web pages 76.
[0026) Ranked web pages 76 are sent over network 64 and fed to training data generaior server 6D. Training data generator server 60 stores ranked web pages 76 and labels 82 for those pages in a training data storage 84, A label 82 is associated with each ranked web page 76 corresponding to the rank of the ranked web page 76 based on keyword 62. The inventors have
determined that a linear distribution of the ranking scores is a good representaiion of those scores Consequently, if L ranked web pages 76 are considered, the highest ranked web page is given a label L, the second highest is given a label L-I , etc.
[0027J A feature extraction engine 72 receives web pages 76 and labels 82 from training data storage 84, and extracts values for defined features in ranked web pages 76. Search engines generate a ranking score and rank pages based on values of certain features. Those features include, for example, features used in the GOOGLE page rank algorithm (such as the links pointing to the page and links in the page pointing somewhere else), the size of the web page, the number of matches between the web page and keywords 62. etc
[00281 The features may be derived from the content and structure of HTML documents.
For example the kinds of features extracted may relate to: keyword frequency in Title tag of HTML documents, keyword frequency in metntags of HTML documents, keyword frequency in the body of HTML documents, keyword frequency in anchor text of in-links to a HTML document, number of back links of a HTML document, or distribution of back links of HTML documents.
|0029] If n features are extracted from ranked web pages 76, then each page P may be represented as a page vector 78 P = {H P, f2P....fnP} where fnP is the nth feature of page P.
|0030] For example, if 100 ranked web pages 76 arc considered, and 6 features are extracted, feature extraction server 72 will produce 100 page vectors 78, labeled 100 to 1, where page 100 would have the form P( I OO)= {fl 100. Ql OO, Bl OO, f4100, f5100, fδl OO} where fL G, 13, f4. f5 and f6 would have vulues corresponding to those respective features. For example, if fl
corresponds to the number of keyword matches, and if page 1 OO has 10 words that match keywords 62, fl may have a value of ] 0.
[0031] Referring to Fig. 3, there is shown an example of a training data structure 1 10 which may be stored in training data storage 84. As shown, for a keyword 1 12 ("telephone" is shown) training data structure 110 may include a label column 1 14 and a web page column 118. Label column 114 includes labels 1 16 for ranked web pages 76 (Fig. 2). The web pages themselves may be stored in web page column 1 18. The contents of training data structure 1 10 may be used as training data in machine learning server 74 (discussed below).
[0032] Referring to Fig. 4, there is shown a flow chart of a process which could be used to implement a process in accordance with an embodiment of the invention, The process could be used with, for example, system 80 described with respect to Fig. 2. As shown at step S2, at least one input or keyword is sent to a search engine or any other system implementing a process. At step S4, the search engine queries a search engine index using the keyword to produce a result set including web pages, or the process uses the keywords as input to produce an output. At step S6, the search engine ranks the web pages, or the process ranks the outputs. At step S8. the search engine or process forwards the inputs or keywords and ranked web pages or output to a training data server or processor. At step Sl O, the training data server or processor assigns a label to each page or output based on the rank. At step S 12, the labels and pages or outputs are used as training data.
[0033J Referring to Fig. 5, there is shown a flow chart of a process which could be used in accordance with an embodiment of the invention. The process shown in Fig. 5 could be used with, for example, system 80 described above with reference to Fig. 2. As shown, at step S22, a web
page is received by a feature extraction server. At step S24, features in the page are identified. At step S26, values are calculated for each feature. At step S28, a page vector is generated including the values for each feature.
[0034] Referring again to Fig. 2, feature extraction server 72 stores page vectors 78 jn a feature storage 86. A machine learning server or processor 74 may receive page vectors 78 from feature storage 86 and labels 82 from training data storage 84, and use page vectors 78 and labels 86 as training data. Machine learning server 74 applies a machine learning algorithm to page vectors 78 and labels 82 to produce an approximated ranking function 88.
[0035] Machine learning server 74 may use any known machine learning techniques on training data to produce an approximated ranking function 88 for a search engine of interest. Although a specific algorithm is disclosed to produce training data, any training data may be used. More specifically, as described above, machine learning server 74 receives page vectors 78 stored in feature storage 86 and labels 82 stored in training data storage 84. Machine learning server 74 then applies a known machine learning algorithm using, for example, boosted decision trees, support vector machines, bayesian learners, any other algorithm, or a combination of these techniques, to calculate an approximated ranking function 88 which is an approximation of the ranking algorithm used by search engine server 66 to score and rank web pages 70. Approximated ranking function 88 for a page P will have the form R(P) = vl fl + v2f2 ÷ , ...vnfn, where vnfn is a weight assigned to the nth feature of a page of interest. For example, vl could have a value of 0.5 and v2 could have a value of 0.25 indicating that in the approximated ranking function 88. feature fl is given twice the amount of weight as feature G. Machine learning server 74 may continually process page vector 78 and labels 82 using many different machine learning algorithms to calculate approximated ranking function 88. For example, machine learning server 74 may calculate a set of
three different possible approximated ranking functions, and in one particular function a combination of values v results in ranking scores R(P) for pages matching labels 82 of ranked web pages 76 80% of the time. That particular function may be assigned as the approximated ranking function 88.
[0036] Approximated ranking function 88 need not be the same as the actual ranking algorithm used by search engine server 66 to be of significant value. For example, the GOOGLE search engine may assign different weights to different features than those assigned by machine learning server 74. In fact, machine learning server 74 may identify new patterns and/or useful features not currently appreciated by known search engines. However, as the inputs to machine learning server 74 (e.g. page vectors 78 and labels 82) are the outputs of a search engine of interest, approximated ranking function 88 yields a valuable result.
[0037] Referring to Fig. 6, there is shown a flow chart of a process which could be used in accordance with the invention. The process shown in Fig. 6 could be used with, for example, system 80 described above with reference to Fig. 2, As shown, at step S40, for each web page or output, a machine learning server or processor receives a page vector including features and values. At step S42. the machine learning server receives labels for each of the pages or outputs indicating how well each page corresponds to a particular keyword or input. At step S44, a query is made as to whether there axe more web pages or outputs to process. If the answer to query S44 is yes, control branches back to step S40. If the answer to query S44 is no, control branches to step S46. At step S46, the machine learning server uses a machine learning algorithm to generate an approximated ranking function based on the features, values and labels. Ranking function 88 may be used to evaluate how well a particular web page would rank in a result set based on a keyword in a search engine of interest.
[0038] Clearly, although different servers are shown for various elements such as the training data server, and the feature extraction server, those servers could be combined in a single processor, housing or location.
[0039] A system in accordance with that described above can be used to collect training data on any search engine. Moreover, the system can adapt automatically to changes in ranking functions of existing search engines and produce new training data accordingly. Prior art systems are significantly limited in that subjective, expensive human capita) is used to analyze only samples of available data. A system in accordance with the invention could analyze one page or thousands of pages easily and efficiently.
[0040] As can be discerned, the system and process described above is more accurate than human labeling because, in part, results of an unknown process such as search engine ranking are used. As the system is automated, it is possible to easily collect large amounts of training data without manual intervention. The use of automated labeling further eliminates the need for manual ' intervention. Learning algorithms produced in accordance with the invention are change resistant. This is because training data is based on search results. If any search engine changes its ranking algorithm the results will change and the training data will change. Prior art systems based on intuition and prior knowledge of humans cannot adapt as easily. The system works with known and to be developed search engines and can easily be applied to specific sites such as, for example, TRAVELOCITY.COM.
[0041 ] The invention has been described with reference to an embodiment that illustrates the principles of the invention and is not meant to limit the scope of trie invention. Modifications and alterations may occur to others upon reading and understanding the preceding detailed
description. It is intended that the scope of the invention be construed as including all modifications and alterations that may occur to others upon reading and understanding the preceding detailed description insofar as they come within the scope of the following claims or equivalents thereof. Various changes may be made without departing from the spirit and scope of the invention.
I J •
Claims
1. A method for approximating a ranking function of a search engine, the method comprising: receiving (raining data, the training data including at least a first page, a first label, a second page and a second label; receiving the first page at a feature extraction processor, the feature extraction processor effective to identify first features in the first page and calculate first values relating to the first features; receiving the second page at the feature extraction processor, the feature extraction processor effective to identify second features and calculate second values relating to the second features; receiving the first features, the first values, the first label, the second features, the second values, and the second label at a machine learning processor; and generating a ranking function at the machine learning processor based on first features, the first values, the first label, the second features, the second values, and the second label.
2. The method as recited in claim 1. wherein: the first label is related to a first rank of the first page based on a keyword input to the search engine; and the second label is related to a second rank of the second page based on the keyword input to the search engine.
3. The method as recited in claim 1. wherein the first and second pages are received from the search engine in response to an input keyword; and the ranking function is an approximation of a ranking performed by the search cnyine.
4. The method as recited in claim 1, wherein the generating the ranking function is performed using at least one of a boosted decision tree, a support vector machine, and a bayesian learner.
5. The method as recited in claim 4. wherein the generating the ranking function is performed using a combination of the boosted decision tree, the support vector machine, and the bayesian learner.
6. The method as recited in claim 1. wherein the ranking function includes respective weights for the first features and for the second features.
7. The method as recited in claim 6, wherein: a value of the weights is a first value; a second value is assigned by the search engine; and I he first and second values are different.
8. A system for approximating a rankini: function of a search engine, the system comprising: a first database including training data, the training data including at least a first page, a first label, a second page and a second label; a feature extraction processor connected to the first database, the feature extraction processor effective to receive the first page and to identify first features and calculate first values relating to the first features, the feature extraction processor further effective to receive the second page and to identify second features and calculate second values relating to the second page; and a machine learning processor connected to the first database and to the feature extraction server, the machine learning processor effective to receive the first features, the first values, the first label, the second features, the second values, and the second label, the machine learning processor further effective to generate a ranking function based on the First features, the first values, the first label, the second features, the second values, and the second label.
9. The system as recited in claim 8. wherein: the first label is related to a first rank of the first page based on a keyword input to the search engine: and the second label is related to a second rank of lhc second page based on the keyword input to the search engine.
10. The system as recited in claim 8. wherein the first and second pages are received from the search engine in response to an input keyword, and the ranking function is an approximation of a ranking performed by the search engine.
1 1. The system as recited in claim 8. wherein the generaiing the ranking function is performed using at least one of a boosted decision tree, a support vector machine, and a baycsian learner.
12. The system as recited in claim 1 1 , wherein the generating the ranking function is performed using a combination of the boosted decision tree, the support vector machine, and the bayesian learner.
13. The system as recited in claim 8. wherein the ranking function includes respective weights for the first features and for the second features.
14. The system us recited in claim 13, wherein: a value of the weight's is a first value; a second value is assigned by the search engine; and the first and second values are different.
15. An approximation of a ranking function for a search engine produced by the process of: receiving training data, the training data including at least a first page, a first label, a second page and a second label: receiving the first page at a feature extraction processor, the feature extraction processor effective to identify first features in the first page and calculate first values relating to the first features; receiving the second page at the feature extraction processor, the feature extraction processor effective to identify second features and calculate second values relating to the second features; receiving the first features, the first values, the first label, the second features, the second values, and the second label at a machine learning processor; and generating a ranking function at the machine learning processor based on first features, the first values, the first label, the second features, the second values, and the second label.
16. The ranking function as recited in claim 1 5, wherein: the first label is related to a first rank of the first page based on a keyword input to the search engine; and the second label is related to a second rank of the second page based on (he keyword input to the search engine.
17. The ranking function as recited in claim 15, wherein the first and second pages arc received from the search engine in response to an input keyword: and the ranking function is an approximation of a ranking performed by the search engine.
18. The ranking function as recited in claim 15, wherein the generating the ranking function is performed using at least one of a boosted decision tree, a support vector machine, and a bavesian learner.
19. The ranking function as recited in claim 18, wherein the generating the ranking function is performed using a combination of the boosted decision tree, the support vector machine, and the bayesian learner.
20. The ranking function as recited in claim 15. wherein the ranking function includes respective weights for the first features and for the second features.
21. The ranking function as recited in claim 20. wherein: a value of the weights is a first value; a second value is assigned by the search engine; and the first and second values are different.
Applications Claiming Priority (8)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US9358608P | 2008-09-02 | 2008-09-02 | |
US61/093,586 | 2008-09-02 | ||
US12/367,646 US8255391B2 (en) | 2008-09-02 | 2009-02-09 | System and method for generating an approximation of a search engine ranking algorithm |
US12/367,656 | 2009-02-09 | ||
US12/367,646 | 2009-02-09 | ||
US12/367,634 | 2009-02-09 | ||
US12/367,656 US20100057719A1 (en) | 2008-09-02 | 2009-02-09 | System And Method For Generating Training Data For Function Approximation Of An Unknown Process Such As A Search Engine Ranking Algorithm |
US12/367,634 US20100057717A1 (en) | 2008-09-02 | 2009-02-09 | System And Method For Generating A Search Ranking Score For A Web Page |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2010027916A1 true WO2010027916A1 (en) | 2010-03-11 |
Family
ID=41726829
Family Applications (3)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/US2009/055342 WO2010027916A1 (en) | 2008-09-02 | 2009-08-28 | System and method for generating an approximation of a search engine ranking algorithm |
PCT/US2009/055355 WO2010027917A1 (en) | 2008-09-02 | 2009-08-28 | System and method for generating training data for function approximation of an unknown process such as a search engine ranking algorithm |
PCT/US2009/055331 WO2010027914A1 (en) | 2008-09-02 | 2009-08-28 | System and method for generating a search ranking score for a web page |
Family Applications After (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/US2009/055355 WO2010027917A1 (en) | 2008-09-02 | 2009-08-28 | System and method for generating training data for function approximation of an unknown process such as a search engine ranking algorithm |
PCT/US2009/055331 WO2010027914A1 (en) | 2008-09-02 | 2009-08-28 | System and method for generating a search ranking score for a web page |
Country Status (2)
Country | Link |
---|---|
US (3) | US20100057719A1 (en) |
WO (3) | WO2010027916A1 (en) |
Families Citing this family (45)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8554638B2 (en) * | 2006-09-29 | 2013-10-08 | Microsoft Corporation | Comparative shopping tool |
US7899763B2 (en) | 2007-06-13 | 2011-03-01 | International Business Machines Corporation | System, method and computer program product for evaluating a storage policy based on simulation |
US8065199B2 (en) | 2009-04-08 | 2011-11-22 | Ebay Inc. | Method, medium, and system for adjusting product ranking scores based on an adjustment factor |
US9846898B2 (en) | 2009-09-30 | 2017-12-19 | Ebay Inc. | Method and system for exposing data used in ranking search results |
US8533191B1 (en) * | 2010-05-27 | 2013-09-10 | Conductor, Inc. | System for generating a keyword ranking report |
US8782037B1 (en) * | 2010-06-20 | 2014-07-15 | Remeztech Ltd. | System and method for mark-up language document rank analysis |
US20120011112A1 (en) * | 2010-07-06 | 2012-01-12 | Yahoo! Inc. | Ranking specialization for a search |
US8818880B1 (en) * | 2010-09-07 | 2014-08-26 | Amazon Technologies, Inc. | Systems and methods for source identification in item sourcing |
US8639686B1 (en) | 2010-09-07 | 2014-01-28 | Amazon Technologies, Inc. | Item identification systems and methods |
US20120066359A1 (en) * | 2010-09-09 | 2012-03-15 | Freeman Erik S | Method and system for evaluating link-hosting webpages |
US8478704B2 (en) | 2010-11-22 | 2013-07-02 | Microsoft Corporation | Decomposable ranking for efficient precomputing that selects preliminary ranking features comprising static ranking features and dynamic atom-isolated components |
US8620907B2 (en) | 2010-11-22 | 2013-12-31 | Microsoft Corporation | Matching funnel for large document index |
US9424351B2 (en) | 2010-11-22 | 2016-08-23 | Microsoft Technology Licensing, Llc | Hybrid-distribution model for search engine indexes |
US8713024B2 (en) | 2010-11-22 | 2014-04-29 | Microsoft Corporation | Efficient forward ranking in a search engine |
US9529908B2 (en) | 2010-11-22 | 2016-12-27 | Microsoft Technology Licensing, Llc | Tiering of posting lists in search engine index |
US8554700B2 (en) * | 2010-12-03 | 2013-10-08 | Microsoft Corporation | Answer model comparison |
US8630992B1 (en) * | 2010-12-07 | 2014-01-14 | Conductor, Inc. | URL rank variability determination |
US8909644B2 (en) | 2011-05-26 | 2014-12-09 | Nice Systems Technologies Uk Limited | Real-time adaptive binning |
US10223451B2 (en) * | 2011-06-14 | 2019-03-05 | International Business Machines Corporation | Ranking search results based upon content creation trends |
US8620840B2 (en) | 2011-07-19 | 2013-12-31 | Nice Systems Technologies Uk Limited | Distributed scalable incrementally updated models in decisioning systems |
US8788436B2 (en) * | 2011-07-27 | 2014-07-22 | Microsoft Corporation | Utilization of features extracted from structured documents to improve search relevance |
US8924318B2 (en) | 2011-09-28 | 2014-12-30 | Nice Systems Technologies Uk Limited | Online asynchronous reinforcement learning from concurrent customer histories |
US8914314B2 (en) | 2011-09-28 | 2014-12-16 | Nice Systems Technologies Uk Limited | Online temporal difference learning from incomplete customer interaction histories |
US8843477B1 (en) * | 2011-10-31 | 2014-09-23 | Google Inc. | Onsite and offsite search ranking results |
CN103530295B (en) * | 2012-07-05 | 2018-12-07 | 腾讯科技(深圳)有限公司 | Webpage pre-head method and device |
US9454732B1 (en) * | 2012-11-21 | 2016-09-27 | Amazon Technologies, Inc. | Adaptive machine learning platform |
US9576053B2 (en) | 2012-12-31 | 2017-02-21 | Charles J. Reed | Method and system for ranking content of objects for search results |
US9141906B2 (en) * | 2013-03-13 | 2015-09-22 | Google Inc. | Scoring concept terms using a deep network |
US20140365453A1 (en) * | 2013-06-06 | 2014-12-11 | Conductor, Inc. | Projecting analytics based on changes in search engine optimization metrics |
CN103530321B (en) * | 2013-09-18 | 2016-09-07 | 上海交通大学 | A kind of ordering system based on machine learning |
US20160283952A1 (en) * | 2013-11-04 | 2016-09-29 | Agingo Corporation | Ranking information providers |
RU2634218C2 (en) | 2014-07-24 | 2017-10-24 | Общество С Ограниченной Ответственностью "Яндекс" | Method for determining sequence of web browsing and server used |
RU2637883C1 (en) * | 2016-06-20 | 2017-12-07 | Общество С Ограниченной Ответственностью "Яндекс" | Method of establishing training object for training machine training algorithm |
US11200242B2 (en) * | 2017-02-21 | 2021-12-14 | International Business Machines Corporation | Medical condition communication management |
US10878058B2 (en) * | 2017-06-16 | 2020-12-29 | T-Mobile Usa, Inc. | Systems and methods for optimizing and simulating webpage ranking and traffic |
US10489511B2 (en) * | 2018-03-01 | 2019-11-26 | Ink Content, Inc. | Content editing using AI-based content modeling |
US20220027400A1 (en) * | 2018-05-21 | 2022-01-27 | State Street Corporation | Techniques for information ranking and retrieval |
US11822588B2 (en) * | 2018-10-24 | 2023-11-21 | International Business Machines Corporation | Supporting passage ranking in question answering (QA) system |
US10353940B1 (en) * | 2018-12-11 | 2019-07-16 | Rina Systems, Llc. | Enhancement of search results |
US11449515B1 (en) * | 2019-06-14 | 2022-09-20 | Grant Michael Russell | Crowd sourced database system |
RU2019128026A (en) | 2019-09-05 | 2021-03-05 | Общество С Ограниченной Ответственностью «Яндекс» | METHOD AND SYSTEM FOR RANKING A SET OF DIGITAL DOCUMENTS |
CN111160001B (en) * | 2019-12-23 | 2022-09-23 | 联想(北京)有限公司 | Data processing method and device |
CN111966946A (en) * | 2020-09-10 | 2020-11-20 | 北京百度网讯科技有限公司 | Method, device, equipment and storage medium for identifying authority value of page |
US11836173B2 (en) * | 2021-05-26 | 2023-12-05 | Banjo Health Inc. | Apparatus and method for generating a schema |
CN115983616B (en) * | 2023-03-21 | 2023-07-14 | 济南丽阳神州智能科技有限公司 | Business process mining method and equipment based on management system |
Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6601075B1 (en) * | 2000-07-27 | 2003-07-29 | International Business Machines Corporation | System and method of ranking and retrieving documents based on authority scores of schemas and documents |
US6865573B1 (en) * | 2001-07-27 | 2005-03-08 | Oracle International Corporation | Data mining application programming interface |
US20060218141A1 (en) * | 2004-11-22 | 2006-09-28 | Truveo, Inc. | Method and apparatus for a ranking engine |
Family Cites Families (19)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7425968B2 (en) * | 2003-06-16 | 2008-09-16 | Gelber Theodore J | System and method for labeling maps |
US7231375B2 (en) * | 2003-10-10 | 2007-06-12 | Microsoft Corporation | Computer aided query to task mapping |
US7617176B2 (en) * | 2004-07-13 | 2009-11-10 | Microsoft Corporation | Query-based snippet clustering for search result grouping |
US20060074905A1 (en) * | 2004-09-17 | 2006-04-06 | Become, Inc. | Systems and methods of retrieving topic specific information |
US7801899B1 (en) * | 2004-10-01 | 2010-09-21 | Google Inc. | Mixing items, such as ad targeting keyword suggestions, from heterogeneous sources |
GB2425195A (en) * | 2005-04-14 | 2006-10-18 | Yosi Heber | Website analysis method |
US7877383B2 (en) * | 2005-04-27 | 2011-01-25 | Microsoft Corporation | Ranking and accessing definitions of terms |
US20070130112A1 (en) * | 2005-06-30 | 2007-06-07 | Intelligentek Corp. | Multimedia conceptual search system and associated search method |
US7831685B2 (en) * | 2005-12-14 | 2010-11-09 | Microsoft Corporation | Automatic detection of online commercial intention |
US7593934B2 (en) * | 2006-07-28 | 2009-09-22 | Microsoft Corporation | Learning a document ranking using a loss function with a rank pair or a query parameter |
US8214272B2 (en) * | 2006-09-05 | 2012-07-03 | Rafael A. Sosa | Web site valuation |
US7617208B2 (en) * | 2006-09-12 | 2009-11-10 | Yahoo! Inc. | User query data mining and related techniques |
US20080077574A1 (en) * | 2006-09-22 | 2008-03-27 | John Nicholas Gross | Topic Based Recommender System & Methods |
US7930302B2 (en) * | 2006-11-22 | 2011-04-19 | Intuit Inc. | Method and system for analyzing user-generated content |
US7925651B2 (en) * | 2007-01-11 | 2011-04-12 | Microsoft Corporation | Ranking items by optimizing ranking cost function |
US7877384B2 (en) * | 2007-03-01 | 2011-01-25 | Microsoft Corporation | Scoring relevance of a document based on image text |
US8407589B2 (en) * | 2007-04-20 | 2013-03-26 | Microsoft Corporation | Grouping writing regions of digital ink |
US7941391B2 (en) * | 2007-05-04 | 2011-05-10 | Microsoft Corporation | Link spam detection using smooth classification function |
US20090326916A1 (en) * | 2008-06-27 | 2009-12-31 | Microsoft Corporation | Unsupervised chinese word segmentation for statistical machine translation |
-
2009
- 2009-02-09 US US12/367,656 patent/US20100057719A1/en not_active Abandoned
- 2009-02-09 US US12/367,646 patent/US8255391B2/en not_active Expired - Fee Related
- 2009-02-09 US US12/367,634 patent/US20100057717A1/en not_active Abandoned
- 2009-08-28 WO PCT/US2009/055342 patent/WO2010027916A1/en active Application Filing
- 2009-08-28 WO PCT/US2009/055355 patent/WO2010027917A1/en active Application Filing
- 2009-08-28 WO PCT/US2009/055331 patent/WO2010027914A1/en active Application Filing
Patent Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6601075B1 (en) * | 2000-07-27 | 2003-07-29 | International Business Machines Corporation | System and method of ranking and retrieving documents based on authority scores of schemas and documents |
US6865573B1 (en) * | 2001-07-27 | 2005-03-08 | Oracle International Corporation | Data mining application programming interface |
US20060218141A1 (en) * | 2004-11-22 | 2006-09-28 | Truveo, Inc. | Method and apparatus for a ranking engine |
Also Published As
Publication number | Publication date |
---|---|
US8255391B2 (en) | 2012-08-28 |
US20100057719A1 (en) | 2010-03-04 |
US20100057718A1 (en) | 2010-03-04 |
US20100057717A1 (en) | 2010-03-04 |
WO2010027914A1 (en) | 2010-03-11 |
WO2010027917A1 (en) | 2010-03-11 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US8255391B2 (en) | System and method for generating an approximation of a search engine ranking algorithm | |
US7647314B2 (en) | System and method for indexing web content using click-through features | |
US6671681B1 (en) | System and technique for suggesting alternate query expressions based on prior user selections and their query strings | |
US7805432B2 (en) | Meta search engine | |
US7475074B2 (en) | Web search system and method thereof | |
US8250061B2 (en) | Learning retrieval functions incorporating query differentiation for information retrieval | |
CN112632359B (en) | Information recommendation method, device, electronic equipment and storage medium | |
US7124127B2 (en) | Search server and method for providing search results | |
Kantorski et al. | Automatic filling of hidden web forms: a survey | |
Sim et al. | Toward agency and ontology for web-based information retrieval | |
US9710543B2 (en) | Automated substitution of terms by compound expressions during indexing of information for computerized search | |
CN104462556A (en) | Method and device for recommending question and answer page related questions | |
US8606793B1 (en) | Business metric score for web pages | |
JP5143182B2 (en) | Machine learning method and machine learning system | |
JP2010146345A (en) | Information filtering apparatus | |
Ahamed et al. | Deduce user search progression with feedback session | |
US8489560B1 (en) | System and method for facilitating the management of keyword/universal resource locator (URL) data | |
Nagappan et al. | Agent based weighted page ranking algorithm for Web content information retrieval | |
Abass et al. | Automatic query expansion for information retrieval: a survey and problem definition | |
Mohan et al. | A survey on web structure mining | |
Kavitha et al. | Learning to rank experts using combination of multiple features of expertise | |
Tiwari et al. | Optimized technique for ranking webpage on search engine optimization | |
Singh | Information Retrieval Based on the Sequential Hypertext Induced Topic Search in Web Pages | |
Arora et al. | Efficient Hybrid Ranking Algorithm for Search Engine | |
JP2010186474A (en) | Retrieval modeling system using association degree dictionary and method |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 09812078 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
32PN | Ep: public notification in the ep bulletin as address of the adressee cannot be established |
Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 21/06/2011) |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 09812078 Country of ref document: EP Kind code of ref document: A1 |