US20100174710A1 - Matching and ranking of sponsored search listings incorporating web search technology and web content - Google Patents

Matching and ranking of sponsored search listings incorporating web search technology and web content Download PDF

Info

Publication number
US20100174710A1
US20100174710A1 US12/705,882 US70588210A US2010174710A1 US 20100174710 A1 US20100174710 A1 US 20100174710A1 US 70588210 A US70588210 A US 70588210A US 2010174710 A1 US2010174710 A1 US 2010174710A1
Authority
US
United States
Prior art keywords
search
listings
sponsored
listing
search listings
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Abandoned
Application number
US12/705,882
Inventor
Charles C. Carson, Jr.
Devika Chawla
James B. Harvey
Matvey Nemenman
Mohit Sabharwal
Marco J. Zagha
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Yahoo Inc
Original Assignee
Yahoo Inc until 2017
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Yahoo Inc until 2017 filed Critical Yahoo Inc until 2017
Priority to US12/705,882 priority Critical patent/US20100174710A1/en
Publication of US20100174710A1 publication Critical patent/US20100174710A1/en
Assigned to YAHOO HOLDINGS, INC. reassignment YAHOO HOLDINGS, INC. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: YAHOO! INC.
Assigned to OATH INC. reassignment OATH INC. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: YAHOO HOLDINGS, INC.
Abandoned legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06QINFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q30/00Commerce
    • G06Q30/02Marketing; Price estimation or determination; Fundraising
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90Details of database functions independent of the retrieved data types
    • G06F16/95Retrieval from the web
    • G06F16/951Indexing; Web crawling techniques
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F7/00Methods or arrangements for processing data by operating upon the order or content of the data handled

Definitions

  • the Internet has transformed into a global marketplace of goods and services, driven in large part by the introduction of the World Wide Web (“the web”), an information system.
  • Computers connected to the Internet may access web pages via a browser program, which has a powerful, simple-to-learn graphical user interface.
  • a web-based search on an Internet search engine a user enters a search term of one or more keywords, which the search engine then uses to generate a listing of web pages that the user may access via a hyperlink.
  • the search engine functionality of the Internet should be focused to facilitate an on-line marketplace that offers searchers quick, easy and relevant search results while providing Internet network information providers with a cost-effective way to target consumers.
  • a consumer utilizing a search engine that facilitates this on-line marketplace finds companies or businesses that offer the products, services, or information that the searcher is seeking.
  • companies providing products, services, or information may bid in an open auction-based ranking environment for ranks on a search result list generated by an Internet search engine.
  • Operators of some auction-based search engines may be losing potential revenue, such as by being unable to show enough sponsored results for certain search terms. In addition, irrelevant results and/or lower effective revenues from the results may occur. Some operators may not be able to tightly control trade-offs between relevance and coverage of the results. Likewise users' needs may not be met for search queries of a commercial intent. Irrelevant results may be shown if sponsored results are shown based only on some of the users' query terms. Moreover, advertisers may experience low traffic with some auction-based search engines. The advertisers may carry a burden of generating bidded terms corresponding to their web pages to try to maximize relevant search traffic to their sites. Missed opportunity to provide high quality matches to specific queries may occur. Likewise, many users may access the advertiser's site but not purchase anything because the result turned out to be irrelevant. As such, advertisers' may pay for click-throughs from a user that had a low probability of an item being purchased.
  • a system generates a search result list in response to a search request from a searcher using a computer network.
  • a first database may be maintained that includes a first plurality of search listings, wherein each search listing may be associated with one or more search terms.
  • a second database may be maintained that includes documents having general web content.
  • a search request is received from the searcher.
  • a first set of search listings is identified from the first database having documents generating a match with the search request and a second set of search listings is identified from the second database having documents generating a match with the search request.
  • a confidence score is determined for each listing from the first set of search listings wherein the confidence score is determined in accordance with a relevance of each listing when compared to the listings of both sets of search listings.
  • the identified search listings from the first set of search listing are ordered, at least in part, in accordance with the confidence score for each search listing.
  • FIG. 1 illustrates a system for providing results to search queries.
  • FIG. 2 shows a system for managing queries and database content.
  • FIG. 3 is a block diagram of an exemplary search pipeline.
  • FIG. 4 is a block diagram illustrating an architecture for presenting a combined set of listings from listings of a variety of servers.
  • FIG. 5 illustrates an exemplary screen shot of a web page that may be returned to the user.
  • FIG. 6 is a block diagram illustrating use of the system for matching query terms and sponsored listings to arbitrary web pages.
  • a system may combine analysis of advertiser sponsored listings, advertiser web site content, world wide web content, query intent, and/or user behavior statistics to deliver relevant matches, such as sponsored matches, and associated properties of a query and result.
  • relevant includes results that generate revenue for a search provider, a depth or broad range of results, and/or results relevant to a searcher.
  • the system may analyze in combination the results of a search query provided by more than one server.
  • a relevance or quality of results from a first search query may be determined in accordance with results of a separate, different search query.
  • FIG. 1 illustrates a system 100 for providing results to search queries.
  • a searcher such as user 110 submits a search query using a web browser to access a front-end server 120 , such as YAHOO or OVERTURE.
  • the query may include one or more terms.
  • the front end server 120 may send the search query to a first backend server 130 such as a sponsored listings server or other type of server.
  • the sponsored listings server may provide results to the search query that include advertisers that pay to be included in the results.
  • the first server 130 may access a sponsored listing match engine 140 .
  • the sponsored listing server 130 may connect with IDP proxies 150 to handle incoming search queries.
  • the IDP proxies 150 may connect with different search clusters such as sponsored listing search clusters 160 and free web crawl search clusters 170 .
  • FIG. 2 shows a system for managing queries and database content.
  • the sponsored listing server 130 may utilize results from the sponsored listing search clusters 160 and the free web crawl search clusters 170 .
  • the sponsored listing search clusters 160 may obtain listings from a sponsored listing database 200 .
  • the sponsored listings database 200 may be used when the sponsored listings server 130 is operated by an entity different from the operator of the free web crawl search clusters 170 .
  • the operator of the sponsored listings server 130 and the free web crawl search clusters 170 may also be the same entity.
  • the system 100 may perform context indexing.
  • the context indexing may include capturing a snapshot of a sponsored listings database 200 for determining listings eligible for an advanced match.
  • Eligible listings include listings whose advertisers have agreed to have their listings displayed using the algorithms of the system 100 .
  • Information from the listings database may be transferred to a sponsored listing crawl cluster 210 such as via a sponsored listing update server 220 .
  • Sponsored listing marshalers 230 may be used to create indexes of the content.
  • An IDPD Tag Daemon 240 may be used to mark a quality of the indexed pages based, for example, on editorial considerations such as removing spam listings, links to listing that do not work and links with a low rate of click-through.
  • the IDPD Tag Daemon 240 may obtain editorial content from an editorial database 250 .
  • Listings may be indexed with associated annotations, crawled content, anchor text such as text associated with inbound links to the target page, or algorithmically generated summaries and keywords.
  • a webmap 300 may be used to provide information to, and receive information from, the sponsored listing crawl cluster 210 , the free web crawl cluster 270 , the editorial database 250 , the sponsored listing marshalers 230 and the freeweb crawl marshalers 280 .
  • the webmap 300 may provide information about hosts and hyperlinks on the web, including anchor text associated with hyperlinks.
  • the marshalers 230 and 280 produce indexes or search databases used by the search clusters 160 and 170 to respond to search queries.
  • a search database of free web content or general web content may be used to determine the quality of results from a search database of sponsored listings.
  • Transfer of sponsored listings to the web crawling/indexing system may be accomplished using a transfer protocol such as XML encoding.
  • Requests and responses from a search provider to a web search system may be accomplished with a protocol such as IDP (INKTOMI Data Protocol) 2.0 and transfers may be accomplished by hyper text transfer protocol (http).
  • An incremental update protocol may be used for the efficient transfer of sponsored listings such as by only transferring those listing that have changed since the last transfer.
  • Sponsored listings may be submitted with a display title, display abstract, display hostname, destination URL, crawl URL, bidded terms, account id, ad group, historical CTR, and the like.
  • the advertiser may submit a set of crawl URLs or root URLs pointing to a tree of web pages, without specifying the titles, abstracts, and bidded terms.
  • an advertiser can specify that there is no content to be crawled for a listing, if the advertiser provides a title, abstract, display hostname, and destination URL for the listing. Multiple listings may use the same crawl URL.
  • An advertiser may submit multiple title and abstract combinations for the same URL and bidded terms, along with an optional specification of how to choose among similar titles and abstracts, for example, by specifying the relative display frequencies or requesting that the system optimize the choice for improved click-through and/or conversion.
  • Multiple listings may be grouped in the sponsored listings database by, for example, account, URL, related group of URLs, and/or bidded terms, in order to improve performance of indexing, retrieval, and/or accounting.
  • a crawl URL shared by multiple listings may be crawled only once. Other amounts of crawls may also be used.
  • Non-indexed properties associated with a listing such as a cost to an advertiser when a user 110 clicks the listing, an account/listing ID, and negative keywords may be included in the listings. Negative keywords include terms chosen by an advertiser such that, if the term is part of a search query, an advertiser's listing will not appear.
  • documents may be annotated with helper words to be indexed, such as by adding stemming variants, synonyms, and/or related terms based on the bidded terms, title, abstract, URL, and/or crawled content for a listing.
  • a new section of the document may be created to add alternate terms to the listing, such as by adding ‘shoe’ and ‘shoes’, ‘running’ and ‘run’ or other variants of terms, and the like.
  • the system 100 may handle search queries from the user 110 by providing results to the search query in accordance, at least in part, with result relevance scoring, such as a confidence score, and/or with ranking criteria.
  • Search results from the sponsored listing search clusters 160 and the free web crawl search clusters 170 , or other database, may be performed, and the results compared to determine results to be displayed to the user.
  • the query may be processed by the proxies 150 .
  • the system 100 may edit the query in ways such as auto-correction for misspelled queries, segmentation of the query into words and phrases, changing case, using concept expansion, and removing less important words, such as based on occurrence frequencies in historical query logs and/or web pages.
  • the system 100 may transform the user query into a preferred form for a back-end query search engine.
  • no results may be returned if the query or transformed query match terms on a list of objectionable terms.
  • no results may be returned if the query contains objectionable terms such as terms associated with videos related to beheadings, adult content, and the like.
  • the objectionable terms may be decided on by an editor of the search engine as a matter of policy or others such as advertisers or automatic adult classifiers.
  • a list of required query phrases may be provided by an editor or advertiser, or generated by an algorithm, and these phrases may be associated with a listing, multiple listings, or all listings from an account or domain. If one or more search terms associated with a sponsored listing contains a required phrase that is not contained in the user query, the listing may be excluded from the set of results for that query.
  • the modified query may be issued to the sponsored listing search clusters 160 and at least one other search cluster, such as the free web crawl search clusters 170 that search for general web content.
  • the free web crawl search clusters 170 may search a full-scale web index or a portion of the full-scale web index chosen in accordance with determined parameters.
  • the full-scale web index may include, for example, 10 billion results while a sample portion of the index may include 200 million results for simplified processing, speed, and/or to increase the relevance to the user 110 of the results.
  • the web content may be sampled by including documents based on properties of the web links such as by using links that are accessed more frequently than others.
  • Information may be provided to the sponsored listing search clusters 160 via a system monitor/executor 310 , such as from a sponsored listings search provider 330 .
  • the sponsored listings search provider may utilize a contract & click data warehouse 340 to maintain data on sponsors and the clicks that occur to the sponsors' websites.
  • the data warehouse 340 may receive information about listings from data quality review module 350 , such as via an account management and reporting module 360 .
  • Information about user clicks to a website may be received via redirect servers 370 , for example, after being processed by click log processing and fraud detection module 380 .
  • the data warehouse 340 may also receive information about the queries issued by users and results shown to users from Sponsored Listing Servers 130 and/or IDP proxies 150 via query and an impression procession module 390 .
  • the computer processing with reference to the servers, databases, and modules as disclosed with reference to FIG. 2 and throughout the application may be performed through a processor-executable code located on a processor-readable medium.
  • the sponsored listing search clusters 160 may provide results in a first way, such as results of advertisers that pay to be included in the sponsored listing database, ranked, for example, by revenue that the results bring to the search provider.
  • the free web crawl search clusters 170 may provide results in another way, such as all possible results or a portion of all possible results whether or not advertiser related, ranked, for example, by popularity. Results of a search of the sponsored listing database may be compared with results of the general web content database, and the results having top ranks from the sponsored listing database may be displayed if the results also occur within top ranks of the results for the general web content database.
  • a relevance or confidence score regarding a relevance of the documents may also be used for optimizing placement on the results page, for example, whether the listing should be shown to the user 110 at a top, side, or bottom of the page.
  • Sponsored results may also be re-ranked or removed as being a duplicate based on a comparison of sponsored listings and web results.
  • the system 100 may also use various scoring parameters for the sponsored listing database query. For example, modified weights may be used for different parts of the search result document. For example, a search term occurring in a title of the document may be considered more important than if the search term only occurred in the body of the document.
  • Relevant results may be determined for each database and the combined results may be sorted by the relevance or confidence score.
  • the relevance score may be based on many factors including a presence of query terms, proximity of the query terms to each other, matching of query terms to document anchor text, a document category, and determined measures of document quality.
  • the sponsored listing database results that occur in the set of sorted top results may be further modified such that results where negative keywords match query occur may be discarded.
  • the query contains terms on a list of designated trademarked terms, any result that does not contain all trademarked terms in a visible portion of result or the bidded terms for the listing may be discarded.
  • the features may be calculated to help determine which results are to be displayed.
  • the features include an actual cost to the advertiser for a user 110 to click the result based, for example, on bid rankings.
  • the features may also be determined in accordance with other factors such as historical click-through and conversion data.
  • Historical click-through data includes information on the number of times a user 110 clicks through to the advertisers' site when the site is provided as a result.
  • Conversion data includes information from the advertiser regarding whether the user 110 performed an action desired by the advertiser, such as making a purchase when the user 110 visiting the advertiser's site.
  • Other features include rank of the result in the full sorted set of top results from the combination of the sponsored listing database and general web content database and a relevance score difference and ratio from top results of the general web content database.
  • Other features of the result include whether the query word/phrase appears in the title, abstract, URL, display host, bidded keyphrase, and/or algorithmically generated keywords or summaries of the result.
  • Individual features may be analyzed alone or in combination to increase a composite confidence that the result is a good match to the query.
  • the features may also be used to create a confidence score for each result, and the sponsored listing server 130 , or other server, may sort the results based on the score such that results that receive a higher score are displayed first on a list of search results displayed to the user.
  • the sponsored listing server 130 may return properties of the input query, such as a categorization of the query, and may return properties of each listing, such as keywords associated with each listing.
  • Functions for relevance scoring and reranking functions may be constructed entirely or in part by machine-learning methods, such as methods to find coefficients of linear formulas, methods to construct decision trees, methods to construct Support Vector Machines, neural networks, Bayesian networks, or other methods.
  • Features, combinations of features, and scoring and reranking functions may be constructed with the aid of historical click and impression data and/or editorially labeled relevance judgments.
  • the results to be displayed may be determined.
  • Real-time reporting of the matching statistics may be provided so that the features determining the confidence score may be modified for a particular search provider.
  • the listing ID may be returned to the search provider along with the confidence score for each web search database result.
  • the listing ID for the web search database results may be used to lookup current title, abstract, display hostname, cost to advertiser for click-through, and listing/account status.
  • the matching of listings may be decoupled based on crawled and indexed content, which may be hours or days old, from the retrieval of the current title, abstract, URL, CPC, and listing/account status.
  • Results determined to be the most relevant in view of any combination of the above-described feature may be displayed. If the advertiser has supplied multiple alternative titles and abstracts for a listing, the choice of title and abstract to be displayed may be made based on relevance to the query, click-through statistics, and/or specification of the desired frequency of display of each title and abstract. Terms in listing in accordance with the query and transformed query may be highlighted for the user. Eligible results may be blended with exact matching and other algorithms based on combination of cost to the advertiser for each click-through, relevance scores and/or preference rules. Flexible policies may be implemented for the tiering or mixing results generated by the system 100 with matches from other systems. Redundant results may be removed based on a combination of account id, the domain name, hostname extracted from the crawl URL, display URL, display hostname, and/or the like.
  • FIG. 3 is a block diagram of an exemplary search pipeline 400 .
  • a raw query is received from the user 110 .
  • the raw query may processed, such as with a sub-phrase match (SPM) match driver (MD) canonical (canon) query module at block 420 and/or MD canonical query module at block 430 , to determine other variations of the terms in the query, remove superfluous terms or endings, such as plural endings or ‘ing’ endings, and determine synonyms for the raw query.
  • the raw query may be sent to IDP proxies at block 440 .
  • the MD canon query may be sent to a title and/or description match server at block 450 .
  • the MD canon query 430 may also be sent to an exact match or orthographic match driver (OMD) match at block 460 .
  • Results of the exact match or OMD match may be sent to a phrase match or broad match server at block 470 , along with results from the SPM MD canon query.
  • the SPM MD canon query results may also be sent to a local match server at block 480 .
  • the results from all the servers 440 , 450 , 470 and 480 may be sent to a server listings module at block 490 . Each listing is analyzed to determine a confidence rating of the listing that the listing is relevant.
  • An exemplary core matching algorithm for the system 100 follows:
  • a “full result set” is a blended set of the top num_candidate results from a sponsored listing database and other databases, ranked by a relevance score. Each result from the sponsored listings database within the full result set may be rescored and reranked by computing a number of “demotion” terms that attempt to capture non-ideal characteristics of a result.
  • Weights of each query word may be set as 1.0 by default, but may be overridden to any value between zero and one. Based on word weights, max_query_score is the sum of weights of each unique query word.
  • edit_distance_demotion (max_query_score ⁇ max)/max_query_score
  • order_distance_demotion 0 for 1-word queries, 1 if no query words appear in section, otherwise computed as 1 ⁇ (ordered_words ⁇ 1)/(nunique_query_words ⁇ 1),
  • the minimum demotion of any section may be used for each of the three demotion types:
  • section_demotion(section_list) edit_distance_demotion_weight*min_of_sections(edit_distance_demotion)+order_distance_demotion_weight*min_of_sections(order_distance_demotion)+absent_words_demotion_weight*min_of_sections(absence_demotion)
  • the “editorial demotion” may be 1 to denote a listing that has not been editorially reviewed, or 0 otherwise. Other notations may be used.
  • the final relevance score may be determined as:
  • sponsored results may be sorted by the bid_cpc associated with each result. Any result with a bid_cpc less than min bid_cpc may be assigned an actual_cpc equal to its bid_cpc. Otherwise, the bottom result may be assigned an actual_cpc of its bid_cpc and each other result is assigned an actual_cpc of the next result's bid_cpc plus cpc_increment, subject to the constraint that actual_cpc may not exceed bid_cpc.
  • Results with a relevance_confidence below min_relevance_confidence may be rejected.
  • Remaining results may be reranked by a ranking_score determined as follows:
  • the ranking_score may be determined as follows:
  • FIG. 4 is a block diagram illustrating an architecture 500 for presenting a combined set of listings from listings of a variety of servers.
  • a user query is received at block 510 .
  • the query is sent to different search servers. Variations of the query may be sent to the different servers. Certainty scores may be attached to each variation of query, as a probability that the variation will return a relevant result.
  • the search servers match listings to the query. A certainty score may be attached to the listings that the listing is a relevant match.
  • the listings are ranked such as in accordance with the certainty scores and other factors such as cost per click (block 550 ), click through rates (block 560 ) and based on editorial considerations (block 570 ).
  • the listing may be presented in accordance with the rankings and other factors. The listing may be shown in order of confidence, such as with the highest confidence rated listing being shown first and/or at a particular area of displayed browser page.
  • FIG. 5 illustrates an exemplary screen shot 600 of a web page that may be returned to the user. If the user 110 enters the terms ‘deboning chicken’ there may be no sponsored results with that term. Since the system 100 compares results from the web search database and the general web content database, a result like ‘Chicken Shears at Shopping.com’ may be discovered. When connecting to the link, the user 110 may observe a result such as ‘Diamond Cut Multi-purpose Kitchen Shears’ that does not include either the terms ‘deboning’ or ‘chicken’ in the title.
  • the system 100 may be customized to address varying needs of different search providers.
  • Sponsored listings may be scored differently than general web documents, for example, to either place more or less emphasis on the sponsored listings.
  • Crawl frequencies for updating the web search database may be changed, such as from weeks to days, depending on a desired freshness of the content in the database.
  • Tunable parameters may be provided for individual matching criteria. Results may be restricted to those where all the query terms are present in determined portions of the results, such as the title or the abstract. Stopwords such as ‘a’, ‘the’ and ‘and’ may be excluded or given less weight. Some query terms may be weighted as being more important than other terms.
  • the system 100 may be tunable; for instance, the query terms may have different weights for different search providers.
  • results may be ranked relative to web content without requiring retrieval of the full web content listings, such as the title and abstracts. Results sets may be cached to reduce latency and lower serving costs.
  • the system 100 may also include other features to help improve relevance and/or revenue such as by using historical click-through data regarding the number of users 110 that accessed an advertiser's site and conversion data regarding the number of users 110 that purchased items from the advertiser's site.
  • the data may be aggregated per URL, domain, or contract for all queries, or for specific queries and query terms.
  • Queries from the user 110 may be modified such as by editorially determining rewrites for specific queries or query terms, or by algorithmically determining rewrites. Certain query terms may be deleted or reduced in weight. Related terms and synonym expansion of the term may be added to the query and weighted accordingly.
  • the system 100 may also expand query terms and add stemming variants, such as by adding or removing ‘s’ and ‘ing’ to the term, and abbreviation variants of the term may be added to the query. Queries may be segmented into phrases for better proximity scoring of the terms. For example, the terms ‘New’ and ‘Mexico’ may be grouped together to get results for the state ‘New Mexico’. Queries written without spaces separating words, such as in Chinese queries, may be segmented into words.
  • Compound words such as in Korean or German queries, may be segmented into component words.
  • Script variants may be allowed such as for Asian languages, such as variants in the Chinese language.
  • a list of terms may be generated to use for hit-highlighting in accordance with rewrites of the term. For example, when the results are displayed, the search terms may be highlighted to aid the user, and variants of the search term may also be highlighted. For the search term ‘S.F. opera’ the variant ‘San Francisco opera’ may also be highlighted when the results are displayed.
  • the system 100 may also modify content of the web page results stored in the web search database and/or the general web content database.
  • Page content may be thinned or increased, based on many factors including page structure, page template extraction, for example, applied to a collection of documents associated with a contract, feed, or site, and click and conversion statistics per the query or query term.
  • Prices may be added to the web page description to avoid irrelevant matches. Synonyms, alternate spellings, stemming variants, abbreviation variants, and related terms may also be added to the web page description.
  • the system 100 may also be configured to accommodate other features. Components requiring development, such as query classification, spelling correction, and query rewriting, may be added to the system 100 .
  • Relevance tuning may be allowed for region and language preferences. Region-specific scoring may be implemented such that query results tied to a particular region are weighed higher for a user 110 from that particular region.
  • Personalization data may be incorporated into relevance scoring, using many factors such as content preferences, user location, user demographics and query history.
  • the search provider may be allowed to control whether results should be weighted towards those that appear more relevant or whether a greater coverage/depth of results is preferred. Editorial judgments, positive or negative, may be incorporated for sponsored listing data based on URLs, domains, or contracts.
  • Sponsored listings feeds may be used to automatically generate bid terms and bid amounts for related listings.
  • the system 100 may be used for implementing exact match to bidded terms in order to lower serving cost and to incorporate relevance ranking of exact match listings based on factors used for inexact match, such as web content.
  • the system 100 may incorporate current advertiser budget in selection and ranking of results. For example, if the advertiser has used up a determined percentage of its budget, a result for that advertiser may not be shown unless the result ranks above a certain rank, even if the result would otherwise have been displayed.
  • Local/geographic queries, domain queries, and product number queries may be scored differently. For example, results that otherwise may not have been displayed, are given extra weight and are shown because a local query has occurred.
  • a queue of listings or query-listing pairs to be editorially reviewed may be automatically generated based on several factors including click data, conversion data, query or query-term frequency, listings display frequency, listing bid amounts, results of previous editorial reviews (if any), and user complaint frequency, and may be aggregated by account, domain, query, or query term.
  • Representative web content may be selected and used for generating relative ranks of web search database content using factors other than the frequency of a selection including, for example, optimizing coverage by language, region, content-type, internet domain, topic, and vocabulary used.
  • the system 100 may be adapted to show offers other than sponsored listings, such as auction listings, job offers, etc., and to evaluate relevance of those offers relative to a content database other than generic web content, such as a database driving a vertical search or directory such as travel, shopping, etc.
  • Other content attributes in relevance evaluation may be incorporated, such as prices, availability, duration of offer, popularity, vendor rating, and product details such as size, color and trim.
  • the system 100 may be used to determine when and where on the page to show links to preferential content, for example, as done with ‘YAHOO! Shortcut’ listings, such that if a query includes the term ‘HONDA CIVIC’, a link to YAHOO! Autos site may be provided.
  • Other features include that advertisers may submit web pages without fully specifying bidded terms, title, abstract. Also, a listing abstract may be modified based on matches to query terms in web page content.
  • FIG. 6 is a block diagram illustrating use of the system 100 for matching query terms and sponsored listings to arbitrary web pages.
  • a web page provider submits one or more URLs each specifying a web page, or a root page pointing to a tree of web pages. URLs may be annotated by the web page owner with optional keywords or text to be associated with the URL.
  • Each web page is placed in a sponsored listings database to be crawled, marshaled, tagged, and retrieved by system 100 . In this way, the system 100 may be used to determine what ads are relevant for displaying on the page.
  • a web page owner may request listings related to a web page by sending an annotated URL request to a context vector processor 700 .
  • the request may contain a URL for the web page and may contain additional annotations about the web page and associated site, and/or annotations about the user viewing the web page, such as the user's location and recent actions.
  • a context vector processor 700 may be used to request from a query data warehouse 720 a set of previous searches with results relevant to the URL, site, or account.
  • the query data warehouse may respond by providing relevant historical search terms and attributes of the set of results associated with those search terms.
  • the context vector processor may analyze the results from the query data warehouse 720 , possibly generating additional candidate search terms, and may also generate a set of selected search terms to be submitted to a sponsored listing server 710 .
  • the sponsored listing server 710 responds with a list of candidate listings and attributes of those listings.
  • the context vector processor 700 analyzes the set of candidate listings, determines which listings to return, optionally reranks the listings, and generates a response to the annotated URL request with listings and associated attributes, which may include the confidence of the match of each listing to the web page.
  • the effectiveness of listings returned by the context vector processor may be optimized by using a freshness optimizer 730 , which may operate asynchronously from the flow of annotated URL requests.
  • the freshness optimizer 730 may request from a query data warehouse 720 search terms that were previously searched, but not searched recently enough to guarantee that a newly resubmitted search would produce the same results.
  • the freshness optimizer 730 may request from a sponsored listing update server 740 a list of search terms that were recently added to the sponsored listings database.
  • the freshness optimizer 730 may collect the search terms from the query data warehouse 720 and/or the sponsored listing update server 740 and submit a set of search terms to the sponsored listing server 710 , which will then produce fresh search results to be stored in the query data warehouse 720 .

Abstract

A system is disclosed for generating a search result list in response to a search request from a searcher using a computer network. A first database is maintained that includes a first plurality of search listings. A second database is maintained that includes documents having general web content. A search request is received from the searcher. A first set of search listings is identified from the first database having documents generating a match with the search request and a second set of search listings is identified from the second database having documents generating a match with the search request. A confidence score is determined for each listing from the first set of search listings wherein the confidence score is determined in accordance with a relevance of each listing when compared to the listings of the second set of search listings. The identified search listings from the first set of search listing are ordered in accordance, at least in part, with the confidence score for each search listing.

Description

    RELATED APPLICATIONS
  • This application claims the benefit of the filing date of U.S. patent application Ser. No. 11/334,096, filed Jan. 18, 2006, and of U.S. Provisional Application No. 60/645,054, filed Jan. 18, 2005, the entire contents of which are incorporated by reference herein.
  • COPYRIGHT NOTICE
  • A portion of the disclosure of this patent document contains material which is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it appears in the Patent and Trademark Office patent file or records, but otherwise reserves all copyright rights whatsoever.
  • BACKGROUND
  • The transfer of information over computer networks is an important way by which institutions, corporations, and individuals conduct business. Computer networks have grown over the years from independent and isolated entities established to serve the needs of a single group into vast internets that interconnect disparate physical networks and allow them to function as a coordinated system. Currently, the largest computer network is the Internet, a worldwide interconnection of computer networks that communicate using a common protocol.
  • The Internet has transformed into a global marketplace of goods and services, driven in large part by the introduction of the World Wide Web (“the web”), an information system. Computers connected to the Internet may access web pages via a browser program, which has a powerful, simple-to-learn graphical user interface. In a web-based search on an Internet search engine, a user enters a search term of one or more keywords, which the search engine then uses to generate a listing of web pages that the user may access via a hyperlink.
  • The search engine functionality of the Internet should be focused to facilitate an on-line marketplace that offers searchers quick, easy and relevant search results while providing Internet network information providers with a cost-effective way to target consumers. A consumer utilizing a search engine that facilitates this on-line marketplace finds companies or businesses that offer the products, services, or information that the searcher is seeking. In the on-line marketplace, companies providing products, services, or information may bid in an open auction-based ranking environment for ranks on a search result list generated by an Internet search engine.
  • Operators of some auction-based search engines may be losing potential revenue, such as by being unable to show enough sponsored results for certain search terms. In addition, irrelevant results and/or lower effective revenues from the results may occur. Some operators may not be able to tightly control trade-offs between relevance and coverage of the results. Likewise users' needs may not be met for search queries of a commercial intent. Irrelevant results may be shown if sponsored results are shown based only on some of the users' query terms. Moreover, advertisers may experience low traffic with some auction-based search engines. The advertisers may carry a burden of generating bidded terms corresponding to their web pages to try to maximize relevant search traffic to their sites. Missed opportunity to provide high quality matches to specific queries may occur. Likewise, many users may access the advertiser's site but not purchase anything because the result turned out to be irrelevant. As such, advertisers' may pay for click-throughs from a user that had a low probability of an item being purchased.
  • BRIEF SUMMARY
  • A system generates a search result list in response to a search request from a searcher using a computer network. A first database may be maintained that includes a first plurality of search listings, wherein each search listing may be associated with one or more search terms. A second database may be maintained that includes documents having general web content. A search request is received from the searcher. A first set of search listings is identified from the first database having documents generating a match with the search request and a second set of search listings is identified from the second database having documents generating a match with the search request. A confidence score is determined for each listing from the first set of search listings wherein the confidence score is determined in accordance with a relevance of each listing when compared to the listings of both sets of search listings. The identified search listings from the first set of search listing are ordered, at least in part, in accordance with the confidence score for each search listing.
  • Other systems, methods, features and advantages of the invention will be, or will become, apparent to one with skill in the art upon examination of the following figures and detailed description. It is intended that all such additional systems, methods, features and advantages be included within this description, be within the scope of the invention, and be protected by the following claims.
  • BRIEF DESCRIPTION OF THE DRAWINGS
  • The invention can be better understood with reference to the following drawings and description. The components in the figures are not necessarily to scale, emphasis instead being placed upon illustrating the principles of the invention. Moreover, in the figures, like referenced numerals designate corresponding parts throughout the different views.
  • FIG. 1 illustrates a system for providing results to search queries.
  • FIG. 2 shows a system for managing queries and database content.
  • FIG. 3 is a block diagram of an exemplary search pipeline.
  • FIG. 4 is a block diagram illustrating an architecture for presenting a combined set of listings from listings of a variety of servers.
  • FIG. 5 illustrates an exemplary screen shot of a web page that may be returned to the user.
  • FIG. 6 is a block diagram illustrating use of the system for matching query terms and sponsored listings to arbitrary web pages.
  • DETAILED DESCRIPTION OF THE DRAWINGS AND THE PRESENTLY PREFERRED EMBODIMENTS
  • A system may combine analysis of advertiser sponsored listings, advertiser web site content, world wide web content, query intent, and/or user behavior statistics to deliver relevant matches, such as sponsored matches, and associated properties of a query and result. As used herein, the term relevant includes results that generate revenue for a search provider, a depth or broad range of results, and/or results relevant to a searcher. The system may analyze in combination the results of a search query provided by more than one server. A relevance or quality of results from a first search query may be determined in accordance with results of a separate, different search query.
  • FIG. 1 illustrates a system 100 for providing results to search queries. A searcher, such as user 110, submits a search query using a web browser to access a front-end server 120, such as YAHOO or OVERTURE. The query may include one or more terms. The front end server 120 may send the search query to a first backend server 130 such as a sponsored listings server or other type of server. The sponsored listings server may provide results to the search query that include advertisers that pay to be included in the results. The first server 130 may access a sponsored listing match engine 140. The sponsored listing server 130 may connect with IDP proxies 150 to handle incoming search queries. The IDP proxies 150 may connect with different search clusters such as sponsored listing search clusters 160 and free web crawl search clusters 170.
  • FIG. 2 shows a system for managing queries and database content. The sponsored listing server 130 may utilize results from the sponsored listing search clusters 160 and the free web crawl search clusters 170. The sponsored listing search clusters 160 may obtain listings from a sponsored listing database 200. The sponsored listings database 200 may be used when the sponsored listings server 130 is operated by an entity different from the operator of the free web crawl search clusters 170. The operator of the sponsored listings server 130 and the free web crawl search clusters 170 may also be the same entity.
  • To obtain listings from the sponsored listings database 200 the system 100 may perform context indexing. The context indexing may include capturing a snapshot of a sponsored listings database 200 for determining listings eligible for an advanced match. Eligible listings include listings whose advertisers have agreed to have their listings displayed using the algorithms of the system 100. Information from the listings database may be transferred to a sponsored listing crawl cluster 210 such as via a sponsored listing update server 220. Sponsored listing marshalers 230 may be used to create indexes of the content. An IDPD Tag Daemon 240 may be used to mark a quality of the indexed pages based, for example, on editorial considerations such as removing spam listings, links to listing that do not work and links with a low rate of click-through. The IDPD Tag Daemon 240 may obtain editorial content from an editorial database 250. Listings may be indexed with associated annotations, crawled content, anchor text such as text associated with inbound links to the target page, or algorithmically generated summaries and keywords.
  • To obtain documents for the free web crawl search clusters 170, content is gathered from web 260 via the free web crawl cluster 260 and free web crawl marshalers 270. Editorial material may be provided from the editorial database 250 to documents at the free web crawl marshalers 280. Editorial material may also be provided at the free web crawl search clusters 170 from the editorial database 250 via an IDPD Tag Daemon 290. A webmap 300 may be used to provide information to, and receive information from, the sponsored listing crawl cluster 210, the free web crawl cluster 270, the editorial database 250, the sponsored listing marshalers 230 and the freeweb crawl marshalers 280. The webmap 300 may provide information about hosts and hyperlinks on the web, including anchor text associated with hyperlinks.
  • The marshalers 230 and 280 produce indexes or search databases used by the search clusters 160 and 170 to respond to search queries. A search database of free web content or general web content may be used to determine the quality of results from a search database of sponsored listings.
  • Transfer of sponsored listings to the web crawling/indexing system may be accomplished using a transfer protocol such as XML encoding. Requests and responses from a search provider to a web search system may be accomplished with a protocol such as IDP (INKTOMI Data Protocol) 2.0 and transfers may be accomplished by hyper text transfer protocol (http). An incremental update protocol may be used for the efficient transfer of sponsored listings such as by only transferring those listing that have changed since the last transfer. Properties of the listings may be converted to appropriate representations for the web search database. Conversion to indexable web content may be accomplished using XSLT transformation.
  • Sponsored listings may be submitted with a display title, display abstract, display hostname, destination URL, crawl URL, bidded terms, account id, ad group, historical CTR, and the like. For simplicity, the advertiser may submit a set of crawl URLs or root URLs pointing to a tree of web pages, without specifying the titles, abstracts, and bidded terms. Alternatively, or additionally, an advertiser can specify that there is no content to be crawled for a listing, if the advertiser provides a title, abstract, display hostname, and destination URL for the listing. Multiple listings may use the same crawl URL. An advertiser may submit multiple title and abstract combinations for the same URL and bidded terms, along with an optional specification of how to choose among similar titles and abstracts, for example, by specifying the relative display frequencies or requesting that the system optimize the choice for improved click-through and/or conversion.
  • Multiple listings may be grouped in the sponsored listings database by, for example, account, URL, related group of URLs, and/or bidded terms, in order to improve performance of indexing, retrieval, and/or accounting. For efficiency, a crawl URL shared by multiple listings may be crawled only once. Other amounts of crawls may also be used.
  • Non-indexed properties associated with a listing such as a cost to an advertiser when a user 110 clicks the listing, an account/listing ID, and negative keywords may be included in the listings. Negative keywords include terms chosen by an advertiser such that, if the term is part of a search query, an advertiser's listing will not appear. In addition to converting properties of the listings, documents may be annotated with helper words to be indexed, such as by adding stemming variants, synonyms, and/or related terms based on the bidded terms, title, abstract, URL, and/or crawled content for a listing. A new section of the document may be created to add alternate terms to the listing, such as by adding ‘shoe’ and ‘shoes’, ‘running’ and ‘run’ or other variants of terms, and the like.
  • The system 100 may handle search queries from the user 110 by providing results to the search query in accordance, at least in part, with result relevance scoring, such as a confidence score, and/or with ranking criteria. Search results from the sponsored listing search clusters 160 and the free web crawl search clusters 170, or other database, may be performed, and the results compared to determine results to be displayed to the user.
  • When the user 110 enters a query, the query may be processed by the proxies 150. At the proxies 150, the system 100 may edit the query in ways such as auto-correction for misspelled queries, segmentation of the query into words and phrases, changing case, using concept expansion, and removing less important words, such as based on occurrence frequencies in historical query logs and/or web pages. The system 100 may transform the user query into a preferred form for a back-end query search engine. Depending on an implementation, no results may be returned if the query or transformed query match terms on a list of objectionable terms. For example, no results may be returned if the query contains objectionable terms such as terms associated with videos related to beheadings, adult content, and the like. The objectionable terms may be decided on by an editor of the search engine as a matter of policy or others such as advertisers or automatic adult classifiers. A list of required query phrases may be provided by an editor or advertiser, or generated by an algorithm, and these phrases may be associated with a listing, multiple listings, or all listings from an account or domain. If one or more search terms associated with a sponsored listing contains a required phrase that is not contained in the user query, the listing may be excluded from the set of results for that query.
  • The modified query may be issued to the sponsored listing search clusters 160 and at least one other search cluster, such as the free web crawl search clusters 170 that search for general web content. The free web crawl search clusters 170 may search a full-scale web index or a portion of the full-scale web index chosen in accordance with determined parameters. The full-scale web index may include, for example, 10 billion results while a sample portion of the index may include 200 million results for simplified processing, speed, and/or to increase the relevance to the user 110 of the results. The web content may be sampled by including documents based on properties of the web links such as by using links that are accessed more frequently than others.
  • Information may be provided to the sponsored listing search clusters 160 via a system monitor/executor 310, such as from a sponsored listings search provider 330. The sponsored listings search provider may utilize a contract & click data warehouse 340 to maintain data on sponsors and the clicks that occur to the sponsors' websites. The data warehouse 340 may receive information about listings from data quality review module 350, such as via an account management and reporting module 360. Information about user clicks to a website may be received via redirect servers 370, for example, after being processed by click log processing and fraud detection module 380. The data warehouse 340 may also receive information about the queries issued by users and results shown to users from Sponsored Listing Servers 130 and/or IDP proxies 150 via query and an impression procession module 390. The computer processing with reference to the servers, databases, and modules as disclosed with reference to FIG. 2 and throughout the application may be performed through a processor-executable code located on a processor-readable medium.
  • The sponsored listing search clusters 160 may provide results in a first way, such as results of advertisers that pay to be included in the sponsored listing database, ranked, for example, by revenue that the results bring to the search provider. The free web crawl search clusters 170 may provide results in another way, such as all possible results or a portion of all possible results whether or not advertiser related, ranked, for example, by popularity. Results of a search of the sponsored listing database may be compared with results of the general web content database, and the results having top ranks from the sponsored listing database may be displayed if the results also occur within top ranks of the results for the general web content database. A relevance or confidence score regarding a relevance of the documents may also be used for optimizing placement on the results page, for example, whether the listing should be shown to the user 110 at a top, side, or bottom of the page. Sponsored results may also be re-ranked or removed as being a duplicate based on a comparison of sponsored listings and web results.
  • The system 100 may also use various scoring parameters for the sponsored listing database query. For example, modified weights may be used for different parts of the search result document. For example, a search term occurring in a title of the document may be considered more important than if the search term only occurred in the body of the document.
  • Relevant results may be determined for each database and the combined results may be sorted by the relevance or confidence score. The relevance score may be based on many factors including a presence of query terms, proximity of the query terms to each other, matching of query terms to document anchor text, a document category, and determined measures of document quality. The sponsored listing database results that occur in the set of sorted top results may be further modified such that results where negative keywords match query occur may be discarded. In addition, if the query contains terms on a list of designated trademarked terms, any result that does not contain all trademarked terms in a visible portion of result or the bidded terms for the listing may be discarded.
  • Features for the results from the sponsored listing database may be calculated to help determine which results are to be displayed. The features include an actual cost to the advertiser for a user 110 to click the result based, for example, on bid rankings. The features may also be determined in accordance with other factors such as historical click-through and conversion data. Historical click-through data includes information on the number of times a user 110 clicks through to the advertisers' site when the site is provided as a result. Conversion data includes information from the advertiser regarding whether the user 110 performed an action desired by the advertiser, such as making a purchase when the user 110 visiting the advertiser's site. Other features include rank of the result in the full sorted set of top results from the combination of the sponsored listing database and general web content database and a relevance score difference and ratio from top results of the general web content database. Other features of the result include whether the query word/phrase appears in the title, abstract, URL, display host, bidded keyphrase, and/or algorithmically generated keywords or summaries of the result.
  • Individual features may be analyzed alone or in combination to increase a composite confidence that the result is a good match to the query. The features may also be used to create a confidence score for each result, and the sponsored listing server 130, or other server, may sort the results based on the score such that results that receive a higher score are displayed first on a list of search results displayed to the user. The sponsored listing server 130 may return properties of the input query, such as a categorization of the query, and may return properties of each listing, such as keywords associated with each listing. Functions for relevance scoring and reranking functions may be constructed entirely or in part by machine-learning methods, such as methods to find coefficients of linear formulas, methods to construct decision trees, methods to construct Support Vector Machines, neural networks, Bayesian networks, or other methods. Features, combinations of features, and scoring and reranking functions may be constructed with the aid of historical click and impression data and/or editorially labeled relevance judgments.
  • In accordance with the confidence score, and other factors, such as whether the search provider desires to display more relevant results and/or more revenue generating results, the results to be displayed may be determined. Real-time reporting of the matching statistics may be provided so that the features determining the confidence score may be modified for a particular search provider. The listing ID may be returned to the search provider along with the confidence score for each web search database result. The listing ID for the web search database results may be used to lookup current title, abstract, display hostname, cost to advertiser for click-through, and listing/account status. The matching of listings may be decoupled based on crawled and indexed content, which may be hours or days old, from the retrieval of the current title, abstract, URL, CPC, and listing/account status.
  • Results determined to be the most relevant in view of any combination of the above-described feature may be displayed. If the advertiser has supplied multiple alternative titles and abstracts for a listing, the choice of title and abstract to be displayed may be made based on relevance to the query, click-through statistics, and/or specification of the desired frequency of display of each title and abstract. Terms in listing in accordance with the query and transformed query may be highlighted for the user. Eligible results may be blended with exact matching and other algorithms based on combination of cost to the advertiser for each click-through, relevance scores and/or preference rules. Flexible policies may be implemented for the tiering or mixing results generated by the system 100 with matches from other systems. Redundant results may be removed based on a combination of account id, the domain name, hostname extracted from the crawl URL, display URL, display hostname, and/or the like.
  • FIG. 3 is a block diagram of an exemplary search pipeline 400. At block 410 a raw query is received from the user 110. The raw query may processed, such as with a sub-phrase match (SPM) match driver (MD) canonical (canon) query module at block 420 and/or MD canonical query module at block 430, to determine other variations of the terms in the query, remove superfluous terms or endings, such as plural endings or ‘ing’ endings, and determine synonyms for the raw query. The raw query may be sent to IDP proxies at block 440. The MD canon query may be sent to a title and/or description match server at block 450. The MD canon query 430 may also be sent to an exact match or orthographic match driver (OMD) match at block 460. Results of the exact match or OMD match may be sent to a phrase match or broad match server at block 470, along with results from the SPM MD canon query. The SPM MD canon query results may also be sent to a local match server at block 480. The results from all the servers 440, 450, 470 and 480 may be sent to a server listings module at block 490. Each listing is analyzed to determine a confidence rating of the listing that the listing is relevant.
  • An exemplary core matching algorithm for the system 100 follows:
  • A “full result set” is a blended set of the top num_candidate results from a sponsored listing database and other databases, ranked by a relevance score. Each result from the sponsored listings database within the full result set may be rescored and reranked by computing a number of “demotion” terms that attempt to capture non-ideal characteristics of a result.
  • Using each item's “rank” and normalized “score” in the full result set:

  • rank_demotion=log 2(rank)

  • rawscore_demotion=1−score/top_score
      • where top_score is the score of #1 result in full result set.
  • Weights of each query word may be set as 1.0 by default, but may be overridden to any value between zero and one. Based on word weights, max_query_score is the sum of weights of each unique query word.
  • Three metrics may be computed separately for each “section”, where “section” is the listing title, abstract, bidded terms, or list of key topics for the listing. If an “editorial_switch” parameter is 1, then the bidded terms may be considered to be empty for listings that have not been editorially reviewed. The absence_demotion, edit_distance_demotion, and order_distance_demotion may be determined as follows:

  • absence_demotion=(max_query_score−match_weight)/max_query_score,
      • where match_weight is the weighted sum of unique query words in the section

  • edit_distance_demotion=(max_query_score−max)/max_query_score,
      • where max may be the maximum weighted sum of unique query words among sequences of consecutive section words in section that are all query words

  • order_distance_demotion=0 for 1-word queries, 1 if no query words appear in section, otherwise computed as 1−(ordered_words−1)/(nunique_query_words−1),
      • where ordered_words may be determined by ignoring non-query words in section and finding the maximum number of unique ordered query words.
  • When determining, such as by computing, the section demotion for a list of sections, the minimum demotion of any section may be used for each of the three demotion types:

  • section_demotion(section_list)=edit_distance_demotion_weight*min_of_sections(edit_distance_demotion)+order_distance_demotion_weight*min_of_sections(order_distance_demotion)+absent_words_demotion_weight*min_of_sections(absence_demotion)
  • The “editorial demotion” may be 1 to denote a listing that has not been editorially reviewed, or 0 otherwise. Other notations may be used.
  • The final relevance score may be determined as:

  • relevance_confidence=base_confidence−rawscore_demotion*rawscore_weight−rank_demotion*rank_weight−section_demotion(title,abstract)*title_abstract_weight−section_demotion(bidded_term)*bidded_term_weight−section_demotion(title,abstract,bidded_term)*title_abstract_bidded_term_weight−section_demotion(topics)*topics_weight−editorial_demotion.*editorial_weight
  • To determine actual_cpc (where cpc is cost per click) for each sponsored result, sponsored results may be sorted by the bid_cpc associated with each result. Any result with a bid_cpc less than min bid_cpc may be assigned an actual_cpc equal to its bid_cpc. Otherwise, the bottom result may be assigned an actual_cpc of its bid_cpc and each other result is assigned an actual_cpc of the next result's bid_cpc plus cpc_increment, subject to the constraint that actual_cpc may not exceed bid_cpc.
  • Results with a relevance_confidence below min_relevance_confidence may be rejected. Remaining results may be reranked by a ranking_score determined as follows:

  • if actual_cpc <=cpc_knee, then cpc=actual_cpc

  • otherwise, cpc=(cpc_knee+cpc_slope*log 2(actual_cpc+1))/100)

  • ranking_score=relevance_confidence+cpc_weight*cpc.
  • Alternatively, the ranking_score may be determined as follows:

  • if actual_cpc <=cpc_knee, then cpc=actual_cpc

  • otherwise, cpc=cpc_knee

  • ranking_score=relevance_confidence+cpc_weight*cpc.
  • Exemplary combination of settings for tunable input parameters for the US market:
      • 0.05 rank_weight
      • 0.24 rawscore_weight
      • 0.45 title_abstract_weight
      • 0.0 bidded_term_weight
      • 0.1 title_abstract_bidded_term_weight
      • 0.01 topics_weight
      • 0.2 edit_distance_demotion_weight
      • 0.3 order_distance_demotion_weight
      • 0.5 absent_words_demotion_weight
      • 0.95 base_confidence
      • 0.01 editorial_weight
      • 1 editorial switch
      • 2.0 cpc_weight
      • 100 num_candidates
      • 0.35 min_relevance_confidence
      • $0.10 min bid_cpc
      • $0.01 cpc_increment.
      • $0.20 cpc_knee
      • $1.00 cpc_slope
  • FIG. 4 is a block diagram illustrating an architecture 500 for presenting a combined set of listings from listings of a variety of servers. A user query is received at block 510. At block 520, the query is sent to different search servers. Variations of the query may be sent to the different servers. Certainty scores may be attached to each variation of query, as a probability that the variation will return a relevant result. At block 530, the search servers match listings to the query. A certainty score may be attached to the listings that the listing is a relevant match. At block 540, the listings are ranked such as in accordance with the certainty scores and other factors such as cost per click (block 550), click through rates (block 560) and based on editorial considerations (block 570). At block 580, the listing may be presented in accordance with the rankings and other factors. The listing may be shown in order of confidence, such as with the highest confidence rated listing being shown first and/or at a particular area of displayed browser page.
  • FIG. 5 illustrates an exemplary screen shot 600 of a web page that may be returned to the user. If the user 110 enters the terms ‘deboning chicken’ there may be no sponsored results with that term. Since the system 100 compares results from the web search database and the general web content database, a result like ‘Chicken Shears at Shopping.com’ may be discovered. When connecting to the link, the user 110 may observe a result such as ‘Diamond Cut Multi-purpose Kitchen Shears’ that does not include either the terms ‘deboning’ or ‘chicken’ in the title.
  • The system 100 may be customized to address varying needs of different search providers. Sponsored listings may be scored differently than general web documents, for example, to either place more or less emphasis on the sponsored listings. Crawl frequencies for updating the web search database may be changed, such as from weeks to days, depending on a desired freshness of the content in the database. Tunable parameters may be provided for individual matching criteria. Results may be restricted to those where all the query terms are present in determined portions of the results, such as the title or the abstract. Stopwords such as ‘a’, ‘the’ and ‘and’ may be excluded or given less weight. Some query terms may be weighted as being more important than other terms. The system 100 may be tunable; for instance, the query terms may have different weights for different search providers. For example the term ‘picture’ may be weighted as being half as important as other words. Results may be ranked relative to web content without requiring retrieval of the full web content listings, such as the title and abstracts. Results sets may be cached to reduce latency and lower serving costs.
  • The system 100 may also include other features to help improve relevance and/or revenue such as by using historical click-through data regarding the number of users 110 that accessed an advertiser's site and conversion data regarding the number of users 110 that purchased items from the advertiser's site. The data may be aggregated per URL, domain, or contract for all queries, or for specific queries and query terms.
  • Queries from the user 110 may be modified such as by editorially determining rewrites for specific queries or query terms, or by algorithmically determining rewrites. Certain query terms may be deleted or reduced in weight. Related terms and synonym expansion of the term may be added to the query and weighted accordingly. The system 100 may also expand query terms and add stemming variants, such as by adding or removing ‘s’ and ‘ing’ to the term, and abbreviation variants of the term may be added to the query. Queries may be segmented into phrases for better proximity scoring of the terms. For example, the terms ‘New’ and ‘Mexico’ may be grouped together to get results for the state ‘New Mexico’. Queries written without spaces separating words, such as in Chinese queries, may be segmented into words. Compound words, such as in Korean or German queries, may be segmented into component words. Script variants may be allowed such as for Asian languages, such as variants in the Chinese language. A list of terms may be generated to use for hit-highlighting in accordance with rewrites of the term. For example, when the results are displayed, the search terms may be highlighted to aid the user, and variants of the search term may also be highlighted. For the search term ‘S.F. opera’ the variant ‘San Francisco opera’ may also be highlighted when the results are displayed.
  • The system 100 may also modify content of the web page results stored in the web search database and/or the general web content database. Page content may be thinned or increased, based on many factors including page structure, page template extraction, for example, applied to a collection of documents associated with a contract, feed, or site, and click and conversion statistics per the query or query term. Prices may be added to the web page description to avoid irrelevant matches. Synonyms, alternate spellings, stemming variants, abbreviation variants, and related terms may also be added to the web page description.
  • The system 100 may also be configured to accommodate other features. Components requiring development, such as query classification, spelling correction, and query rewriting, may be added to the system 100. Relevance tuning may be allowed for region and language preferences. Region-specific scoring may be implemented such that query results tied to a particular region are weighed higher for a user 110 from that particular region. Personalization data may be incorporated into relevance scoring, using many factors such as content preferences, user location, user demographics and query history. The search provider may be allowed to control whether results should be weighted towards those that appear more relevant or whether a greater coverage/depth of results is preferred. Editorial judgments, positive or negative, may be incorporated for sponsored listing data based on URLs, domains, or contracts. Sponsored listings feeds may be used to automatically generate bid terms and bid amounts for related listings. The system 100 may be used for implementing exact match to bidded terms in order to lower serving cost and to incorporate relevance ranking of exact match listings based on factors used for inexact match, such as web content.
  • Other features of the system 100 include the ability to determine results sets editorially, by specifying complete results sets or specific results to include, exclude, promote, or demote. The system 100 may incorporate current advertiser budget in selection and ranking of results. For example, if the advertiser has used up a determined percentage of its budget, a result for that advertiser may not be shown unless the result ranks above a certain rank, even if the result would otherwise have been displayed. Local/geographic queries, domain queries, and product number queries may be scored differently. For example, results that otherwise may not have been displayed, are given extra weight and are shown because a local query has occurred.
  • A queue of listings or query-listing pairs to be editorially reviewed may be automatically generated based on several factors including click data, conversion data, query or query-term frequency, listings display frequency, listing bid amounts, results of previous editorial reviews (if any), and user complaint frequency, and may be aggregated by account, domain, query, or query term. Representative web content may be selected and used for generating relative ranks of web search database content using factors other than the frequency of a selection including, for example, optimizing coverage by language, region, content-type, internet domain, topic, and vocabulary used.
  • The system 100 may be adapted to show offers other than sponsored listings, such as auction listings, job offers, etc., and to evaluate relevance of those offers relative to a content database other than generic web content, such as a database driving a vertical search or directory such as travel, shopping, etc. Other content attributes in relevance evaluation may be incorporated, such as prices, availability, duration of offer, popularity, vendor rating, and product details such as size, color and trim. The system 100 may be used to determine when and where on the page to show links to preferential content, for example, as done with ‘YAHOO! Shortcut’ listings, such that if a query includes the term ‘HONDA CIVIC’, a link to YAHOO! Autos site may be provided. Other features include that advertisers may submit web pages without fully specifying bidded terms, title, abstract. Also, a listing abstract may be modified based on matches to query terms in web page content.
  • FIG. 6 is a block diagram illustrating use of the system 100 for matching query terms and sponsored listings to arbitrary web pages. A web page provider submits one or more URLs each specifying a web page, or a root page pointing to a tree of web pages. URLs may be annotated by the web page owner with optional keywords or text to be associated with the URL. Each web page is placed in a sponsored listings database to be crawled, marshaled, tagged, and retrieved by system 100. In this way, the system 100 may be used to determine what ads are relevant for displaying on the page.
  • A web page owner may request listings related to a web page by sending an annotated URL request to a context vector processor 700. The request may contain a URL for the web page and may contain additional annotations about the web page and associated site, and/or annotations about the user viewing the web page, such as the user's location and recent actions. Upon a request such as annotated URL request, a context vector processor 700 may be used to request from a query data warehouse 720 a set of previous searches with results relevant to the URL, site, or account. The query data warehouse may respond by providing relevant historical search terms and attributes of the set of results associated with those search terms. The context vector processor may analyze the results from the query data warehouse 720, possibly generating additional candidate search terms, and may also generate a set of selected search terms to be submitted to a sponsored listing server 710. The sponsored listing server 710 responds with a list of candidate listings and attributes of those listings. The context vector processor 700 analyzes the set of candidate listings, determines which listings to return, optionally reranks the listings, and generates a response to the annotated URL request with listings and associated attributes, which may include the confidence of the match of each listing to the web page.
  • The effectiveness of listings returned by the context vector processor may be optimized by using a freshness optimizer 730, which may operate asynchronously from the flow of annotated URL requests. The freshness optimizer 730 may request from a query data warehouse 720 search terms that were previously searched, but not searched recently enough to guarantee that a newly resubmitted search would produce the same results. The freshness optimizer 730 may request from a sponsored listing update server 740 a list of search terms that were recently added to the sponsored listings database. The freshness optimizer 730 may collect the search terms from the query data warehouse 720 and/or the sponsored listing update server 740 and submit a set of search terms to the sponsored listing server 710, which will then produce fresh search results to be stored in the query data warehouse 720.
  • While various embodiments of the invention have been described, it will be apparent to those of ordinary skill in the art that many more embodiments and implementations are possible within the scope of the invention. Accordingly, the invention is not to be restricted except in light of the attached claims and their equivalents.

Claims (26)

1. A method of generating a search result list in response to a search query from a searcher using a computer network, the method executed with a computer having a processor, the method comprising:
(a) maintaining a first database including a first plurality of search listings comprising sponsored search listings having a sponsored content;
(b) maintaining a second database including documents having a general web content;
(c) receiving, by the processor, a search query from the searcher;
(d) identifying, by the processor, from the first database a first set of search listings comprising sponsored search listings having documents generating a match with the search query, and from the second database a second set of search listings comprising general web listings having documents generating a match with the same search query as used to identify the sponsored first set of search listings;
(e) determining, by the processor, a confidence score for each listing from the first set of search listings, wherein the confidence score is determined in accordance with a relevance of each listing when compared to the listings of the second set of search listings after execution of step (d); and
(f) ranking, by the processor, the sponsored first set of search listings in accordance, at least in part, with the confidence score for each search listing thereof;
(g) wherein the processor associates the sponsored first set of search listings with at least one feature different than the relevance of each search listing when compared to the listings of the second set of search listings, and the processor further orders the sponsored first set of search listings in accordance with the at least one feature, the at least one feature comprising any of a group consisting of a number of clicks, a click-through rate, and a conversion rate derived from historical click-through and conversion data from user activity on the sponsored first set of search listings.
2. The method of claim 1, wherein each search listing is associated with one or more search terms, at least one of the one or more search terms associated with the at least one search listing is automatically generated based on information located at a first address associated with the at least one search listing.
3. The method of claim 1, further comprising:
modifying the query before identifying the first and second sets of search listings in step (d).
4. The method of claim 1, wherein each listing from the first set of search listings is associated with a first set of attributes, each attribute of the first set of attributes associated with an attribute weight, and wherein the confidence score is determined by weighting at least one attribute for each listing with the associated attribute weight.
5. The method of claim 4, wherein the searcher is associated with a geographical region, and wherein the confidence score is determined by weighting at least one attribute for each listing with a second set of attribute weights associated with the geographical region.
6. The method of claim 1, comprising discarding at least one of the sponsored first set of search listings in accordance with a list of negative keywords.
7. The method of claim 1, computing a plurality of demotion terms related to a plurality of the top-ranked sponsored first set of search listings, wherein the demotion terms capture non-ideal characteristics thereof;
updating the confidence score of each of the top-ranked sponsored first set of search listings based on the plurality of demotion terms; and
re-ranking the top-ranked sponsored first set of search listings according to the updated confidence scores.
8. The method of claim 7, wherein the demotion terms comprise demotion metrics computed for one or more of the group consisting of: (i) a document section selected from the group consisting of an abstract, a bidded term, and a list of key topics, and (ii) whether or not the listing has been editorially reviewed.
9. The method of claim 1, wherein the confidence scores are determined using a machine-learning method.
10. The method of claim 1, further comprising:
optimizing placement of the search listings on a location of a search results page based on the confidence score, wherein the location comprises a top, a side, or a bottom of the search results page.
11. The method of claim 1, wherein the at least one feature further comprises a rank among a sorted set of top search listings of the sponsored first set of search listings from a combination of the sponsored first set of search listings and the second set of search listings.
12. The method of claim 1, wherein the at least one feature further comprises whether the search query appears in any selected from the group consisting of: a display host, an algorithmically generated keyword or summary, and a combination thereof.
13. A server for generating a search result list in response to a search query from a searcher using a computer network, comprising:
(a) a memory for use in storing data and instructions; and
(b) a processor in communication with the memory and for enabling actions based on the stored instructions, the stored instructions including instructions for:
(i) communicating with a first database, the first database including a first plurality of search listings comprising sponsored search listings having a sponsored content;
(ii) communicating with a second database, the second database including documents having a general web content;
(iii) receiving a search query from the searcher;
(iv) identifying from the first database a first set of search listings comprising sponsored search listings having documents generating a match with the search query, and from the second database a second set of search listings comprising general web listings having documents generating a match with the same search query as used to identify the sponsored first set of search listings;
(v) determining a confidence score for each listing from the first set of search listings, wherein the confidence score is determined in accordance with a relevance of each listing when compared to the listings of the second set of search listings after execution of step (b)(iv); and
(vi) ordering the identified sponsored first set of search listings in accordance, at least in part, with the confidence score for each search listing thereof;
(c) wherein the sponsored first set of search listings are associated with at least one feature different than the relevance of each search listing when compared to the listings of the second set of search listings, and the sponsored first set of search listings are further ordered in accordance with the at least one feature, the at least one feature comprising any of a group consisting of a number of clicks, a click-through rate, and a conversion rate derived from historical click-through and conversion data from user activity on the sponsored first set of search listings.
14. The server of claim 13, wherein the stored instructions further comprise instructions for
modifying the search query in a first manner for identifying the first set of search listings; and
modifying the search query in a second manner for identifying the second set of search listings.
15. The server of claim 14, wherein the first manner comprises giving query terms of the search query different weights for different search providers.
16. The server of claim 14, wherein each listing from the first set of search listings is associated with a first set of attributes, each attribute of the first set of attributes associated with an attribute weight, and wherein the confidence score is determined by weighting at least one attribute for each listing with the associated attribute weight.
17. The server of claim 13, wherein the stored instructions further comprise instructions for:
storing the second set of search listings; and
updating, automatically, the second set of search listings after a predetermined period of time.
18. The server of claim 13, wherein the first database is maintained by a first entity and the second database is maintained by a second, specific entity.
19. The server of claim 13, wherein the processor executes instructions to discard at least one of the sponsored first set of search listings in accordance with a list of negative keywords.
20. The server of claim 13, wherein each search listing is associated with one or more search terms.
21. The server of claim 13, wherein the at least one feature further comprises a rank among a sorted set of top search listings of the sponsored first set of search listings from a combination of the identified search listings and the second set of search listings.
22. The method of claim 13, wherein the at least one feature further comprises whether the search query appears in at least one of group consisting of: a display host, an algorithmically generated keyword or summary, and a combination thereof.
23. A processor-readable medium having processor-executable code thereon for enabling and performing operations to generate a search result list in response to a search query from a searcher using a computer network, the operations comprising:
(a) maintaining a first database including a first plurality of search listings comprising sponsored search listings having a sponsored content;
(b) maintaining a second database including documents having a general web content;
(c) receiving a search query from the searcher;
(d) identifying from the first database a first set of search listings comprising sponsored search listings having documents generating a match with the search query, and from the second database a second set of search listings comprising general web listings having documents generating a match with the same search query as used to identify the sponsored first set of search listings;
(e) determining a confidence score for each listing from the first set of search listings wherein the confidence score is determined in accordance with a relevance of each listing when compared to the listings of the second set of search listings after execution of step (d); and
(f) ordering the sponsored first set of search listings in accordance, at least in part, with the confidence score for each search listing thereof;
(g) wherein the sponsored first set of search listings are associated with at least one feature different than the relevance of each search listing when compared to the listings of the second set of search listings, and the sponsored first set of search listings are further ordered in accordance with the at least one feature, the at least one feature comprising any of a group consisting of a number of clicks, a click-through rate, and a conversion rate derived from historical click-through and conversion data from user activity on the sponsored first set of search listings.
24. The processor-readable medium of claim 23, wherein the identified search listings are further ordered in accordance with a list of trademark keywords.
25. The processor-readable medium of claim 23, wherein the at least one feature is selected from the group consisting of whether a match occurred in an abstract, whether a match occurred in a display host, whether a match occurred in an algorithmically generated keyword, and whether a match occurred in an algorithmically generated summary.
26. The processor-readable medium of claim 23, wherein each search listing may be associated with one or more search terms.
US12/705,882 2005-01-18 2010-02-15 Matching and ranking of sponsored search listings incorporating web search technology and web content Abandoned US20100174710A1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US12/705,882 US20100174710A1 (en) 2005-01-18 2010-02-15 Matching and ranking of sponsored search listings incorporating web search technology and web content

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
US64505405P 2005-01-18 2005-01-18
US11/334,096 US7698331B2 (en) 2005-01-18 2006-01-18 Matching and ranking of sponsored search listings incorporating web search technology and web content
US12/705,882 US20100174710A1 (en) 2005-01-18 2010-02-15 Matching and ranking of sponsored search listings incorporating web search technology and web content

Related Parent Applications (1)

Application Number Title Priority Date Filing Date
US11/334,096 Continuation US7698331B2 (en) 2005-01-18 2006-01-18 Matching and ranking of sponsored search listings incorporating web search technology and web content

Publications (1)

Publication Number Publication Date
US20100174710A1 true US20100174710A1 (en) 2010-07-08

Family

ID=36692861

Family Applications (2)

Application Number Title Priority Date Filing Date
US11/334,096 Expired - Fee Related US7698331B2 (en) 2005-01-18 2006-01-18 Matching and ranking of sponsored search listings incorporating web search technology and web content
US12/705,882 Abandoned US20100174710A1 (en) 2005-01-18 2010-02-15 Matching and ranking of sponsored search listings incorporating web search technology and web content

Family Applications Before (1)

Application Number Title Priority Date Filing Date
US11/334,096 Expired - Fee Related US7698331B2 (en) 2005-01-18 2006-01-18 Matching and ranking of sponsored search listings incorporating web search technology and web content

Country Status (6)

Country Link
US (2) US7698331B2 (en)
EP (1) EP1839124A4 (en)
JP (1) JP4977624B2 (en)
KR (1) KR100932318B1 (en)
CN (1) CN101601032A (en)
WO (1) WO2006078794A2 (en)

Cited By (16)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20090327278A1 (en) * 2008-06-26 2009-12-31 Baran-Sneh Alex System and method for ranking web content
US20100235343A1 (en) * 2009-03-13 2010-09-16 Microsoft Corporation Predicting Interestingness of Questions in Community Question Answering
US20100235311A1 (en) * 2009-03-13 2010-09-16 Microsoft Corporation Question and answer search
US20110072045A1 (en) * 2009-09-23 2011-03-24 Yahoo! Inc. Creating Vertical Search Engines for Individual Search Queries
US20120010994A1 (en) * 2010-07-08 2012-01-12 American Express Travel Related Services Company, Inc. Systems and methods for transaction account offerings
WO2013003500A1 (en) * 2011-06-28 2013-01-03 Redbox Automated Retail, Llc System and method for searching and browsing for directly and indirectly matching media content
US20130054556A1 (en) * 2010-05-13 2013-02-28 Adthena Ltd. Method and system for compiling competitive advertiser and keyword information for search engine advertisers
US9098569B1 (en) * 2010-12-10 2015-08-04 Amazon Technologies, Inc. Generating suggested search queries
US9110993B1 (en) * 2010-01-15 2015-08-18 Google Inc. Search query reformulation using result term occurrence count
US20150254342A1 (en) * 2011-05-30 2015-09-10 Lei Yu Video dna (vdna) method and system for multi-dimensional content matching
US9183323B1 (en) 2008-06-27 2015-11-10 Google Inc. Suggesting alternative query phrases in query results
US20150347596A1 (en) * 2010-08-20 2015-12-03 Bitvore Corporation Bulletin Board Data Mapping and Presentation
CN105516319A (en) * 2015-12-15 2016-04-20 华为技术有限公司 Method and device for managing HTTP messages
US9361388B1 (en) * 2015-07-07 2016-06-07 Yext, Inc. Suppressing duplicate listings on multiple search engine web sites from a single source system given that a publisher selects a different listing as a synchronized listing
CN105975632A (en) * 2011-06-24 2016-09-28 阿里巴巴集团控股有限公司 Searching method, relation building method and associated device
US10762156B2 (en) 2015-07-07 2020-09-01 Yext, Inc. Suppressing duplicate listings on multiple search engine web sites from a single source system triggered by a user

Families Citing this family (141)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7617197B2 (en) * 2005-08-19 2009-11-10 Google Inc. Combined title prefix and full-word content searching
US7536382B2 (en) * 2004-03-31 2009-05-19 Google Inc. Query rewriting with entity detection
US7606793B2 (en) 2004-09-27 2009-10-20 Microsoft Corporation System and method for scoping searches using index keys
US7761448B2 (en) 2004-09-30 2010-07-20 Microsoft Corporation System and method for ranking search results using click distance
US7739277B2 (en) * 2004-09-30 2010-06-15 Microsoft Corporation System and method for incorporating anchor text into ranking search results
US7827181B2 (en) 2004-09-30 2010-11-02 Microsoft Corporation Click distance determination
US20110208732A1 (en) 2010-02-24 2011-08-25 Apple Inc. Systems and methods for organizing data items
US7979457B1 (en) 2005-03-02 2011-07-12 Kayak Software Corporation Efficient search of supplier servers based on stored search results
US7792833B2 (en) * 2005-03-03 2010-09-07 Microsoft Corporation Ranking search results using language types
US8832055B1 (en) 2005-06-16 2014-09-09 Gere Dev. Applications, LLC Auto-refinement of search results based on monitored search activities of users
US8086605B2 (en) * 2005-06-28 2011-12-27 Yahoo! Inc. Search engine with augmented relevance ranking by community participation
US20070027750A1 (en) * 2005-07-28 2007-02-01 Bridgewell Inc. Webpage advertisement mechanism
US20070250501A1 (en) * 2005-09-27 2007-10-25 Grubb Michael L Search result delivery engine
US20070214118A1 (en) * 2005-09-27 2007-09-13 Schoen Michael A Delivery of internet ads
US20070244868A1 (en) * 2005-10-31 2007-10-18 Grubb Michael L Internet book marking and search results delivery
US7783632B2 (en) * 2005-11-03 2010-08-24 Microsoft Corporation Using popularity data for ranking
US20070260520A1 (en) * 2006-01-18 2007-11-08 Teracent Corporation System, method and computer program product for selecting internet-based advertising
US20070239659A1 (en) * 2006-03-30 2007-10-11 Oracle International Corporation Query generator
US7814112B2 (en) 2006-06-09 2010-10-12 Ebay Inc. Determining relevancy and desirability of terms
US7657626B1 (en) 2006-09-19 2010-02-02 Enquisite, Inc. Click fraud detection
US8484283B2 (en) * 2006-08-18 2013-07-09 Akamai Technologies, Inc. Method and system for mitigating automated agents operating across a distributed network
US7752195B1 (en) 2006-08-18 2010-07-06 A9.Com, Inc. Universal query search results
US8401921B2 (en) * 2006-09-21 2013-03-19 Alejandro Backer Seller-driven method for a bid set system
WO2008091411A2 (en) * 2006-09-26 2008-07-31 Accoona Corp Apparatuses, methods and systems for an information comparator comparison engine
US20080104049A1 (en) * 2006-10-25 2008-05-01 Microsoft Corporation Document ranking utilizing parameter varying data
WO2008086299A2 (en) 2007-01-08 2008-07-17 Skaf Mazen A System and method for tracking and rewarding users
KR100898456B1 (en) * 2007-01-12 2009-05-21 엔에이치엔(주) Method for offering result of search and system for executing the method
WO2008097810A2 (en) * 2007-02-02 2008-08-14 Veoh Networks, Inc. Indicator-based recommendation system
US8060505B2 (en) * 2007-02-13 2011-11-15 International Business Machines Corporation Methodologies and analytics tools for identifying white space opportunities in a given industry
US9449322B2 (en) 2007-02-28 2016-09-20 Ebay Inc. Method and system of suggesting information used with items offered for sale in a network-based marketplace
US7827172B2 (en) * 2007-03-14 2010-11-02 Yahoo! Inc. “Query-log match” relevance features
US8086624B1 (en) 2007-04-17 2011-12-27 Google Inc. Determining proximity to topics of advertisements
US8229942B1 (en) 2007-04-17 2012-07-24 Google Inc. Identifying negative keywords associated with advertisements
US8051040B2 (en) 2007-06-08 2011-11-01 Ebay Inc. Electronic publication system
US8015181B2 (en) * 2007-06-19 2011-09-06 Conduit, Ltd System for providing enhanced search results on the internet
US20090006179A1 (en) 2007-06-26 2009-01-01 Ebay Inc. Economic optimization for product search relevancy
US7783630B1 (en) * 2007-06-29 2010-08-24 Emc Corporation Tuning of relevancy ranking for federated search
US7783620B1 (en) * 2007-06-29 2010-08-24 Emc Corporation Relevancy scoring using query structure and data structure for federated search
US9178848B1 (en) 2007-07-23 2015-11-03 Google Inc. Identifying affiliated domains
US8200694B1 (en) 2007-07-23 2012-06-12 Google Inc. Identification of implicitly local queries
US20090076927A1 (en) * 2007-08-27 2009-03-19 Google Inc. Distinguishing accessories from products for ranking search results
WO2009032770A2 (en) * 2007-08-29 2009-03-12 Partnet, Inc. Systems and methods for providing a confidence-based ranking algorithm
US8639714B2 (en) * 2007-08-29 2014-01-28 Yahoo! Inc. Integrating sponsored media with user-generated content
US8229970B2 (en) * 2007-08-31 2012-07-24 Microsoft Corporation Efficient storage and retrieval of posting lists
US20090070322A1 (en) * 2007-08-31 2009-03-12 Powerset, Inc. Browsing knowledge on the basis of semantic relations
US8712758B2 (en) 2007-08-31 2014-04-29 Microsoft Corporation Coreference resolution in an ambiguity-sensitive natural language processing system
US8209321B2 (en) * 2007-08-31 2012-06-26 Microsoft Corporation Emphasizing search results according to conceptual meaning
WO2009029923A2 (en) * 2007-08-31 2009-03-05 Powerset, Inc. Emphasizing search results according to conceptual meaning
US8316036B2 (en) * 2007-08-31 2012-11-20 Microsoft Corporation Checkpointing iterators during search
US8280721B2 (en) * 2007-08-31 2012-10-02 Microsoft Corporation Efficiently representing word sense probabilities
US8229730B2 (en) * 2007-08-31 2012-07-24 Microsoft Corporation Indexing role hierarchies for words in a search index
US8463593B2 (en) * 2007-08-31 2013-06-11 Microsoft Corporation Natural language hypernym weighting for word sense disambiguation
US8346756B2 (en) * 2007-08-31 2013-01-01 Microsoft Corporation Calculating valence of expressions within documents for searching a document index
US8868562B2 (en) * 2007-08-31 2014-10-21 Microsoft Corporation Identification of semantic relationships within reported speech
US20090100051A1 (en) * 2007-10-10 2009-04-16 Yahoo! Inc. Differentiated treatment of sponsored search results based on search context
US7840569B2 (en) * 2007-10-18 2010-11-23 Microsoft Corporation Enterprise relevancy ranking using a neural network
US20090106221A1 (en) * 2007-10-18 2009-04-23 Microsoft Corporation Ranking and Providing Search Results Based In Part On A Number Of Click-Through Features
US9348912B2 (en) 2007-10-18 2016-05-24 Microsoft Technology Licensing, Llc Document length as a static relevance feature for ranking search results
US8051076B1 (en) * 2007-12-13 2011-11-01 Google Inc. Demotion of repetitive search results
US20090164268A1 (en) * 2007-12-21 2009-06-25 Hogan Christopher L System and method for advertiser response assessment
US7814108B2 (en) * 2007-12-21 2010-10-12 Microsoft Corporation Search engine platform
KR100903499B1 (en) * 2007-12-27 2009-06-18 엔에이치엔비즈니스플랫폼 주식회사 Method for offering advertisement according to search intention segmentation and system for executing the method
US20090210391A1 (en) * 2008-02-14 2009-08-20 Hall Stephen G Method and system for automated search for, and retrieval and distribution of, information
US20090216710A1 (en) * 2008-02-27 2009-08-27 Yahoo! Inc. Optimizing query rewrites for keyword-based advertising
KR101188939B1 (en) 2008-03-12 2012-10-08 엔에이치엔(주) Method for offering result of search and system for executing the method
US20090248627A1 (en) * 2008-03-27 2009-10-01 Yahoo! Inc. System and method for query substitution for sponsored search
US20090255118A1 (en) * 2008-04-11 2009-10-15 General Electric Company Method of manufacturing mixers
US8812493B2 (en) 2008-04-11 2014-08-19 Microsoft Corporation Search results ranking using editing distance and document information
US20090292674A1 (en) * 2008-05-22 2009-11-26 Yahoo! Inc. Parameterized search context interface
US20090313227A1 (en) * 2008-06-14 2009-12-17 Veoh Networks, Inc. Searching Using Patterns of Usage
US9323832B2 (en) * 2008-06-18 2016-04-26 Ebay Inc. Determining desirability value using sale format of item listing
US8788490B1 (en) 2008-06-27 2014-07-22 Google Inc. Link based locale identification for domains and domain content
US20100010895A1 (en) * 2008-07-08 2010-01-14 Yahoo! Inc. Prediction of a degree of relevance between query rewrites and a search query
US8364529B1 (en) 2008-09-05 2013-01-29 Gere Dev. Applications, LLC Search engine optimization performance valuation
US20100082542A1 (en) * 2008-09-30 2010-04-01 Yahoo! Inc. Comparison of online advertising data consistency
US8095545B2 (en) * 2008-10-14 2012-01-10 Yahoo! Inc. System and methodology for a multi-site search engine
US8316037B1 (en) * 2009-01-30 2012-11-20 Google Inc. Providing remedial search operation based on analysis of user interaction with search results
CN101887436B (en) * 2009-05-12 2013-08-21 阿里巴巴集团控股有限公司 Retrieval method and device
US8577910B1 (en) 2009-05-15 2013-11-05 Google Inc. Selecting relevant languages for query translation
US8577909B1 (en) 2009-05-15 2013-11-05 Google Inc. Query translation using bilingual search refinements
US8572109B1 (en) 2009-05-15 2013-10-29 Google Inc. Query translation quality confidence
US8538957B1 (en) * 2009-06-03 2013-09-17 Google Inc. Validating translations using visual similarity between visual media search results
US8918365B2 (en) 2009-06-19 2014-12-23 Blekko, Inc. Dedicating disks to reading or writing
RU2012101682A (en) 2009-06-19 2013-07-27 БЛЕККО, Инк. SCALABLE CLUSTER DATABASE
CN101930438B (en) * 2009-06-19 2016-08-31 阿里巴巴集团控股有限公司 A kind of Search Results generates method and information search system
US20100324993A1 (en) * 2009-06-19 2010-12-23 Google Inc. Promotional content presentation based on search query
US8600993B1 (en) * 2009-08-26 2013-12-03 Google Inc. Determining resource attributes from site address attributes
US9305089B2 (en) * 2009-12-08 2016-04-05 At&T Intellectual Property I, L.P. Search engine device and methods thereof
US8903812B1 (en) 2010-01-07 2014-12-02 Google Inc. Query independent quality signals
EP2533163A4 (en) 2010-02-04 2015-04-15 Ebay Inc List display on the basis of list activities and related applications
US9953083B2 (en) * 2010-02-16 2018-04-24 Excalibur Ip, Llc System and method for determining an authority rank for real time searching
US10204163B2 (en) * 2010-04-19 2019-02-12 Microsoft Technology Licensing, Llc Active prediction of diverse search intent based upon user browsing behavior
US9317613B2 (en) * 2010-04-21 2016-04-19 Yahoo! Inc. Large scale entity-specific resource classification
US8732177B1 (en) * 2010-04-26 2014-05-20 Jpmorgan Chase Bank, N.A. Ranking online listings
US20110288927A1 (en) * 2010-05-19 2011-11-24 Microsoft Corporation Estimating a discounted cumulative gain
US8738635B2 (en) 2010-06-01 2014-05-27 Microsoft Corporation Detection of junk in search result ranking
US8650084B2 (en) 2010-06-18 2014-02-11 Microsoft Corporation Tool for analysis of advertising auctions
US9020922B2 (en) * 2010-08-10 2015-04-28 Brightedge Technologies, Inc. Search engine optimization at scale
US8504567B2 (en) 2010-08-23 2013-08-06 Yahoo! Inc. Automatically constructing titles
CN102411584B (en) * 2010-09-25 2014-09-17 中国移动通信集团公司 Data searching method and system
US20120158712A1 (en) * 2010-12-16 2012-06-21 Sushrut Karanjkar Inferring Geographic Locations for Entities Appearing in Search Queries
US8706725B2 (en) * 2011-03-14 2014-04-22 Microsoft Corporation Ranking contextual signals for search personalization
US8751591B2 (en) 2011-09-30 2014-06-10 Blackberry Limited Systems and methods of adjusting contact importance for a computing device
US20130117105A1 (en) * 2011-09-30 2013-05-09 Matthew G. Dyor Analyzing and distributing browsing futures in a gesture based user interface
US10438246B1 (en) * 2011-11-21 2019-10-08 Rightquestion, Llc Advertising model
US9495462B2 (en) 2012-01-27 2016-11-15 Microsoft Technology Licensing, Llc Re-ranking search results
CN103309864B (en) * 2012-03-07 2018-10-19 深圳市世纪光速信息技术有限公司 A kind of search result display methods, apparatus and system
US9922120B2 (en) 2012-08-24 2018-03-20 Microsoft Technology Licensing, Llc Online learning of click-through rates on federated search results
CN103678362A (en) * 2012-09-13 2014-03-26 深圳市世纪光速信息技术有限公司 Search method and search system
US10817887B2 (en) 2012-10-04 2020-10-27 Groupon, Inc. Method, apparatus, and computer program product for setting a benchmark conversion rate
US9940635B1 (en) 2012-10-04 2018-04-10 Groupon, Inc. Method, apparatus, and computer program product for calculating a supply based on travel propensity
US10242373B1 (en) 2012-10-04 2019-03-26 Groupon, Inc. Method, apparatus, and computer program product for setting a benchmark conversion rate
US10108974B1 (en) 2012-10-04 2018-10-23 Groupon, Inc. Method, apparatus, and computer program product for providing a dashboard
US10032180B1 (en) 2012-10-04 2018-07-24 Groupon, Inc. Method, apparatus, and computer program product for forecasting demand using real time demand
US10255567B1 (en) 2012-10-04 2019-04-09 Groupon, Inc. Method, apparatus, and computer program product for lead assignment
US9330357B1 (en) 2012-10-04 2016-05-03 Groupon, Inc. Method, apparatus, and computer program product for determining a provider return rate
US9947024B1 (en) 2012-10-04 2018-04-17 Groupon, Inc. Method, apparatus, and computer program product for classifying user search data
US9152680B1 (en) * 2013-02-08 2015-10-06 Educationdynamics Llc Systems and methods for providing leads and appointments
US9536259B2 (en) 2013-03-05 2017-01-03 Google Inc. Entity-based searching with content selection
US20150019287A1 (en) 2013-03-14 2015-01-15 Groupon, Inc. Method, apparatus, and computer program product for providing mobile location based sales lead identification
US10176500B1 (en) * 2013-05-29 2019-01-08 A9.Com, Inc. Content classification based on data recognition
CA2917140A1 (en) * 2013-07-01 2015-01-08 Monster Worldwide, Inc. Social network for employment search
US9779074B2 (en) * 2013-12-20 2017-10-03 International Business Machines Corporation Relevancy of communications about unstructured information
US20150302467A1 (en) * 2014-04-22 2015-10-22 Taptica Ltd System and method for real time selection of an optimal offer out of several competitive offers based on context
CN105095625B (en) * 2014-05-14 2018-12-25 阿里巴巴集团控股有限公司 Clicking rate prediction model method for building up, device and information providing method, system
US9916328B1 (en) 2014-07-11 2018-03-13 Google Llc Providing user assistance from interaction understanding
US11282012B1 (en) 2014-08-04 2022-03-22 Educationdynamics, Llc Graphical user interface including configurable electronic cards
US9965559B2 (en) * 2014-08-21 2018-05-08 Google Llc Providing automatic actions for mobile onscreen content
CN104598611B (en) * 2015-01-29 2018-03-23 百度在线网络技术(北京)有限公司 The method and system being ranked up to search entry
US10489463B2 (en) * 2015-02-12 2019-11-26 Microsoft Technology Licensing, Llc Finding documents describing solutions to computing issues
US10229219B2 (en) * 2015-05-01 2019-03-12 Facebook, Inc. Systems and methods for demotion of content items in a feed
US9298836B1 (en) * 2015-07-07 2016-03-29 Yext, Inc. Suppressing duplicate listings on multiple search engine web sites from a single source system given a synchronized listing is unknown
RU2632148C2 (en) 2015-12-28 2017-10-02 Общество С Ограниченной Ответственностью "Яндекс" System and method of search results rating
US10650475B2 (en) * 2016-05-20 2020-05-12 HomeAway.com, Inc. Hierarchical panel presentation responsive to incremental search interface
US10535005B1 (en) 2016-10-26 2020-01-14 Google Llc Providing contextual actions for mobile onscreen content
GB2559314A (en) 2016-11-15 2018-08-08 Olx Bv Data retrieval system
MY188718A (en) * 2017-02-24 2021-12-25 Mimos Berhad System and method for providing customised content items
US11328005B2 (en) 2018-10-05 2022-05-10 Accenture Global Solutions Limited Machine learning (ML) based expansion of a data set
CN110647998B (en) * 2019-08-12 2022-11-25 北京百度网讯科技有限公司 Method, system, device and storage medium for implementing automatic machine learning
US11681720B2 (en) * 2019-12-11 2023-06-20 Target Brands, Inc. Database searching while maintaining data security
US11302327B2 (en) * 2020-06-22 2022-04-12 Bank Of America Corporation Priori knowledge, canonical data forms, and preliminary entrentropy reduction for IVR

Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4972349A (en) * 1986-12-04 1990-11-20 Kleinberger Paul J Information retrieval system and method
US6269361B1 (en) * 1999-05-28 2001-07-31 Goto.Com System and method for influencing a position on a search result list generated by a computer network search engine
US20020169764A1 (en) * 2001-05-09 2002-11-14 Robert Kincaid Domain specific knowledge-based metasearch system and methods of using
US20030046161A1 (en) * 2001-09-06 2003-03-06 Kamangar Salar Arta Methods and apparatus for ordering advertisements based on performance information and price information
US6622140B1 (en) * 2000-11-15 2003-09-16 Justsystem Corporation Method and apparatus for analyzing affect and emotion in text
US20040093327A1 (en) * 2002-09-24 2004-05-13 Darrell Anderson Serving advertisements based on content
US20060155751A1 (en) * 2004-06-23 2006-07-13 Frank Geshwind System and method for document analysis, processing and information extraction
US20060265399A1 (en) * 2005-05-23 2006-11-23 Giotto De Filippi Keywords auto-segmentation and auto-allocation system to increase search engines income
US7197497B2 (en) * 2003-04-25 2007-03-27 Overture Services, Inc. Method and apparatus for machine learning a document relevance function
US7257570B2 (en) * 2003-11-13 2007-08-14 Yahoo! Inc. Geographical location extraction

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR100311355B1 (en) * 1999-04-07 2001-11-02 맹성현 An Information Retrieval method that Incorporates Different Types of Links
US7110993B2 (en) * 1999-05-28 2006-09-19 Overture Services, Inc. System and method for influencing a position on a search result list generated by a computer network search engine
US8352499B2 (en) * 2003-06-02 2013-01-08 Google Inc. Serving advertisements using user request information and user information
WO2004051418A2 (en) * 2002-11-27 2004-06-17 Findwhat.Com System and method for pay for performance advertising having biddable advertising units utilizing rotating routing to advertiser websites

Patent Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4972349A (en) * 1986-12-04 1990-11-20 Kleinberger Paul J Information retrieval system and method
US6269361B1 (en) * 1999-05-28 2001-07-31 Goto.Com System and method for influencing a position on a search result list generated by a computer network search engine
US6622140B1 (en) * 2000-11-15 2003-09-16 Justsystem Corporation Method and apparatus for analyzing affect and emotion in text
US20020169764A1 (en) * 2001-05-09 2002-11-14 Robert Kincaid Domain specific knowledge-based metasearch system and methods of using
US20030046161A1 (en) * 2001-09-06 2003-03-06 Kamangar Salar Arta Methods and apparatus for ordering advertisements based on performance information and price information
US20040093327A1 (en) * 2002-09-24 2004-05-13 Darrell Anderson Serving advertisements based on content
US7197497B2 (en) * 2003-04-25 2007-03-27 Overture Services, Inc. Method and apparatus for machine learning a document relevance function
US7257570B2 (en) * 2003-11-13 2007-08-14 Yahoo! Inc. Geographical location extraction
US20060155751A1 (en) * 2004-06-23 2006-07-13 Frank Geshwind System and method for document analysis, processing and information extraction
US20060265399A1 (en) * 2005-05-23 2006-11-23 Giotto De Filippi Keywords auto-segmentation and auto-allocation system to increase search engines income

Cited By (25)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20090327278A1 (en) * 2008-06-26 2009-12-31 Baran-Sneh Alex System and method for ranking web content
US9183323B1 (en) 2008-06-27 2015-11-10 Google Inc. Suggesting alternative query phrases in query results
US20100235343A1 (en) * 2009-03-13 2010-09-16 Microsoft Corporation Predicting Interestingness of Questions in Community Question Answering
US20100235311A1 (en) * 2009-03-13 2010-09-16 Microsoft Corporation Question and answer search
US20110072045A1 (en) * 2009-09-23 2011-03-24 Yahoo! Inc. Creating Vertical Search Engines for Individual Search Queries
US9110993B1 (en) * 2010-01-15 2015-08-18 Google Inc. Search query reformulation using result term occurrence count
US20130054556A1 (en) * 2010-05-13 2013-02-28 Adthena Ltd. Method and system for compiling competitive advertiser and keyword information for search engine advertisers
US9064266B2 (en) * 2010-05-13 2015-06-23 Adthena Ltd Method and system for compiling competitive advertiser and keyword information for search engine advertisers
US20120010994A1 (en) * 2010-07-08 2012-01-12 American Express Travel Related Services Company, Inc. Systems and methods for transaction account offerings
US20150347596A1 (en) * 2010-08-20 2015-12-03 Bitvore Corporation Bulletin Board Data Mapping and Presentation
US11599589B2 (en) * 2010-08-20 2023-03-07 Bitvore Corp. Bulletin board data mapping and presentation
US9098569B1 (en) * 2010-12-10 2015-08-04 Amazon Technologies, Inc. Generating suggested search queries
US20150254342A1 (en) * 2011-05-30 2015-09-10 Lei Yu Video dna (vdna) method and system for multi-dimensional content matching
CN105975632A (en) * 2011-06-24 2016-09-28 阿里巴巴集团控股有限公司 Searching method, relation building method and associated device
WO2013003500A1 (en) * 2011-06-28 2013-01-03 Redbox Automated Retail, Llc System and method for searching and browsing for directly and indirectly matching media content
US10216807B2 (en) * 2015-07-07 2019-02-26 Yext, Inc. Suppressing duplicate listings on multiple search engine web sites from a single source system given that a publisher selects a different listing as a synchronized listing
US9443025B1 (en) * 2015-07-07 2016-09-13 Yext, Inc. Suppressing duplicate listings on multiple search engine web sites from a single source system given a known synchronized listing
US9519721B1 (en) * 2015-07-07 2016-12-13 Yext, Inc. Suppressing duplicate listings on multiple search engine web sites from a single source system given that a publisher selects a different listing as a synchronized listing
US20170052961A1 (en) * 2015-07-07 2017-02-23 Yext, Inc. Suppressing duplicate listings on multiple search engine web sites from a single source system given that a publisher selects a different listing as a synchronized listing
US9361388B1 (en) * 2015-07-07 2016-06-07 Yext, Inc. Suppressing duplicate listings on multiple search engine web sites from a single source system given that a publisher selects a different listing as a synchronized listing
US10762156B2 (en) 2015-07-07 2020-09-01 Yext, Inc. Suppressing duplicate listings on multiple search engine web sites from a single source system triggered by a user
US11074263B2 (en) * 2015-07-07 2021-07-27 Yext, Inc. Suppressing duplicate listings on a search provider system
US11775537B2 (en) 2015-07-07 2023-10-03 Yext, Inc. Suppressing duplicate listings on a search provider system using API-based communications
US11775603B2 (en) 2015-07-07 2023-10-03 Yext, Inc. Suppressing a duplicate listing from search results generated by a provider system
CN105516319A (en) * 2015-12-15 2016-04-20 华为技术有限公司 Method and device for managing HTTP messages

Also Published As

Publication number Publication date
EP1839124A4 (en) 2009-12-30
US7698331B2 (en) 2010-04-13
JP2008533553A (en) 2008-08-21
JP4977624B2 (en) 2012-07-18
WO2006078794A2 (en) 2006-07-27
CN101601032A (en) 2009-12-09
KR100932318B1 (en) 2009-12-16
EP1839124A2 (en) 2007-10-03
US20060161534A1 (en) 2006-07-20
WO2006078794A3 (en) 2009-04-16
KR20070092763A (en) 2007-09-13

Similar Documents

Publication Publication Date Title
US7698331B2 (en) Matching and ranking of sponsored search listings incorporating web search technology and web content
US10733250B2 (en) Methods and apparatus for matching relevant content to user intention
US7680786B2 (en) Optimization of targeted advertisements based on user profile information
JP3860036B2 (en) Apparatus and method for identifying related searches in a database search system
US8014997B2 (en) Method of search content enhancement
US8015065B2 (en) Systems and methods for assigning monetary values to search terms
CA2530400C (en) Serving advertisements using a search of advertiser web information
US7831474B2 (en) System and method for associating an unvalued search term with a valued search term
US7805441B2 (en) Vertical search expansion, disambiguation, and optimization of search queries
US20010049674A1 (en) Methods and systems for enabling efficient employment recruiting
US7668808B2 (en) System for modifying queries before presentation to a sponsored search generator or other matching system where modifications improve coverage without a corresponding reduction in relevance
US8849818B1 (en) Searching via user-specified ratings
US20110238662A1 (en) Method and system for searching a wide area network
US8645367B1 (en) Predicting data for document attributes based on aggregated data for repeated URL patterns
US20050065774A1 (en) Method of self enhancement of search results through analysis of system logs
US20070038602A1 (en) Alternative search query processing in a term bidding system
US9529861B2 (en) Method, system, and graphical user interface for improved search result displays via user-specified annotations
US20070239452A1 (en) Targeting of buzz advertising information
WO2005119423A2 (en) System and method for automated mapping of items to documents
WO2009014837A2 (en) System and method to facilitate matching of content to advertising information in a network
US9275145B2 (en) Electronic document retrieval system with links to external documents
US8832097B2 (en) Vertical search expansion, disambiguation, and optimization of search queries

Legal Events

Date Code Title Description
STCB Information on status: application discontinuation

Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION

AS Assignment

Owner name: YAHOO HOLDINGS, INC., CALIFORNIA

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:YAHOO| INC.;REEL/FRAME:042963/0211

Effective date: 20170613

AS Assignment

Owner name: OATH INC., NEW YORK

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:YAHOO HOLDINGS, INC.;REEL/FRAME:045240/0310

Effective date: 20171231