CA2413105C - Recommending search terms using collaborative filtering and web spidering - Google Patents

Abstract

2 In a pay-for-placement search system, the system makes search term recommendations to advertisers managing their accounts in one or more of two ways. A first technique involves looking for good search terms directly on an advertiser's web site. A second technique involves comparing an advertiser to other, similar advertisers and recommending the search terms the other advertisers have chosen. The a first technique is called spidering and the second technique is called collaborative filtering. In the preferred embodiment, the output of the spidering step is used as input to the collaborative filtering step. The final output of search terms from both steps is then interleaved in a natural way.

Description

RI;COMMENDING SEARCH TERMS (JSING COLLABORATIVE
FILTERING ANL) WEB SPIDERING

U_S. Patent Number 6,269,361 discloses a database having accounts for advertisers. Each account contains contact and billing infornlation for an advertiser. In addition, each account contains at least one seareh listing having at least three components: a description, a search term comprising one or morc keywrds, and a bid amount. Tlie advertiser may add, delete, or modify a search listing after logging into his on ccr account via an authentication process.
The advertiser influences a position for a search listing in the advertiser's account by

2 first sele.cting a search tci-ni relevant to the content of'the web site or other information source to be listed. "1'he advertiser enters the search term and the description into a search listini,,. '1'he advertiser inlluences the position for a search listing throtigh a corntinuous orrline con,ipetitive bidding process. `The bidding process occurs when the advertiser e-nters a new bid amount, which is preferably a money amount, for a search listing. The disclosed system then compares this bid amount with all other bid atnounts for the sanne search term, and generates a--ank value for all search listings having that search ter7n. 'I'he rank value generated by the bidding process detennines where the adver-tiset-'s listing will appear on the search results list page that is generated in response to a query of the search terni by a searcher or user on the computer network. A higher bid by an advertiser will result in a higher rank value and a rnore advantageous placement. This system i~, known as a pay-for-placement search engine.

Thus, wllen a user perfonns a search on a pay-for-placement search engrne, the results are conventionally sorted based on how much each advertiser has bid on the user's search term. Because different users will use different words to find the same inforrnation, it is important for an advertiser to bid on a wide variety of' search terms in order to maxiniize the traflic to his site. The better and more extensive an advertiser's list ofsearch terms, the more traffic the advertiser will see.

As an exanlple, a seafood vendor will want to bid not only on the word "seafood", but also on terrns like "fish", "tuna", "halibut", and "fresh fish". A
well thought out list will often contain hundreds of'terms. Good search terrns have three significant properties: they are appropriate to the advertiser's site, they are popular enough that many users are likely to search on them, and they provide good value in terms of the amount the advertiser must bid to get a high ranking in the search results. An advertiser willing to take the tinie to consider all these factors will get good results.

Unfortunately, few advertisers understand how to create a good list of search terms, and right now thcre are only limited tools to help them. "I'he typicai state of the art is the Search "1'crni Suggestion 'I'ool (STS"1') provided by Overture

3 Services, Inc., located at http:/!iiivcntory.overturc.com. S"I'S'I' provides suggestions based on strinIg nritching. (iiven a word, S"I,ST returns a sorted list of all thc search tenns that contnin that word. "I'his list is sorted by how often users have searched for the terrns in the past month. In ihe seafood example, if the advertiscr enters the word "Gsh", his results will includc terrns like "fresh fish,"
"fisti mai-ket," "tropical fish," and "tish bait," but not wor-ds like "tuna"
or "halibut" because they do not contain the stritlg "fish." To create his initial list of search terms, a new advertiser will often enter a few words into STST and then bid on all of the terms that it returns.

] 0 There are three problenis with this approach. First, although STST finds many good terms like "fresti fish" and "fish market," it also finds many bad terms like "fishing," "tropical fish," and "fish bait" that have no relation to the advertiser's site. T'hese create extra work for the search engine provider, since its editorial staff must filter out inappropriate terrns that an advertiser submits.

Second, S'I,S'I' niisses many good terms like "tuna" and "halibut." These result in lost traffic for the advertiser and less revenue for the provider, since every bid helps to drive up the price for search terms and increase the provider's revenue.
Third, it is easy for an advertiser to simply overlook a word that he should enter into STST, thereby rnissing a whole space of search terms that are appropriate for his site. These nzissed terms also result in lost traffic for the advertiser and less revenue for the provider.

An improved version of STST is the GoTo Super Term Finder (STF) which may be found at http://users.idcalab.com/-charlie/advertisers/start.html. This tool keeps track of two lists: an accept list of good words for an advertiser's site, and a reject list of bad words or words that have no relation to the advertiser's site or its eontent. S"1,F displays a sorted list of all the search terms that contain a word in the first list, but not in the second list. As with S"['ST, the result list is sorted by how often uscrs have searched for the terms in the past month. In the seafood example, if the accept list contains the word "fish," and the reject list contains the word "hait," then the output will display ternis like "fresh fish" and "tropical fish"

4 but not "fish bait." An advertiser can use this output to retine his accept and re:ject lists in an iterati\,e process. -Although STF is an improvcment over STST, it still suffers from similar problem. In the seafood exaniple, rnany search terms contain the word "fish"
that are irrelevant to a seafood site "1'he advertiser must still manually identify these and reject each one. Unless the rejected terms share conunon words, the amount of work the advertiser must do with STF is the same as with STST. Both tools also share the weakness of not being abie to identify good search terms like "tuna"
or "halibut". There may be nlany such semantically related terms; they may even I 0 appear commonly on the advertiscr-'s web site. E3ut the burden is still on the advertiser to think of each one_ The problem with STST and STF is that they both look for search terms based on syntactic properties, and they force the advertiser k think of the root words himself. There is a clear need for a better approach, one that takes into accoutit the meaning of words and that can identify them automatically by looking at an advertiser's web site.

A system that finds semantically related ter-rns is Wordtracker, which may be found at http://www.wordtracker.com. Given a search term, Wordtracker recommends new terms in two ways. First, Wordtracker- recomrnends words by looking them up in a thesaurus. Second, Wordtracker recommends words by searching for them using an algorithm called lateral search. Lateral search runs the original search tenn through two popular web search engines. It then downloads the top 200 web page results, extracts all the terms from the KEYWORD and DESCRIPTIUN nleta tags for the pages and returns a list sortcr,l by how frequently each term appcars in these tags.

Wordtracker is only a marginal improvement over STST and STF. In the seafood example, if an advertiser searches for ttie word "fish" he is very likely to see results that include "tuna" and "halibut" but he will still see bad terms like "tropical frsh" and "fish bait" that are not relevant to his sitc. A more specifrc search for "seafood" will get rid of some of these bad terms, but introduce othcrs like "restaurant" and "steak" that come fi-om seaf6od restaurants. Unlike with STF, there is no way to reject such bad terms and refine the search. Nor is there a way to provide a broad list of good terms, since the weh search engines work poorly \~ith more than one search tenn. 'I'hese t\A,, o Iimitations are significant, since it is very rare that an advcrtiser can identify a single search term that exactly describes his site and others like it. Wordtracker- also suffers from the problem that meta keywords are not always indicative of a web site. 'I'here is no editorial revic\\,, so web site designers often include spurious keywords in an attempt to make their pages rnore prominent on search engines. The search engines themselves are also limited, and can return many pages in their list of 200 that ar-e irrelevant to an advertiser's site. Finally, like STS'I' arid STF, Wordtracker still requires an advertiser to think of liis own search terms to get started.

Given these shortcomings, there is a clear need for a better tool, one that can find all of the good search terms for an advertiser's site while getting rid ot` tf1c bad ones.

BRIEF SUMMARY

By way of introduction only, the present embodiments make search terni r-ecommendations in one or more of two ways. A first technique involves looking for good search terms directly on an advertiser's web site. A second technique involves coinparing an advertiser to other, similar advcrtisers and recommending the search terms the other advertisers have chosen. The first technique is called spidering and the second technique is called collaborative.filtering. In the prcfcrred embodiment, the output of the spidering step is used as input to the collaborative filtering step. The final output of search terms from both steps is then interleaved in a natural way.

"I'he foregoing discussion of the preferYed cmbodiments has been provided only by way of introduction. Nothing in this section should be taken as a limitation of the claims, which define the scope of the invention.

5a According to a first broad aspect, the invention provides for a method for recommending search terms in a computer network search apparatus for generating a result list of items representing a match with infonnation entered by a user through an input device connected to the computer network, the search apparatus including a computer system operatively connected to the computer network and a plurality of items stored in a database, each item including information to be communicated to a user and having associated with it at least one search term, an information provider and a bid amount, the method comprising: (a) obtaining a set of potential search terms for acceptance by a new information provider who is adding items to the database, including; receiving from the new information provider a website uniform resource locator (URL); and spidering the website associated with the website URL to obtain search terms for the set of potential search terms; (b) computing correlations between the potential search terms for the new information provider and search terms of other information providers stored in the database; (c) computing an estimated rating for the each potential search term for the new information provider; (d) sorting the potential search terms according to the computed estimated ratings; (e) presenting to the new information provider on an output device the sorted potential search terms;
(f) receiving from the new information provider at an input device an indication of accepted search terms; (g) repeating (b) through (e) until a completion indication is received from the new information provider; and (h) storing the accepted search terms in the database for the new information provider upon receipt of the completion indication.
In some embodiments of the invention spidering the website comprises:
receiving data from pages of the website; recording potential search terms from the data; and determining a quality metric for each potential search term.

In some embodiments of the invention computing an estimated rating comprises: combining a rating based on the computed correlations and a rating based on the quality metric determined for each candidate search term.

5b Some embodiments of the invention further provide for sorting the candidate search terms according to the quality metric; and adding to the set of potential search terms only candidate search terms having a quality metric exceeding a threshold.
In some embodiments of the invention spidering comprises: receiving data from one or more pages of the website; and examining text from the one or more pages for candidate search terms.

In some embodiments of the invention examining text comprises: examining substantially all text from the one or more pages; and examining meta tags from the one or more pages.
In some embodiments of the invention receiving a website URL comprises:
receiving the advertiser's URL as the web site URL.
In some embodiments of the invention receiving a website URL comprises:
receiving the web site URL from the advertiser.
In some embodiments of the invention computing correlations comprises:
assigning ratings to search terms; and computing a correlation between the advertiser and one or more of the other advertisers using the assigned ratings of advertiser search terms.
In some embodiments of the invention computing an estimated rating comprises: predicting a likelihood that a search term will be relevant to the advertiser.
In some embodiments of the invention predicting comprises: determining a quality metric for potential search terms; and predicting relevance of the potential search terms based on the quality metric.
In some embodiments of the invention presenting the sorted potential search terms to the new information provider comprises sending the sorted potential search terms with a web page to the output device.
According to a further broad aspect, the invention provides for a computer network search engine apparatus which includes a database having stored therein a plurality of search listings, each search listing being associated with an information provider, at least one keyword, a money amount and a computer network location and 5c a search engine to identify search listings having a keyword matching a keyword entered by a searcher, to order the identified listings using the money amounts for the respective identified listings, and to generate a result list including at least some of the ordered listings, the apparatus comprising: an account management server including a processing system which is operative in conjunction with program code to recommend potential search terms to a new information provider adding search listings to the database; collaborative filtering code operable in conjunction with the processing system to compute correlations between potential search terms for the new information provider and search terms of other information providers stored in the database and to compute an estimated rating for the each potential search term for the new information provider; sorting code operable in conjunction with the processing system and configured to sort the potential search terms according to the computed estimated ratings; spidering code overable in conjunction with the processing system to find initially accepted search terms in a web site by spidering the web site and to include the initially accepted search terms among the sorted potential search terms; an output device configured to provide the sorted potential search terms to the new information provider for review; and an input device configured to receive from the new information provider an indication of accepted search terms, the accepted search terms being stored in the database in association with the new information provider upon receipt of the indication from the new information provider.

In some embodiments of the invention the spidering code is configured to spider a web site of the new information provider.
In some embodiments of the invention the spidering code is configured to spider a web site specified by the new information provider.
In some embodiments of the invention the spidering code is configured to retrieve pages from the web site of the new information provider, record terms contained in the retrieved pages and score the terms according to a quality metric.

In some embodiments of the invention the spidering code is configured to include terms scoring above a threshold score among the sorted potential search 5d terrns.
According to yet another broad aspect, the invention provides for a method for making search term recommendations to an advertiser in a pay for placement market system in which search listings of advertisers may be searched by users entering search terms, the method comprising: receiving from the advertiser a website uniform resource locator (URL); spidering the website associated with the website URL
to obtain an initial list of search terms to form a set of potential search terms for the advertiser; computing correlations between the set of potential search terms for the advertiser and search terms of other advertisers stored in a database of the pay for placement market system; computing an estimated rating for each potential search term for the advertiser; sorting the potential search terms according to the estimated ratings; providing the sorted potential search terms to the advertiser;
receiving from the advertiser the advertiser's indication of accepted search terms; and storing the accepted search terms in the database for searching by the users.
Some embodiments of the invention further provide for repeating the acts of computing correlations, computing an estimated rating, sorting and providing the potential search terms and receiving an indication of accepted search terms until the advertiser indicated the process is complete.

BRIEF DESCRIPTION OF SEVERAL VIEWS OF THE DRAWINGS
FIG. I is a block diagram illustrating the relationship between a large network and one embodiment of the system and method for generating a pay-for-performance search result of the present invention;
FIG. 2 is a chart of menus, display screens, and input screens used in one embodiment of the present invention;
FIG. 3 is a flow chart illustrating the advertiser user login process performed in one embodiment of the present invention;
FIG. 4 is a flow chart illustrating the administrative user login process 1 ~1 performed in one embodiment of the present invention;
FIG. 5 is a diagram of data for an account record for use with one embodiment of the present invention;
FIG. 6 is a flow chart illustrating a method of adding money to an account record used in one embodiment of the present invention;

15 FIG. 7 illustrates an example of a search result list generated by one embodiment of the present invention;
FIG. 8 is a flow chart illustrating a change bids process used in one embodiment of the present invention;
FIG. 9 illustrates an example of a screen display used in the change bids 20 process of FIG. 8;
FIG. 10 is a flow diagram illustrating a method for recommending search terms to an advertiser on a pay-for-placement search engine;
FIG. I I is a flow diagram illustrating a method for rating search terms by spidering a web site;
25 FIGS. 12-14 are flow diagrams illustrating a method for rating search terms by collaborative filtering;
FIGS. 15-17 are tlow diagrams illustrating computation of the Pearson correlation between two advertisers; and FIGS. 18-20 are flow diagrams illustrating combination of predictions 3(i from spidering and collaborative filtering.

DI-'"I'AILIa_) llES('RIPTION OFTHE l'IZESI;NTI,Y PREFERREU
f;M13ODIMENTS

Methods and syster>>s for generating a pay-for-perfonnance search result deterniined by a site pro-noter, such as an advertiser, over a client/server based computer network systern are disclosed. The following description is presented to enable any person skilled in the art to ntake and use the invention. For purposes of explanation, specific nomenclature is set forth to provide a thorough understanding of the presetit invention. Descriptions of specific applications are provided only as examples. Various modifications to the preferred embodimenis will be readily apparent to those skilled in the art, and the general principles defined herein rnay be applied to otller embodimcnts and applications without departing from the spirit and scope of the invention. 7'hus, the present invent.ior-not intended to be lintited to the embodiments showti, but is to be accorded the widest scope consistent with the principles and features disclosed herein.

Referring now to the dt-awings, FIG. 1 is an exanlple of a distributed systern 10 configured as client/server architecture used in a preferrcd embodirt-ent of the present invention. A "client" is a member ofa class or group that uses the services of another- class or group to which it is not related. In the context of a computer network, such as the Interrtet, a client is a process (i.e_ roughly a program or task) that requests a service which is provided by another process, known as a server program. The client process uses the requested service without having to know any working details about the other server program or the server itself. In networked systenls, a chent process usually runs on a c.omputer tliat accesses shared network resources provided by another computer running a corresponding server process. However, it should also be noted that it is possible for ttie client process and the server process to run on the same cornputer.

A "server" is typically a remote cornputer system that is accessible ovc- a conimunications niedium strch as the Internet. "I'he client process may be activc in a second cornputer system, an(i comrnunicate with the server process over a comrnunic~~ttions -nedium that allows multiple clients to take advantage of the information-gathering capabilities of'thc serve.r. 'I'hus, the server essentially acts as an information provider for a c0mputer network.

Thc block diagram of'1-1(i. I therefore sho%ks a distributed system 10 comprising a plurality ol'client coiYiputers 12, a plurality of advertiser web servers 14, an account managenient server 22, and a search engine web server 24, all o(' which are connected to a network 20. The networ-k 20 will be hereinafter generally referred to as the Internet. Although the system and method of the present invention is specifically useful for the Inter7iet, it should be understoocl that the client computers 12, advertiser web servers 14, account management server 22, and search engine web server 24 may be connected together througlr one of a number of different types of networks. Such networks may include local area networks (LANs), other wide area networks (WANs), and regional network;, accessed over telephone lines, such as cornmercial inforniation services. The client and server processes mav even comprise diffcrent programs executing simultaneously on a single computer.

The elient computers 12 can be conventional personal cornputers (P('s), workstations, or coniputer systems of any other size. Each client 12 typically includes one or more processors, memories, input/output devices, and a network interface, such as a conventional modenl. The advertiser web servers 14, account management server 22, and the search engine web server 24 can be similarly configured. However, advertiser web servers 14, account management server 22, and search engine web server 24 may each include many computers connected by a separate private network. In tact, the network 20 may include hundreds of thousands of individual networks of coniputers.

"I'he client eomputers 12 can execute web browser programs 16, such as the NAVIGATOR, EXPLORER, or MOSAIC browser programs, to locate the web pages or records 30 stored on advertiser server 14. "T`he browser programs 16 allow the users to enter addresses of specific web pages 30 to be retrieved.
'1'hesc addresses are referred to as Uniform Resource Locators, or URI_s. In addition, once a page has been retr-ieved, the browser progranis 16 cari provide access to other pages or records when the. user "clicks" on hyperlinks to other web pages.

O

Such hyperlinks are located w-thin the vveb pages 30 arrd provide an automated way for the user to enter the URL, of another page and to retrieve that page.
'I'he pages can be data records incluciing as content plain textual information, or more cornplex digitally encoded multirnedia content, such as software programs, graphics, audio signals, videos, and so furth.

In a preferred embodiment of the present invention, shown in FIG. 1, client computers 12 communicate through the network 20 with various network inforination providers, including account managernent server 22, search engine server 24, and advertiser servers 14 usir-g the functionality provided by a HyperText Transfer Protocol (IITTP), although other communications protocols, such as FTP, SNMP, TELNE7, and a nuniber of other protocols known in the art, may be used. Preferably, search engine server 24, account management server 22, and advertiser servers 14 a1-e located on the World Wide Web.

As discussed above, at least two types of server are contemplated in a preferred ernboditnent of the present invention. The first server contemplated is an account management server 22 comprising a computer storage medium 32 and a processing system 34. A database 38 is stored on the storage niedium 32 of the account management server 22. "I'he database 38 contains advertiser account infornlation. It will be appreciateci from the description below that the system and method of the present invention may be implemented in software that is stored as executable instructions on a computer storage medium, such as memories or mass storage devices, on the account rnanagement ser-ver 22. Conventional browser programs 16, running on client computers 12, may be used to access advertiser account information stored on account managenlent server 22. Preferably, access to the account management server 22 is accomplished tlu-ough a firewall, not shown, which protects the accouiit management and search result placement programs and the account information from external tarnpering. Additional security may be provided via enhancements to the standard comrnunications protocols such as Secure H"I,TP or the Secure Sockets Layer.

"I'lle secorrd server type contemplated is a search engine .veb server 24. A
search engine program permits network users, upon navigating to the search engine web server URL or sites on other web servers capable of'submitting queries to the search-engine wcb server 24 throuoh their browser program 16, to type keyword queries to identify pages ofintere.st ,rmong the millions of pages available on the World Wide Web. In a preferred embodiment of the present

5 invention, the search engine web server 24 generates a scarch result list that includes, at least in part, relev.rnt entries obtained ironi and formatted by the results of the bidding process conducted by the account managernent server 22_ The search engine web server 24 generates a list of'hypertext links to documents that contain information relevant to search ter-ms entered by the user at the client 10 computer 12. The search engine web server transmits this list, in the form of a web page, to the network user, where it is displayed on the browser 16 runninf; on the client computer 12. A presently preferred cmbodiment of the search enginc web server may be found by navigating to the web page at URL
http://www.goto.com/. In addition, the search result list web page, an example of which is presented in FIG. 7, will bc discussed below in further detail.

Search engine web server 24 is connected to the Internet 20. In a prefcrre(i embodiinent of the present invention, search engine web server 24 includes a search database 40 comprised of search listing records used to generate search results in response to user queries. In addition, search engine web server 24 may also be connected to the account manag,ement server 22. Accoutit management server 22 may also be conrrected to the Internet. The search engine web server and the account management server 22 of the present invention address the different information needs of the user-s located at client computers 12.

For example, one class of users locateci at client computers 12 may be network information providers such as advertising web site promoters or-owners havitig advertiser web pages 30 located on advertiser web servers 14. "I'hese advertising web site promoters, or advertisers, may wish to access account inforrnation residing in storage 32 on account management server 22. An advertising web site promoter may, through the account residing on the account management server 22, participate in a competitive bidding process with other advertisers_ An advertiser mav bid on any nurnber of search terms r-elevant to thc content of'tlic advertiser's web site. In one erribociiment of the present invention, the relevancc of a bidded search term to an adve.r-tiser's web site is determined througli a manual editorial process prior to insertion of the search listing containing the search term and advertiser web site URL into the database 40.
In an alternate embodiment of the present invention, the relevance of a bidded search term in a search listing to the corresponding web site may be evaluated using acomputer program executing at processor 34 of account management server 22, where the computer prograni will evaluate the search term and corresponding web site according to a set ofpredetined editorial rules.

"The higher bids receive more advantageous placement on the search result list page generated by the search engine 24 when a search using the search term bid on by the advertiser is executed. In a preferred cmbodiment of the present.
invention, the amount bid by an advertiser comprises a money amount that is deducted from the account of the advertiser for each tiine the advertiser's web site is accessed via a hyperlink on the search result list page. A searcher "clicks" on the liyperlink with a computer input device to initiate a retrieval request to retrievc the inforYnation associated with the advcrtiser's hyperlink. Preferably, each access or "click" on a scarch result list hyperlink will be redirected to the search engine web server 24 to associate the "click" with the account identifier for an advertiser This redirect action, which is not apparent to the searcher, will access account identification information coded into the search result page before accessing the advertiser's URI, using the search result list hyperlink clicked on by the searcher.
The account identiiication information is recorded in the advertiser's account along with inforniation fi-om the retrieval request as a retrieval request event.

Since the information obtained through this mechanism conclusively matches an account identifier with a URL in a manner not possible using conventional server system logs known in the art, accurate account debit recor-ds will be maintained.
Most preferably, the advertiser's web site description and hyperlink on the search result list page is accompanied by an indication that the advertiser's listing is a paid listing. Most preferably, each paici listing displays a"cost to advertiser,"

which is an amount corresponding to a"priee-per-click" paid by the advertiser f'()a cach refffral to the a(Ivertiser's site through the search result list. A
second elass of'users at client computers 12 may comprise searchers seeking specific information oii the web. Tlie searchers may access, through their browsers 16, a search engine web page 36 residing on web server 24. 7'he search engine web page 36 incltrdes a query box in which a searcher may type a search term comprising one or more keywords. Alternatively, the searcher may query the search engine web server 24 through a query box hyperlinked to the search engine web server 24 and located on a web page stored at a remote web server. When thc searcher has finished entering the search term, the searcher may transmit the quer}, to the search engine web server 24 by clicking on a provided hyperlink. The search engine web server- 24 will then generate a search result list page and transmit this page to the searcher at the client computer 12.

The searcher may click on the hypertext links associated with each listing on the sear-ch results page to access the corresponding web pages. The hypertext links may access web pages anywhere on the Intei7iet, and include paid listingti tca advertiser web pages 18 located on advertiser web servers 14. In a preferred embodiment of the present invention, the search result list also includes non-paid listings that are not placed as a result of advertiser bids and are generated by a conventional World Wide Web search engine, such as the INKTOMI, LYCOS, or YAHOO! search engines. The non-paid hypertext links may also include links manually indexed into the database 40 by an editorial teanl. Most preferably, the non-paid listings follow the paid advertiser listings on the search results page.

FIG. 2 is a diagram showing menus, display screens, and input screens presented to an advertiser accessing the account management server 22 through a conventional browser program 16. "I'he advertiser, upon entering the URL of the account management server 22 into the browser program 16 of FIG. 1, invokes zj login application, discussed below as shown at screen 110 of FIG. 2, rtmning ora the processing system 34 of the server 22. Once the advertiser is logged-in, the processing system 34 provides a mcnu 120 that has a number of options and further services for advertisers. 'l'hese itenis, which will be discussed in more detail below, cause routines to be invoked to either implement the advertiser's request or i-equest further Hiiormation prior to implementing the advertiser's request. In one enibodiment of the present invention, the advertiser may access several options through menu 120, including requesting customer service 130, viewing advertiser policies 140, perforniing account administration tasks 150, adding money to the advertiser's account 160, managing the account's advertising presence on the search engine 170, and viewing activity reports 180. Context specific help 190 may also generally be available at menu 120 and all of the above-mentioned options.

The login procedure of the preferred embodiment of the present inventiov.
is shown in FIGS. 3 atid 4 for two types of user. FIG. 3 shows the login procedures 270 for an advertiser. FIC-. 4 shows the login procedures 290 for an administrator managing and maintaining the system and method of the present invention. As discussed above, the advertisei- or aclministrator at a client computer 12 must first use a browser program at steps 271 or 291 to access the account management server. After the advertiser navigates to the URL of the login pagk to start the login process at step 272 or 292, the processing system 34 of the account management server 22 invokes a login application at steps 274 or 294.
Accordinl., to this application, the processor provides an input screen 110 (FIG. 2) that requests the advertiser's or administrator's user name and password. These items of information are provideci at steps 276 or 296 to a security application knowri in the art for the purpose of authentication., based on the account information stored in a database stored in storage 32 of account management server 22.

According to FIG. 3, after the user has been authenticated as an advertiser, the advertiser is provided with the menu screen 120 of FIG. 2 and limited read/write access privileges only to the corresponding advertiser account, as shown in step 278. The advertiser login event 278 may also be recorded in step 280 in an audit trail data structure as part of the advertiser's account record in thc database. "I'he audit trail is preferably implemented as a series of entries in database 38, where each entry cor7-esponds to an event wherein the advertiser's account record is accessed. Preferably, the audit trail inlormation for an account record may be viewed by the account orvner and other appropriate administrators.

Ilowever, if the user is authenticated as an administrator in step 295 of FIG.
4, the administrator is provided with specified administrative access privileges to i;
all advertiser accounts as shown in step 296. "The administrator login event recorded in step 297 in the audit trail data structure portion of the adlninistrator's account record. This atrdit trail is preferably implemented as a series of entries kqz database 38, where each entry corresponds to an event wherein the administrator's account record is accessed. Most preferably, the administrator's audit trail information may be viewed by the account owner and otller appropriate adnlinistrators.
Furtherrnore, instead of` the general advertiser main menu shown to thc, authenticated advertiser users in step 282, the authenticated administrator is provided in step 298 with access to search the database 38 of advertiser accounts Preferably, a database search interface is provided to the administrator that enables the administrator to select an advertiser account to monitor. For example, the interface rnay include query boxes in which the administrator may enter an account number or username or contact nanle corresponding to an account the administrator wishes to access. When the administrator selects an advertiser20 account to monitor in step 299, the administrator is then brought to the main advertiser page 120 of FIG. 2, which is also seen by the advertisers.

Access to the account inforrnation 32 located on the account management server 22 is restricted to users having an account record on the system, as only those users are provided with a valid login name and password. Password anci login name information is stored along with the user's other account information in the database 38 of the account management. server 22, as shown in FIG. 1.
Account information, including a login user name and password, is entered in the database 38 of FIG. I via a separate online registration process that is outside tkle scope of the present invention.
FIG. 5 is a diagram showing the types of information contained in each advertiser account record 300 in the dalabase. First, an advertiser account record 300 contains a username 302 and a password 304, used 1or onlinc authent.ication as described above. 'Tlie account record also contains contact information 310 (e.g., contact nanle, company name, str-(--,et address, phone, e-mail address).

Contact information 310 is preferably utilized to direct communications to 5 the advertiser when the advertiser has rcquested notification of key advertiser events under the notification option, discussed below. The account record 300 also contains billing inforniation 320 (e.g., current balance, credit card information). The billing information 320 contains data accessed when the advertiser selects the option to add money to the advertiser's account. In addition, 10 certain billing information, such as the current balance, may trigger events requiring notifrcation under the notifrcation option. The audit trail section _325 of an account record 300 contains a list of all events wherein the account record ittib is accessed. Eacii time an account record 300 is accessed or modified, by an administrator or advertiser a short entry describing the account access and/or 15 modification event will be appended to the audit trail section 330 of the administrator or advertiser account that initiated the evetlt. The audit trail inforniation may then be used to help generate a history of transactions made b~the account owner under the account.

The advertisi.ng infornlation section 330 contains information needed to conduct the online bidding process of the present invention, wherein a position is determined for- a web site description and hyperlink within a search result list generated by a search engine. "I'he advertising data 330 for each user account may be organized as zero or rnore subaccounts 340. Each subaccount 340 comprises at least one search listing 344. Each search listing corresponds to a bid on a search term. An advertiser may utilize subaccounts to organize multiple bids on multiple search terins, or to organize bids for multiple web sites.
Subaccounts are also particularly usef'ul for advertisers seeking to track the performance of targeted n-rarket segments. The subaccount superstructure is introduced for thc benefit of the advertisers seeking to organize their advertising efforts, and docs not affect the method ofoperation of the present invention. Alternatively, the advertising information section need not include the added organizational layer of subaccounts, but may simply comprise one or niore search listings.

"The searcli listini-, 344 corresponds to a search term/bid pairing and contains key infornnation to conduct the online competitive bidding process.

Preferably, each search listing comprises the following information: search terrn 352, web site ciescription 354, IJRL 356, bid amount 358, and a title 360. The search term 352 comprises one or more keywords which may be common words in English (or any other language). Eacli keyword in turn comprises a character string. The search t.er7n is the object of the cornpetitive online bidding process.

The advertiser selects a search term to bid on that is relevant to the content of the advertiser's web site. Ideally, the advertiser may select a search term that is targeted to terms likely to be entered by searchers seeking the information on the advertiser's web site, although less conunon search terms may also be selected to ensure comprehensive coverage of relevant search terms for bidding.
The web site description 354 is a short textual description (preferably less than 190 characters) of the content of the advertiser's web site and may be displayed as part of thc advertiser's entry in a search result list. The search listint;
344 may also contain a title 360 of the web site that may be displayed as the hyperlinked heading to the advertiser's entry in a search result list. The URL

contains the Uniform Resourc.e. Locator address of the advertiser's web site.
When the user clicks on the hyperlink provided in the advertiser's search result list eritry, the URL is provided to the browser program. T'he browser program, in turn, accesses the advertiser's web site through the redirection mechanism discussed above. The URL may also be displayed as part of the advertiser's entry in a search result list.

The bid amount 358 pre.ferably is a money amount bid by an advertiser for a listing. This money amount is deducte.d fiom the advertiser's prepaid account or is recorded for advertiser accounts that are invoiced for each time a search is executed by a user on the corresponding search term and the search result list hyperlink is used to refer the scarcher to the advertiser's web site. Finally, a rank value is a value generated dynamically, preferably by the processing system 34 of the account management server 22 shown in FIG. 1, each time an advertiser places a bid or a search enter-s a search query. "I'he rank value of an advertiser's search listing determines the placement location ol'the advertiser's entry in the search result list generated when a search is executed on the corresponding search term.

Preferably, rank value is an ordinal value dete--inincd in a direct relationship to the bid amount 358; the higher the bici amount, the higher the rank value, and the more advantageous the placemcnt location on the search result list. Most preferably, the rank value of I is assigned to the highest bid amount with successively higher ordinal values (e.g., 2, 3, 4, . . .) associated with successively lower ranks and assigned to successively lower bid amounts.

Once logged in, an advertiser can perform a number of straightforward tasks set forth in menu 120 of F1G. 2, irrcluding viewing a list of rules and poli<_;ics for advertisers, and requesting customer service assistance. These itenls cause routines to be invoked to implement the request. For example, when "Customer Service" is selected, an input screen 130 is displayed to allow the advertiser to select the type of customer service requested. In additiorr, forms may be provided on screen 130 so that an advertiser may type a customer comment into a web-based input form.

When "View Advertiser Policies" is selected, a routine will be invoked by processing system 34 of the account management server 22 FIG. 1. As shown in FIG. 2, the routine will display an inforrnational web page 140. The web page sets forth the advertiser policies currently in cffect (e.g., "All search listing descriptions must clearly relate to the search term").

Menu I 20 of FIG. 2 also includes an "Account Administration" selection 150 which allows an advertiser, anlong other things, to view and change the advertiser's contact information and billing information, or update the advertiser's access profile, if arry_ Web-based forms well known in the art and similar to those discussed above are provided for updating account information.

The "Accourit Administration" nienu also includes a selection enabling an advertiser to view the transaction history of the advertiser's account. Under the "View Transaction History" selection, the advertiser may invoke routines to view a listing of past account transactions (e.g_, adding money to account, adding or deleting bidded search terms, or changing a bi(i aniount). Additional routines rnay be implemented to perniit advertisers to display a history of transactions of a specified type, or that occur within a specified time.. 'I'he transaction information may be obtained from the audit trail list 325 of FIG. 5, described above.
Clickablc buttons that may be implernented in software, web-based forins, and/or menus may be provided as known in the art to enable advertisers to specify such limitations.

In addition, the "Ac.count Administration" menu 150 of FIG. 2 includes a selection enabling an advertiser to set notification options. Under this selection, the advertiser may select options that will cause the system to notify the advertisei when certain key events have occufTed. For example, the advertiser may elect to set an option to have the systern send conventional electronic mail messages to the advertiser when the advertiser's account balance has fallen below a specified level.

In this manner, the advertiser -nay receive a "warning" to replenish the account before the account is suspended (meaning the advei-tiser's listings will no longer appear in search result lists). Anotller key event for which the advertiser may wish notification is a change in position of an advertiser's listing in the search result list generated for a particular search term. For example, an advertiser may wish to have the system send a conventional electronic mail message to the advertiser if the advertiser has been outbid by another advertiser for a particular search term (meailitig that the advertiser's listing will appear in a position farther down on the search result list page than previously). When one of the system-specified key events occurs, a database search is triggered for each affected search listing. The system will then execute the appropriate notitication routine in accordance with the notification options specified in the advertiser's account.

Referring back to FIG. 2, a selection also appears in menu 120 that perrnits an advertiser to add money to the advertisei-'s account, so that the advertiser will have funds in their account to pay for referrals to the advertiser's site through the search results page. Preferably, only advertisers with funds in their advertiser's accounts may have their paid listings includecf in any search result lists generated.

Most preferably, advertisers meeting selected business eriteria may elect, in placo of maintaining a positive accotrnt balance at all tirnes, incur account charges regardless of account balance and pay an invoiced amount at regular intervals which reflects the charges inctn-red by actual referrals to the advertiser's site generated by the search cngine. "hhe process that is executed when the "Add Money to Account" selection is invoked is shown in further detail in FIG. 6, beginning at step 602. When the "Add Money to Account" selection is clicked sr3 step 604, a function is invoked which receives data identifying the advertiser and retrieves the advertiser's account from the database. The executing process the.n stores the advertiser's default billing information and displays the default billin,g information for the advei-tiser in step 606. The displayed billing information includes a default amount of money to be added, a default payment type, an(i default instrument inforrnation.
In the preferred embodiment of the present invention, an advertiser may add funds online and substantially in real tirne through the use of a credit card, although the use of other paynient types are certainly well within the scope of the present invention. For example, in an alternate embodiment of the present invention, advertisers may add funds to their account by transferring the desired amount from the advertiser's bank account through an electronic funds verificataon mechanism known in the art such as debit cards, in a manner similar to that set fortti in U.S. Pat. No. 5,724,424 to Giff)rd. In another alternate embodimerrt ot the present invention, advertisers can add funds to their account using conventional paper-based checks. In that case, the additional funds may be updated in the account record database through manual entry. The instrument information includes further details regarding the type of payment. For example, for a credit card, the instrument information may include data on the name of the credit card (e.g., Master('ard, Visa, or American E'xpress), the credit card number, the expiration date of the credit card, and billing information for the credit card (e.g., billing riame and address). In a preferred embodiment of the present inverition, only a partial credit card uumber is displayed to the advertiser for security purposes.

The default values displayed to the advertiser are obtained from a persistent state, e.g., stored in the account database. In an-embodiment of the present inventioil, the stored billing infOrmation values may comprise the values set by thc advertiser the last (e.g. nlost recent) time the process of adding money was 5 invoked and completed for the advertiser's account. The default billing information is displayed to the advertiser in a web-based forni. The advertisci may click on the appropriate text enti-y boxes on the web-based form and makc changes to the default billing information. After the advertiser completes the changes, the advei-tiser may click on a hyperlinked "Submit" button provide(tl ar 10 the form to request that the system update the billing information and current balance in step 608. Once the advertiser has requested an update, a function is invoked by the systeni which validates the bilhng information provided by the advertiser and displays it back to the advertiser for confirmation, as shown in stcp 610. The confirmation billing information is displayed in read-only form and 111ay 15 not be changed by the advertiser.

The validation step functions as follows. If payment is to be debited 11,0n1 an advertiser's external account, payment may be authenticated, authorized and completed using the systeni set forth in U.S. Pat. No. 5,724,424 to Gifford.
However, if the payment type is by credit card, a validating algorithm is invokcEl 20 by the system, which validates the credit card number using a method such as that set forth in U.S. Patent No. 5,836,241 to Stein et al. The validating algorithm also validates the expiration date via a straightforward coniparison with the current system date and time. In addition, the function stores the new values in a temporary instance prior to confirmation by the advertiser.

Once the advertiser ascertains that the displayed data is correct, the advertiser may click on a"Confirm" buttoti provided on the page to indicate that the account should be updated in step 612. In step 612, a function is invoked by the system which adds money to the appropriate account balance, updates the advertiser's bil(ing information, and appends the billing information to the advertiser's payment history. "1'he advertiser's updated billing i:nformation is stored to the persistent state (e.g., the account record database) trom ttie temporary instance. Witllin the function invoked at step 012, a credit card payment function may be invoked by the systerii at step 614. In an alternate ernbodiment of the present invention, other payment functions such as debit card payments may be invoked by defining rnultiple payment types depending on the updated value of thc payment type.

If the payment type is credit card, the user's account is credited immediately at step 616, the user's credit card having already been validated in step 610. A screen sliowing the status of the add money transaction is displayed, showing a transaction number and a new current balance, reflecting the amount added by the just-completed credit card transaction.

In an alternate embodinient of the present invention, after the money has been added to the account, the amount of tnoney added to the account may be allocated between subaccounts the end of the add money process at step 616. If the advertiser has no subaccot.ints, all of the money in the account is a general allocatioti. However, if the advertiser has rnore than one subaccount, the systern will display a confrT-mation and default message prompting the advertiser to "Allocate Money Between Subaccounts".

The menu selection "Allocate Money Between Subaccounts" may be invoked when money is added to the advertiser account after step 616 of FIG.

6, or it may be invoked within the "Account Management" menu 170 shown in FIG. 2 The "Account Management" metuI 170 is accessible trom the Advertiser Main Page 120, as shown in FIG_ 2. "hhis "Allocate Morley Between Subaccounts"

menu selection permits an advcrtiser to allocate current and any pending balances of the advertiser's account among the advertiser's subaccounts_ "11e system will then update the subaccount balances. 1'hc cur-rent halance allocations will be made in real time, while the pending balance allocations will be stored in the persistent state. A routine will he invoked to update the subaccount balances to reflect the petiding balance allocations when the payment for the pending balance is processed_ Automatic notification may be sent to the adver-tiser at that time, if requested_ This intuitive onliiie- account management and allocation permits advertisers to manage their online advertising budget quickly and efficiently.
Advertisers may repienish tlieir- accounts with furids and allocate their budgets, all in one easy web-based session. The computer based implementation eliminates time consuming, high cost niailual entry of the advertiser's accouttt transactions_ The "Allocate Money 13etween Subaccounts" routine begins when an advertiser indicates the intent to allocate money by invoking the appropriate rnenu selection at the execution points indicated above. When the advertiser indicates the intent to allocate, a function is invoked by the system to determine whether there are funds pending in the current balance (i.e., unactivated account credits) that have not yet been allocateci to the advertiser's subaccounts, and displays the balance selection options. In a preferred embodirnent of the present invention, an account instance is created anci a pending current balance account field is set from the persistent state.

If there are no unallocated pending funds, the system may display the current available baiances for the account as a whole as well as for each subaccount. The advertiser then distributes the current available balance between subaccoutits and submits a request to update the balances. A funetion is invoked which calculates and displays the current running total for subaccount balances.

The curretit running total is stored in a temporary variable which is set to the sum of current balances for all subaccounts for the specified advertiser. The function also validates the new available subaccount balances to make sure that the total does not exceed the authorized amount. If the new advertiser-set available subaccount balances does not exceed the authorized amount, a function is invoked which will update all of the subaccount balances in the persistent state and display the update in read-only format.

If there are pending funds in the current account balance, the pending funds must be allocated separately from the available current balance. 7'he pertding funds will then be added into the available current balance when the funds are received. "I'he function must tfierefore prompt the advertiser to choose between allocating pending funds or allocatitig available funds_ "I'he allocating pending funds selection works in much the same manner as the allocating available funds selection outlined above. After the advcrtiser chooses to allocate perlding funds, a routine is invoked to display current pending balances for the account and the subaccounts. The advertiser drstributes the pending subaccount balances between campaigns and subniits a requcst to update the balances. A function is invoked which calculates and displays the current running totals for the pending subaccount balances. This function also validates the new pending subaccount allocations to make sure that the allocations do not exceed any authorized amount.
The current running total of pending allocatiotis is set to the sum of current pending balances for all subaccounts for the advertiser. If the new user-set pending subaccount balances or the total of such balances do not exceed any attthorized amount, the ftrnction will update all of the pending subaccount allocations in the persistent state, e.g. the advertiser's account in the database, and display the update in read-only format.
As indicated above and shown in FIG. 2, a routine displaying the account manageinent menu 170 rnay he invoked from the advertiser main menu 120.
Aside from the "Allocate Money 13etween Subaccounts" selection described above, the remaining selections all use to some extent the search listings present in the advertiset-'s account on the database, and may also affect the advertiser's entry in the search result list. Thus, a further description of the search result list generated by the search engine is needed at this point.

When a remote searcher accesses the search query page on the search engine web server 24 and executes a search request according to the procedure described previously, the search engine web server 24 preferably generates and displays a search result list wltere the "canonicalized" entry in search term field of eacli search listing in the search result list exactly matches the canonicalized search term query entered by the remote searcher_ The canonicalization of search terms used in queries and search listings renioves cornmon irregularities of search terms entered by sear-ches and web site pronioters, such as capital letters and pluralizations, in order to generate relevant results. However, alternate schemes for determining a match between the search term field of-the search listing and the search term quei-y entered by the remote searcher are well within the scope of the present invention. For example, string niatching algorithrns known in the art may be employed to generate matches where the keywords ofthe search listing search term and the search term query have the same root but are not exactly the same (e.g., conlputing vs. computer). Alternatively a thesaurus database of synonyms may be stored at search engine web server 24, so that matches may be generated for a search term having s}iionyms. Localization tnethodologies may also be employed to refine certain searches. For exan-iple, a search for "bakery" or "grocery store" may be limited to those advertisers within a selected city, zip code, or telephone area code. This information may be obtained through a cross-reference of the advertiser account database stored at storage 32 on account management server 22. Finally, internationalization methodologies may be employed to refine searches for users outside the United States. For example, country or language-specific scarch results may be generated, by a cross-reference of the advertiser account database, for example.

An example of a search result list display used in an ernbodiment of the present invention is shown in FIG. 7, which is a display of the first several entries resulting from a search for the term "zip drives". As shown in FIG. 7, a single entry, such as entry 710a in a search result list consists ofa description 720 of the web site, preferably comprising a title and a short textual description, and a hyperlink 730 which, when clicked by a searchei-, directs the searcher's browser to the URL where the described web site is located. The iJRL 740 may also be displayed in the search result list entry 710a, as shown in FIG. 7. The "click through" of a search result item occurs when the remote searcher viewing the search result item display 710 of FIG. 7 selects, or "clicks" on the hyperlink of the search result item display 710. In order for a "click through" to be completed, the searcher's click should be recorded at the account management server and redirected to the advertiser's URL via the redirect mechanism discussed above.

Search result list entries 710a - 7 1 Oh niay also show the rank value of the advertiser's search listing. 'Che rank value is an ordinal value, preferably a number, generated and assigned to the <;earch listing by the processing system of FIG. 1. Preferably, the rank value is assigned through a process, implemetited in software, that establishes an association between the bid amount, the rank, and the search term of a search listing. 'I,he process gathers all search listings that 5 match a particular search term, sorts the search listings in order from highest to lowest bid amount, and assigns a rank value to each search listing in order.
The highest bid amount receives the highest rank value, the next highest bid amount receives the next highest rank value, proceeding to the lowest bid amount, which receives the lowest rank value. Most preferably, the highest rank value is I
with 10 successively increasing ordinal values (e.g., 2, 3, 4, ...) assigned in order of successively decreasing r-ank. 'I'he correlation between rank value and bid atnount is illustrated in FIG. 7, where each of the paid search list entries 710a through 710f display the advertiser's bid amount 750a througli 750f for that entry.
Preferably, if two search listings having the same search term also have the same bid amount, 15 the bid that was received earlier in tirne will be assigned the higher rank value.
Unpaid listings 710g and 71011 do not display a bid amount and are displayed following the lowest-ranked paid listing. Preferably, unpaid listings are displayeci if there are an insufficient nuniber of listings to fill the 40 slots in a search results page. Unpaid listings are generated by a search engine utilizing objective 20 distributed database and text searching algorithnis known in the art. An example of such a search engine niay be operated by Inktomi Corporation. The original searcll query etitered by the reniote searcher is used to generate unpaid listings through the conventionai searclt engine.

As shown in the campaign managemerit menu 170 of FIG. 2, several 25 choices are presented to the advertiser tc- manage search listings. First, in the "Change Bids" selection, the advertiser may change the bid of search listings currcntly in the account. 7'he process invoke<i by the system for the change bids function is shown in FIG. 8_ After the advertiset- indicates the intent to change bids by selecting ttic "Change I3ids" metiu option, the system searches the user's account in the database and displays the search listings for the entire account or a default stibaccount in the advertiser's account, as shown in step 810. Search listings may be grouped into subaccounts defined by the advertiser and may comprise one or more search listings. Only one subaccount may be displayed at a tirne. 'I'lie display should also preferably permit the advertiser to change the subaccount selected, as shown in step 815. The screen display will then show the search listings for the selected subaccount, as indicated in step 820.

An example of screen display shown to the advertiser in step 810 is showti in FIG. 9 and will be discussed below. 'To change bids, the advertiser user may specify new bids for sear-ch terms for- which the advertiser already has an existing bid by entering a new bid amount into the new bid input field for the search terau The advertiser-entered bid changes are displayed to the advertiser at step 820 of FIG. 8 as discussed above. "I'o update the bids for the display page, the advertiser requests, at step 830 of FIG. 8, to update the result of changes. The advertisernaa,,~
transmit such a request to the account rnanagement server by a variety of means, including clicking on a button graphic.

As shown in step 840 of'FIG. 8, upon receiving the request to update the advertiser's bids, the system calculates the new current bid an-iounts for every search listing displayed, the rank values, and the bid amount needed to becorrre the highest ranked search listing niatching the search term field. Preferably, the systein then presents a display of changes at step 850. After the user confirms the changes, the systerrt updates the persistent state by writing the changes to the account in the database.

"The search listing data is displayed in tabular fonnat, with each search listing corresponding to one row of the table 900. "I'he search tertn 902 is displayed in the leftrnost column, followed by the current bid amount 904, and the current rank 906 of the search listing. The current rank is followed by a column entitled "Bid to become I I I " 907, defined as the bid amount needed to become the highest ranked search listing tor the displayed search term. "I,he rightmost column of each row cornprises a new bid input field 908 which is set initially to the current bid amount_ As shown in FIG. 9, the search listings niay be displayed as "subaccounts."
Each subaccount cornprises one search listing group, with multiple subaccounts residing within one advertiser account. Each subaccount may be displayed on a separate display page having a separate page. The adve;r-tiser should preferably be able to change the subaccount being displayed by rnanipulating a pull-down nienu 910 on the display shown in F1G. 9. In addition, search listing groups that cannot be displayed completely in one page may be separated into pages which may be individually viewed by manipulating pull-down tnenu 920. Again, the advertiser should preferably be able to change the page ciisplayed by clicking directly on a pull-down menu 920 located on the display page of F1G. 9. The advertiser may specify a new bid for a displayed searcli listing by entering a new bid amount into the new bid input field 908 for the search listing. `I'o update the result of the advertiser-entered chatiges, the advertiser clicks on button graphic 912 to transrnit an update request to the account nianagement server, which updates the bids as described above.

Many of the other selections listed in the "Account Management" menu 170 of FIG. 2 function as variants of the "Change Bid" funetion described above, For example, if the advertiser selects the "Change Rank Position" option, the advertiser rnay be presented with a display siniilar to the display of FIG. 9 used in the "Change Bid" function. However, in the "Change Rank Position" option, the "New Bid" field would be replaced by a"New Rank" field, in which the advertise.r enters the new desired rank position for a search term. After the advertiser requests that the ranks be updated, the system then calculates a new bid price by any of a variety of algorithms easily available to one skilled in the art. For example, the system may invoke a routine to locate the search listing in the search database having the desired rank/search term combination, retrieve the associated bid amount of said combination, and then calculate a bid amount that is N
cents higher; where N=1, for example. After the systeni calculates the new bid price and presents a read-only confirmation display to the advertiser, the system updates the bid prices and rank values upon receiving approval fronr the advertiser.

'I'he "Modify Listing C'omponent" selection on Account Management menu 170 of F1G_ 2 may also generate a display similar to the format of FIG. 9.
When the advertiser selects the "Modify Listing Component" option, the advertiser may input changes to the URL, title, or description of a search listing via web-based forrns set up for each search listing. Similar to the process discussed above,-the forms for the IJRL, title, and description fields may initially contain the old tIRI,, title and description as default values. After the advertiser enters the desired changes, the advertiser may transmit a request to the system to update the changes.
The system then displays a read-only confirination screen, and then writes the changes to the persistent state (e.g., the user account database) after the advertiser approves the changes.

A process similar to those discussed above may be implemented for changing any other peripheral options related to a search listing; for example., changing the matching options related to a bidded search term. Any recalculations of bids or ranks required by the changes may also be determined in a manner similar to ttie processes discussed above.

In the "Delete Bidded Search Term" option, the system retrieves all of the search listings in the account of the advertiser arid displays the search listings in <iIi organization and a format similar to the display of FIG. 9. Each search listing entry may include, instead of the riew bid field, a check box for the advertiser to click on. The advertiser would then click to place a check (X) mark next to each search term to be deleted, although any other means known in the art for selecting one or more items from a list on a web page may be used. After the advertiser selects all the search listings to be deleted and requests that the system update the changes, the system preferably presents a read-only confirmation of the requested chailges, and updates the advel-tiser's account only after the advertiser approves thc changes. The "deleted" search listings are removed from the search database 30 and will not appear in subsequcnt searclies. However, the search listing will remain as part of the advertiser's account record for billing and account activity monitoring purposes.

In the "Add Bidded Search Term" option, the system provides the advertiser with a display having a number of entry fields corresponding to thc elements of a search listing. The advertiser then enters into each field informatiou corresponding to the respective search listing element, including the search term, the web site URL, the web site title, the web site description, and the bid amount, as well as any otller relevant rnformation. Attcr the advertiser has completed entering the data and has indicated thus to the system, the systeni returns a read-only confirmation screen to the advertiser. 'fhe system then creates a new search listing instance and writes it into the account database and the search database upon receiving approval from the advertiser.

Preferably, the "Account Management" menu 170 of FIG. 2 provides a selection for the advertiser to "Get Suggestions On Bidded Search Term". In this case, the advertiser etiters a biddeci search ternl into a form-driven query box displayed to the advertiscr. The system reads the search term entered by the advertiser and generates a list ofadditional related search terms to assist the advertiser in locating search terrns relevant to the content of the advertiser's web site. Preferably, the additional search terms are generated using methods such as a string matching algorithni applied to a database of bidded search terms and/or a thesaurus database implemented in software. The advertiser may select search terms to bid on from the list generated by the system. In that case, the system displays to the advertisers the entry fielcis described above for the "Add Bidded Search Terrn" selection, with a forni for entering a search listing for each search term selected. Preferably, the selected search term is inserted as a default value into the form for each search listing. Default values for the other search listing components may also be inserted into the forms if desired. Thus, in one embodiment, the disclosed system receives a list of search terms associated with an advertiser on the database search system, determines candidate search terms based on search terrns of other advertisers on the database search system, and recominends the additional searcll terms from aniong the candidate search ter7ns.
In another embodiment, the disclosed systeni provides receiving a search term of an advertiser. in response to the rcceived search term, generating a list of additional related search terms, and receiving adver-tiser selected search terms from the list of additional related search terms.
The "Accourrt Management" nierru 170 of FIG. 2 also preferably provides advertisers with a "Project Expcnses" selection. In this selection, the advertiser specifies a search listing or suhaccount for which the advertiser would like to predict a "daily run rate" and "days remaining to expiration." The system calculates the projections based on a cost projection algorithrn, and displays the predictions to the advertiser oii a read-only screen. Thc predictions may be 5 calculated using a nunlber of different algorithms known in the art.
However, since the cost of a search listing is calculated by multiplying the bid amount by the total nuinber of clicks received by the search listing at that bid amount during a specified time period, every cost projection algorithm must generally deterrnine an estimated number of clicks per month (or other specified time period) for a search 10 listing. The clicks on a search listing may be tracked via implementation of a software counting nlechanism as is well known in the art. Clicks for all search listings may be tracked over time, this data may be used to generate estimated numbers of clicks per month overall, and for individual search terms. For a particular- search terrn, an estimated nurnber of searches per day is determined and 15 is multiplied by the cost of a click. This product is then multiplied by a ratio of the average number of clicks over the average number of' impressions for the rank of the search listing in question to obtain a daily run rate. The current balance may be divided by the daily run rate to obtain a projected number of days to exhaustion or "expiration" of account funds.

20 One embodiment of the present invention bases the cost projection algorithm on a simple predictor niodel that assunies that every search term performs in a similar- fashion. This model assumes that the rank of the advertiser's search listing will i-emain constant and not fluctuate throughout the month.
This algorithm has the advantages of being simple to implement and fast to calculate.

25 The predictor rnodel is based on the fact that the click through rate, e.g.
the total number of clicks, or referrals, ior a particular searcher listitig, is considered to be a function of the rank of the search listing. 'l'he niodel therefore assumes that the usage curve of each search term, that is, the curve that r-esult when the number of' clicks on a search listing is plotted against the rank of the search listing, is similar 30 to the usage curve for all search terms. Thus, known values extrapolated over time for the sum of all clicks (or all search ternis, the sum of all clicks at a given rank for all search terms, and the sum of all clicks for the selected search term may be employed in a sinlple proportion to determine the total of all clicks for the given rank for the selected search term. "hhe estimated daily total of all clicks for the selected search term at the selected rank is then multiplied by the advertiser's current bid amount for the search tcrm at that rank to determine a daily expense projection. In addition, if particular search ternis or classes of search terms are known to differ markedly froni the general pattern, correction values specific to the search term, advertiser, or other parameter may be introduced to fine-tune the projected cost estimate.

Finally, the "Account Management" menu 170 of FIG. 2 provides several selections to view information related to the advertiser's campaigns. The "View Subaccount Information" selection displays read-only information related to the selected subaccount. The "View Search Term List" selection displays the list of the advertiser's selected search terms along with the corresponding URLs, bid price, and rank, witli the search tertns preferably grouped by subaccount. The advertiser may also view current top bids for a set of search terms selected froni a list of search terms from a read-only display generated by the system upon receiving the requested search terms from the advertiser.

For an advertiser who requires a more comprehensive report of search listing activity, the "View Report" option may be selected from the Advertiser Main Page 120 of FIG. 2. In an ernbodiment of the present invention, the "View Report" options generate reports comprehensive for up to one year preceding the current date. For example, daily reports are available for the each of the immediately preceding 7 days, weekly reports for the preceding four weeks, monthly reports for the preceding twelve months, and quarterly reports for the last four quarters_ Additional reports may also be rnade available depending on advertiser interest. Other predefined report types may include activity tracked during the following time periods: Since Inception of the Account, Year "To Date, Yearly, Quarter To Date, Month To Date, and Week to Date. Report Categories may include a I)etail Report, viewable by Advei-tiser Account, by Search Listirig, and by URL, and a Surnrnary Report, viewable by Advertiser Account and by Subacc.or_rnt. "I'Iie rcport5 rnay include identification data such as advertiser account and subaccOunt nanic:, the datc,; covered by the report and the type of report. In addition, the reports, may rnclude key search listing account data such as current balance, pending cur-rerrt balance, averigc. daily account debit, and run rate Furthermore, the reports may .rlso inclr.rde key data, such as: search terms, I. IRI,s bids, current ranks, and number of clicks, nunrber of searches done for the search terni, number of impressions (times tllat the search listing appeared in a search result list), and click through rate (defined as Number of Clicks/Number of Impressions). 13referably, the report is available in at least 1-I"TML view optiorrs l:or-] 0 viewing via a browser program, printing, or downloading. Note, however, that other view options may be made available, such as Adobc Acrobat, PostScript, ASCII text, spreadsheet interchange formats (e.g., CSV, tab-delimited), and otl7cr well-known forrnats.

When the advertiser- has selected the "View Report" option, the system invokes a funetion whicli displays a list of available report types, dates, categories, and view options. 1'he system preferably creates a report instance with the following fields, all of wtlich are initially set to null: report type, report date, report category, and view option. Oncc the advertiser has def ned the pararneters described above, the system invokes a function to generate the requested report, based on the advertiser-set parameters, and to display the report, based on the view option parameter.

Finally, a preferred embodiment of the present invention implements an option for context specific help that the advertiser niay rcquest at any time the advertiser is logged in. "I,he help option may be implemented as a small icon or button located on the system generated display page. The advertiser may click on the icon or button graphic on the display page to request help, upon which the system generates and displays a help page keyed to the function of the particr.rl:lr display the user is viewing. The help may be irnplemented as separate display pages, a searchable index, dialog boxes, or by any other rnethods well known un the art.

FIGS. 10-20 illustrate particular ernbodinicnts of a tnethod and apparatus for making searc.h term recommendations to a weh site promoter or advertiser in a pay for placement markc:t systcm such as that descrihed above in conjunction with FIGS. 1-9. Uisclosed ernbodirTtents provide a ntethod Cor a database search system. '1'he rnethod includes maintaining a database- of search listings includins!, associated search terms, reeeiving a list of search terms associated with an advertiser, recommetiding additiotlal scarch terms to the advertiser. Other disclosed embodiments provide a data base oper'ating method for a database search system which stores advertiset- search listings including advertiser selected seazcll tertns. The method includes spidering a specified web site to obtain an initial lisi of advertiser search ternis for an advertiser. 'The ntetllod further includes filterini-, the initial list of advertiser search terms using search terrns of other advertisers ;3rW
storing in a search listing database search listings for the advertiser, the search listings formed with the filtered search terms.

Disclosed ernbodiments also include a database search system which includes a database of search tcrms in which each search term is associated wrtli one or rnore advertisers. Progtarn code is cotifigured to recomrnend additional search terms for an adver-tiset- based on search terrrrs in tlte database.
Still furthtt, embodiments disclosed herein provide a niethod for a database search system which includes receiving a search term of an advertiser and, in response, generating a list of additional related search terms. The method then includes receiving advertiser selected search terms frorn thc list of additional related searclt terms.
In the enibodiments shown here, spidering and collaborative filtering are used to identify possible search terms to recomnlenci to an advertiser. The following introduction first describes the individual techniques of spidering and collaborative filtering, and tlien shows how the two may be cotnbined.
Spidering is a sirnple technology for dowtiloading a web site rooted at it uniform resource locator (URL). A program downloads the home page given by the IJRt., then scans it for- hyperlinks to other pages and downloads them.
"I'he spidcring process contintres until the progrant reaches a predefined link depth, downloads a predetermined nuniher of pages, or- reaches some other stopping criterion. The order in which pa2es ai-e downloaded ciin be eithcr breadth-first or deptli-first. In breadth-first spidering, Ihe prograni adds new URI.'s to the end of' its list of pages to download; in depth-first spidering, it adds thetn to the beginning. These algoritlurts ,ire straightforward and well known to engineers skilled in the state of the art. I"urther information abotit thesc techniqi.ies may be found by consulting Cho, Molina, and Page, "I_?flicicnt Crawling tilrough tJR1, Ordering", avaiiable t:rom Rescarchlndex, http://citeseer.nj.nec.conl and Nilssoti.
Principles ofArtiJicial Intelligence, ISBN 0934613109_ Sorne embodiments (iescribed hereiri use spidering to find search terms thar appear directly on an advertiser's web site. Startirig at the root of the advertise.r"!a site, the method and systern in accordance with the present embodirnents downloads pages breadth first and scans thcm for search ternls. It records every term it finds that the provider's database indicates has bcen searched in the past montll. As an example, if the text on a page inc.lucies the plirase "tropical fish store," then the program will find the six terms "tropical," "fish," "store,"
"tropi(.:a1 fish," "fish store," and "tropical tish store." 'I'he program scores these terms using a quality metric, adding the ones that are above a particular threshold to its list of recommendations. In the preferred embodiment the quality metric considers two factors: how cotnmon a search term is on the World Wide Web, and how often users searcil for it. When the program has accuniulated enough recommendatiuns, it sorts thein by either their quality or by the number of times they have occurred in the downloaded pages and retur-ns the list.
"I'he spidering cornponent of the current cmbodiments differs from previous tools in three important ways. First, it looks directly at the pages in an advertiser's web site, as opposed to downloading other pages that are not in the advertiser's web site, and that might he coinpletely unrelated. Second, it looks at all of the text on a web page, as opposed to just the words in the DESCRIPTION and KEYWORU tags. Third, it uses its quality rnetric to eliminate poor search te:rrns without cvcr showing them to thc advertiser.

Collaborative tiltering is a te,chnology for tnaking recommendations based on user siniilarity. As an example, a company like Amazon.com uses collaborative tiitering to ntake book reconimendattons- Unce a customer has bought several books using the otl lirne service available at www.amazon.conn, Amazon_coiTl recommends new books by coniparing the customer to others in its database_ When it finds another c.ustotner that has niade nlany of the same purchases, it recommends the choices of each to the customer. '1'he current embodiments extend this idea to recomniending search terms for advertisers on a pay-for-placement. search engine.

For example, suppose a typical provider has a database of 50,000 advertisers. A portion of that database might look like this:

Fish Tuna Halibut Bait Wot-nls Cars Joe's Fish X X X - - -Rick's Car Shop - - - - - X
Bill's "I'ackle X - - X X - !
-- -- --- -- - ------ - - - -- -- -~
An X in the table indicatcs that an advertiser has bid on a term. In the seafood example, an advertiser that is i.nitially interested in "fish" is similar to both Joe and Bill, and the progranl will recommend "tuna," "halibut," "bait," and "worrns."
[f the advertiser refines his search terms to include "tUna" but exclude "bait,"
then he is no longer similar to Bill, and the program will stop recornmending "worms."
Like STF, the current invention allows the advertiser to iteratively accept and reject wot-ds until he is satisfied with the list of recotnmendations.

Quatititative.ly, collaborative filtering computes the Pearson corrclation betwcen the tiew advertiser and all of the existing advertisers. `I'o calculate thts correlation, a numeric rating is assigned to each entt-y in the advertiser/term table In one possible assignnient, the highest rating is 5, indicating that a tcnn is a perfect description of an advertiser's site, an<i the lowest rating is 0, indicating that .36 a term is irrelevant. In the prefcrred ernbodinnent, an advertiser gets a rating of 5 for every terni he has bid on and a rating oi'l1NKNOWN for- every other te.r-rn.
fhe new advert-ser gets a rating ot' ti f0r ternrs the advertiser has accepted, a 1 for terms lie has rejected, and a 2 for every othcr term. 1'he Pearson correlation between the ne\\ advertiser and an existing advertiser is then ~) - - --------------a CT (T
i u In this formula, n is the new a(ivertiser, is lris correlation to advertiser a, r, is the rating he assigrrs to ten t, and r and 6õ are the mean and standard deviation of his ratings. '1'he terms with the a subscripts have thC cor-responding meaningr, for the existing advertiser. The sum is taken over all search terms. A rating of' UNKNOWN is replace(i by thc mean of'an advertiser's ratings, so any term witli an UNKNOWN cancels out oi'the equation. Correlations range between -1 and 1, with zero being no correlation and a positive correlation indicating that two advertisers have similar ratings. This formula is well known fi-om statistics anil familiar to engineers skilled in the state of the art. Further details may be fourrd by consulting Wadswortli [cd], "I'hc, Ilandhook of Statisticnl Methods for EngineY.r.s and Scientists, ISBN 007067678X.

Once the collaborative filter has computed the correlation between the new advertiser and the existing advertisers, it predicts how likely it is that each term is a good search te.rrrr for the new advertiser. It does this by computing the average rating of each term, where an advertiser's contribution to the average is determined by its correlation to the new advertiser. An advertiser that has a high correlation receives full weight; an advertiser that has a low correlation receives little weight; an advertiser that has rcro correlation receives no weight. One formula for this pr-ediction is r )Pr, E 1; +
!'õ
la In this formula, ir is the ncw aclvc.rtiser and e, is his estimated rating for term t.

l hc reniaining ternns h<.rve the sa111c: nreanirig as in the hrcvious fi)rmula. The sunt is taken ovcr all etisting aclvertisers. ,An UNKNOWN rating is again replaced by the mean of'an advertiser's known ratings, so it cancels out of the equation.
`I'he formula is a weighted sum that estimates ratings on the same 0 to 5 scale as the original ratings. A terrn receives a high estimate if all thc highly correlated advertisers rate it highly_ The output of the collaborative filter is the list of search terrns sorted by their estimated ratings.

These fortnuias provide a straightforwarci technique for calculating ratings based on sirnilar-ity. There are many similar foi-mulas and variations. For example, when making predictions it is usually better not to take a weighted average over all advertisers, but just over the 10-20 most highly correlated ones.
There are also tecliniques for iniproving the efhciency of the calculations, or for doing collaborative filtering without using correlations or distance metrics.
"I'hese variations are rcadily found in the literature on collaborative filtering, and the current embodiments are not constrained to any one of them. More details oti ttie advantages and disadvantages ofdifterent collaborative filtering algorithms can bc fouiid at the GroupLens web site http_//www.cs_umn.edu/Research/GroupLens.

Given the core building blocks of'spidering and collaborative filtering, the complete system and method accot-ding to one present embodiment works as follows: starting with an initial list of accepted and rejected search terms, run the collaborative filtering algorithni, allow the advertiser to accept and reject new terms, and then rerun the collaborative tiltering. End this process when the advertiser is satisfied with his list of accepted tertns_ The technique gets its initial list of accepted terms iti one of three ways: eithcr directly from the advertiser, or from an existing advertiser's bid list, or ti-orn the list of recommendations returned by running the web spider on the new advertiser's web site. This last inethod is the preferrcd embodiment. When using thc web spider, the search terms that it recommends receive initial rati.ngs that vary on a linear scale from 4.9 down to 2.1. Wlienever the inventiort displays recomrnendations to the advertiser, it interleaves the original spider reconiniendations with the output of the collabor.itive tiltering, since the recommendations fiom the two teehniques are often complementarv. The interleaving f'OrmuLi weights the recommendations of ttie web spider less and less as the advertiser accepts and rejects more terms.

In typical use, a new advertiser will start with the [)RL of his web site and go through 3-5 iterations of accepting and rejecting tetins. As long as his web site is sitiular to those of existing advcrtisers, the system will quickly identify thern and niake high quality recommendations. The reconimendations will be good even if no single advertiser is a perfcct match, since the weighted sum allows the system to combine recotnmendations from rnany advertisers. And when there is tio advet-tiser that is similai- to the new advertiset-, the web spider still makes goo(l recommendations by finding search terms directlv on the advertiser's web site.
In contrast to the existing state of the art, the current embodiments provide excellcni coverage of good search terrns while eliininating bad ones.

Referring now to the drawing, FIG. 10 is a ilow diagram illustrating a method for recommendittg seai-ch terms to an advertiser on a pay-for-placement search engine. The mettiod may be implemented on a server or other data processing device associated with the pay-for placement search engine. The method may be embodied as software code opet-able on the data processing device in conjunction with stored data of a database or other storage element. An advertiser accesses the server to run the program using any suitable device such as a reniotely-located personal computer linked to the server over the internet.
One exemplary embodiment of a suitable system is sliowtl above in conjunction with FIG. 1. The method begins at block 1000.
In block 1002, the system pt-onipts the advertiser to choose an input method to create the irtitial list ot' accepted search tet-ms. 'I'his list may come from direct advertiser input, from a uniform resource locator (URL) specified by the advertiser, or froin a preexisting advertiser specitied by the advertiser.
After prompting thc advertiser for the method he wants to use, the program follows ont=
of the three patlis shown in FIG. 10.
If the advertiser chooses to specify ttle initial list of search terms directly, at block 1004 the. ternis are received froni thc advertiser. In one exenlplary ernbodinient, the pt-ogram displays a text box in whicli the advertiser can enter a conrma-separated list of initial terms_ It'the advertiscr chooses to specify a t1RL
as the source of-the initial list of search ternts, thc advertiser is then prompted to enter a web site URL. 1'he systent runs a spider algorithni to extract search ternts from that site, block 1008. An exemplary embodinlent of such a spider algorithrt, will be described below in conjunction with FIGS. 11-13. If the advertiser chooses to specify a preexisting advertiser as the source of the initial list of seatch ternls, at block 1010 identiticatioti information for the preexisting advertiser is received from the advertiser. "I'he new advertiser picks an existing advertiser ae-z9 ( 0 the program sets the list of initially accepted terms to be the list of terms that advertiser has bid on, block 1012.

The method now enters its nlain loop, including blocks 1014, 1016, (01 K, 1020. During each iteration, it t-uns the collaborative filtering algorithm, block.
1016, displays a sorted list of reconimended search tenns, and allows the advertiser to accept and reject terms, block 1018. In the exemplary embodiment, a web page including the recommended search tertns is sent to the advertiser, providirig a user interface for advertiser interaction with the system. The advet-tiser accepts and rejects tet-ms by clicking ori suitable check boxes next tot the terms. When he is done -naking his changes, he clicks a button to transmit the page of data to the server and rerun the collaborative filtering algorithm.
The advertiser can continue through as niany iterations as he likes, repeating the loop, block 1014, until he is satisfied with the terms he has accepted. He then clicks a final button to exit the loop, block 1020, and store or print out his selected search terms. Preferably, communication with the advertiser is over the internet usirig a suitable data transfer pt-otocol such as TCP/IP. Other data communication channels may be substituted. The method ends at block 1022.

F1C.i. I I is a flow diagram showing a t-itethod for performing a spidering algoritltni. "I'his algori:thtn may be called, tor example, at block 1008 of FIG 1{) 'I'he method begins at block l 100. The procedure is called passing a URL that is the root of an advet-tiser's web site. Starting with this URL, the procedure enters a loop including blocks 1102, 1104, 1106, 1108. "I'he procedure downloads pages 4t) using a breadth-first spidering algorithrn. For cach page that it downloads, block 1 104, it scans the text on the page to find cvery phrase that has been used as a sear-ch term in the past nionth. In the preferred embodiment, this scanning is done by constructing a firiite state ntachine that recognizes the regulat-expression s, .. .i,,, where each .r; is a valid search term. 'I'he prograni scans a page one character at a time using tllis statc machi.ne, arid ernits eacli searcli term as it finds it. Because the state machine only depends on the current set of valid search ternls, the preferred embodiment only constructs it at regular intervals when the database of terms that users have searched changes. Algorithms for constructing such a finite state rnachine are readily available in the literature and appear in comnion search utilities such as grep, as described in Aho and Hopcraft, Thc Design of C'oniputer Algorithms, IS13N 0:201000296. '1'hey are well known to practitioners of ordinary skill in the art of computer systetn design.

Each time the spider finds a new term on a page, it adds it to the list of terms it has found on the web site, block 1106. It keeps track of frow many times it has seen each term in an array ('O1 JNT["C1. The loop repeats at block 1 108.
'The downloading and scanning process ends when the spider has found 1000 terms as indicated by the looping control of block 1102. Other tliresholds or looping control techniques may be used. The looping operation of FIG. 1 I i5 exemplary only.

The next step is to filter out bad tenns. This is performed in a loop including block 11 10, 1112, 1114. 13ad is a subjective measure, and there are many possible nletrics that an inlplenientation might use. In the preferred embodiment the quality metric depends oii two quantities: the frequency with whi.ch a ter-m appears in ciocurnents on the World Wide Web, and the fi-equency with which users search for it. 'Che quality metric is evaluated at block 1112. 'I'he tnethoci fincis a term's frequency on the World Wide Web by querying a search engine that returns the nutnber of docurnents containing the term. It finds the frequency with which users search for it by looking up that infonnation in the provider's database. The quality measure eniployed in the illustrated ernbodiment is the log of the ratio of thesc two nuntbers, as shown in block 1 1 12 of h IG. 11, To achieve a high quality ratin', a term must be a popular one, for people to search on, but not so comnion in web doc.iiments that i1 is useless as a search terni.
13ecause quality mcasures only change slowly, the preferred ernbodiment only calculates them at periodic inte.rvals and caches the results. Other quality measures may be substituted.

Once the method has calculated the quality of the 1000 terms it has found, the loop is exited at block 1114 and the metliod discards or thr-ows out all the terms that fall below a predetermined quality threshold, block 1116. This threshold may be variable, changing over time, because it depends on how many pages are indexed on the Worl(l Wide Web and how many users are conducting searches using the provider's search engine. In the preferred embodiment, the progranl automatically calibrates the thresllold by looking tip the quality of known terms that are on the borderline of being good scarch terms. lt sets the threshold to the average quality of these terms. 'fhe exact list of'terms depends on the search engine provider and is not constrained by the particular embodiment.

The final step in the spidering algorithm is to sort the terms that are above the quality ihreshold by how otlen they occur in the pages the spider has downloaded and scanned, at block 1116. These counts are stored in the COUNT["Tj array. "The sorted list is the output of the spider algorithm. In a typical embodiment the quality filter discards about 80% of the terms, and the algorithm returns about 200 terrns. The spidering method ends at block 1118.
FIG. 12 is a flow diagrarn showing one nicthod for performing the collaborative filtering algorithni. The tnethod begins at block 1200. At block 1202 and block 1204, rating values for the new advertiser and existing advertisers are initialized. Embodiments for performing these operations are described below in conjunction with FIGS. 13 and 14. At block 1206, control enters a loop including blocks 1206, 1208 and 1210. In this loop, the rnethod processes the search terms selected by the collaborative filtering algorithm of FIG. 1 1 and calculates the new advertiser--'s estimated rating for each term, block 1208.
C)ne embodiment for this rating prediction method is described below in conjunction with FIGS. 18-20. After proccssing all search terms, the Ioop is exited at block 1210. At the erld of'tlie alf,orithm ternis are son-ted by their predicted r-atings., block 1212. "flIe. 111ethod returns the linal list as its ranked list cif recornmendations and tlicn ends at block 1214.
In tliis algorithm and in following algorithms, there are many effrcienc_y optimizations that an implementation might inclucic. For example, it might retr-rrr;
only the top 100 search terms, ratliet- than the entire list, or it might cache cotnputational results to avoid repeating work. All of these optimizations will be readily apparent to practitioners ordinarily skilled in the art of conlputing system design, and the embodiments shown here do not depend on particular optimizatiotis an implernentation Lrses.
FIG. 13 is a flow diagranl illustrating a prefcrred algorithm for initialiling the rating values of existing aravertiscrs. "I'lie algorithm is a loop over every advertiser/search term pair. For each pair, the prograrn scts the rating to 5 if the advertiser has bid on the ter-m, and to UNKNOWN otherwise. Ratings are stored in the V[A][T] array so that othet- parts of the program can access theni.
'I,he nietliod begins at biock 1300_ An advertiser-processing loop is entered at block 1302 using an advertiser variable A. A term-processing loop is entered at block 1304 using a terni variable T. At block 1306, the method determines if the advertiser associated with the adver-tiser variable A has bid on the term associated with the variable T. If not, at block 1308, the rating V[A][T] is set to a value of UNKNOWN in an ar-ray of rating values. If the advertiser has bid on the tenn, at block 1310 the array enti-y V[A][T] is set to 5, which is an arbitrarily chosen value.
At block 1312, the term variable is incremented or otherwise changed to select a next term. Control rernains in the loop including blocks 1304, 1306, 1308, 1310, 1312 until all search terrns have been proccssed for the variable associated with variable A. T'11en at block 1314, the advertiser variable A is incremented or otherwise changed and looping proceeds through scarch terms for the newly selected advertiser. After all advertisers have been processed for all search tertns.
the method ends at block 1316.

FIG. 14 is a flow diagrarn showing, a preferred algorithm for initializing the rating values of the- new adve.rtiser. Thc algorithin is a loop ovei- every search ternl. For each term, the pr-of;ranr sets thc rating to 5 if the ncw advertiser has accepted the terni, and to I if he has rejected it. lf he has done neither, and the spider has recommendeci the tcrni, then the program sets thc rating to the spider's estimated rating. If nonc of these three cases hold, the prograrn sets the rating value to 2.

The method begiris at block 1400. At block 1402, a loop is entered using a term variable T as the looping variable. At block 1404, it is determined if the advertiser has accepted the tenn associated with the variable T for the advertiser's search terrrms. If so, at block 1406. the rating V[A][T] for the advertiser and term is set to a value of 5 in the array of ratings. Control proceeds to block 1418 to select a next term for the looping variable T. l f the adver-tiser has not accepted the current search term T, at block 1408 it is determined if the advertiser has rejected it. If so, at block 1410, the rating V[A]I"I"] for the advertiser and term is set to a value of I and control proceed:; to block 1418 to increment the looping variable_ If the advertiser has not rejected the term T, at block 1412 it is determined if the spidering algorithm has recommended the term associated with the variable T.
If so, at block 1414, the rating VjA]["I'] for the advertiser and term is set to a valuc equal to the rating established by the spidering algorithm. Otherwise, the rating V[A][T] for the advertiser an(i terrn is set to a value of 2. Control then proceeds to block 1418 to increinent the looping variable. After all terms have been processed, the method cnds at block 1420.

FIG. 15 is a flow diagram illustrating an algorithm for calculating the Pearson correlation between two advertisers. "I'his algorithm is a loop over every search ter-m. For each ter-m, the program accumulates values that allow it to calculate the Pearson correlation torniula.

rl) ( n,l a/ (T (f n n "flie X va-iablcs accumulate the value of'the numerator, and tlie Y variables accumulate thc valuc of the- denominator. Atter the program has looped over all the search tern1s, it calculates the correlation using the tinal expression in the flowchart.

The method begins at block 1500. At block 1502, variables X, Y1 aruf Y2 are initialized. A loop is entered at block 1504 for processing each search tern> >n the list of search terms. At block 1506, variables X 1 and X2 are calculated using a rating algorithm. 'I,he rating algorithni compirtes the rating an advertiser assigns to a search term. One embodiment of a suitable rating algorithm is described below in conjunction with FIG. 16. At block 15023, the values of Xl and X2 ar-c combined with the previous value of X as shown to produce the current value of X. At block 1510, values of Y I arrd Y2 are updated using the calculated values of X 1 and X2. At block 15 I 2, control loops back to block 1504 until all search terrns have been processed. Thc Pearson correlation is then calculated as shown at block 1514. The method ends at block 1516 and the value of the Pearson correlation is rettrrned.

FIG. 16 is a flow diagram sliowing one embodinient of an algorithrn for calculating the rating that an advertiser assigns to a tenn. If the rating recorded iit the V[A]['I'] array is not IJNKNOWN, the algorithrn simply returns it.
Otherwrsc it returns the advertiser's mean r-ating.

The method begins at block 1600. Two variables are passed, an advertrsr,r-variable and a term variable. At block 1602, it is determined if the rating associated with the advertiser and the term is unknown. If not, at block 1604 the rating is set equal to the rating value in the array of ratings. If the variable is unknown, at block 1606 the rating is set equal to the advertiser's mean rating.
One method for calculating the advertiser's mean r-ating is described below in conjunctiorr with FIG. 17. The r-ating is i-etur7ied and the method ends at block 1608.

FI(i. 17 is a flow diagram showing one embodiment of an algorithm for calculating an advertiser's mean rating. 'fhe algorithni is a loop over every search ternl. For each search terrn that has a known rating, the program adds the rating to the suni S and increnients the counter N. At thc end of the loop, the rnean rating is sinlply the ratio SIN. -"I'hc metliod begins at block 1700. At block 1702, a sum variable S and a count variable N are initialized. At block I 704, a loop is entered, selecting se.arch terms of the advertiser's list according to the looping variable. At block 1706, it is deterniined if ttie rating tor the search terni, stored in the rating array, has a valuc of UNKNOWN. If not, at block 1708, the value of the rating V[A][T] is added to the sum variable S and the count variable N is incremented. Control proceeds to block 1710 where the loop is repeated until all seat-ch terms in the advertiser's list of search terms have been processed. At block 1712, the mean rating is calculateci as the ratio of S to N. At block 1714, the niethod ends and the mean rating is rettrrned.
FIG. 18 is a flow diagram showing one embodintent of an algorithm for combining recommendations from the web spider and collaborative filter_ A
term's combined rating is a weighted sum of the spider's rating and the collaborative filter's rating_ Itiitially, whetl the advet-tiser has not yet accepted oi rejected any terms, the algorithm weights the ratings of the collaborative tilter twice as strotlgly as it weights the recommendations of the spider. As the number of accepted and rejected ter-ms increases, the weight of the spider ratings decreases proportionally. There are many othet- possible formulas for generating a combined rating from the individual ratitigs, and the current invention is not limited to any one of thcni.
In the embociirnent of FIG. 18, the method begins at block 1800. At block 1802, a variable N is set equal to the number of.recommended search terms accepted by the advet-tiser and a variable M is set equal to the nurnber of recommended terms t-ejected by the advertiser_ At block 1804, two routines are called to c.alculate the predicted rating from the spicler and the predicted rating from collaborative tiltering. Exemplary entbodiments of these routines are discussed below in conjunction with FIGS. 19 and 20 respectively. At block 1800, the predictions are combined and the result returned as the method ends at block 1808.

FI(-_ 19 is a flow diagram showing one enibodiment of an algoritltm fior calcttlatin~g the spider's ratittgs. lt'the spider has not IOUnd a term, or if the terin did not pass its cluality filter, then the algorithnn assigns it a rating of 2. The remaining terms receive ratings on a linear scale fiom 4.9 down to 2.1. '1'he top 5 term that tlte spider recommends receives a rating of-4.9, and the last term that it recommetlds receives a rating of 2. 1. 'I'here. are rnany other possible formulas for generating ratings from the spider's ranked recorntnendations, and the current, invention is not limited to any one of them.

The method begins at block 1900. At block 1902, it is determined if the 10 spider found the term passed to the nlethod irt the term variable 7'. If so, at block 1904 a variable N is set equal to the number of tertrls found by the spider and a variable M is set equal to the position of the term T in the sorted list of recommendatiotls returned by the spider.

At block 1906, the predicted rating from the spider is calculated according 15 to the illustrated formula. At block 1908, if the spider did not find the term'T, the predicted rating from the spider is set equal to 2. 'I'he niethod ends at block 19U~
and the predicted rating trom the spider is returned.

FIG. 20 is a flow diagram showing one embodinient of an algorithni for calculating the collaborative filter's ratings. 'I'he algorithm is a loop over every 20 advertiser. Fo-- each advertiser, the program accumulates values that allow it to calculate the rating according to the forrnula ~( c a)pn +

25 A variable X accumulates the value of the numerator, and a variable Y
accumulates the value of the denotninator. In the last step, the algorithm calculates the fitial rating usitig the expression shown in the flowchart_ 'I'Itis tinal rating may fall outside of the range 0 to 5, but it can still be correctly interprctc-d on this scalc.

`I'lic method begins at block 2000, At block 2002, the variables X rind Y
are initialir.ed_ A loop is entered at block 2004, one advertiser bcin(T
processed f'or each iteration through the loop At block 2000, valnes for variables XA and W
are evaluated as shown. At block 1005, values for X and Y ai-e updated using the values of W and XA. At block 2010, control retunis to the start of the loop at block 2004 to process the next advertiser. After all advertisers have been processed, the prediction froni collaborative filtering is calculated using the formula in block 2012 and the rnean rating algorithrn described above in conjunetion with FIG. 17. The method ends at block 2014 and the prediction frorrr collaborative filtering is returned.

From the foregoing, it can be seen that the present etnbodiments provide a method and apparatus for recommending search terms to an advertiser on a pay for-placement search system. 'I'he method and apparatus make search term recommendations based on the contents of the advertiser's web site and by comparing the advertiser to other similar advertisers and recomrnending search ternis they have chosen. In this manner, the system recornniends good search terms, or terms having a relation to the adve--tiser's web site or its content, whule avoiding bad search terms wliich have no sucli relation. "['he system is interactivc witti the advertiser, allowing him to decide when the set of search terms is sufficient for his requirements. However, the process of identifying and ranking search ternis is automated and is based on actual pages of the advertiser's web site and by cornparisons to other advertisers.

While a particular embodiment of the present invention has been shown and described, rnodifications may be nlade. It is therefore intended in the appended clainls to cover suclt changes and modifications, which follow in the true spirit and scope of the invention.

Claims (19)

WHAT IS CLAIMED IS:

1. A method for recommending search terms in a computer network search\
apparatus for generating a result list of items representing a match with information entered by a user through an input device connected to the computer network, the search apparatus including a computer system operatively connected to the computer network and a plurality of items stored in a database, each item including information to be communicated to a user and having associated with it at least one search term, an information provider and a bid amount, the method comprising:

(a) obtaining a set of potential search terms for acceptance by a new information provider who is adding items to the database, including; receiving from the new information provider a website uniform resource locator (URL); and spidering the website associated with the website URL to obtain search terms for the set of potential search terms;

(b) computing correlations between the potential search terms for the new information provider and search terms of other information providers stored in the database;

(c) computing an estimated rating for the each potential search term for the new information provider;

(d) sorting the potential search terms according to the computed estimated ratings;

(e) presenting to the new information provider on an output device the sorted potential search terms;

(f) receiving from the new information provider at an input device an indication of accepted search terms;

(g) repeating (b) through (e) until a completion indication is received from the new information provider; and (h) storing the accepted search terms in the database for the new information provider upon receipt of the completion indication.

2. The method of claim 1 wherein spidering the website comprises: receiving data from pages of the website; recording potential search terms from the data; and determining a quality metric for each potential search term.

3. The method of claim 2 wherein computing an estimated rating comprises:
combining a rating based on the computed correlations and a rating based on the quality metric determined for each candidate search term.

4. The method of claim 2 further comprising: sorting the candidate search terms according to the quality metric; and adding to the set of potential search terms only candidate search terms having a quality metric exceeding a threshold.

5. The method of claim 1 wherein spidering comprises: receiving data from one or more pages of the website; and examining text from the one or more pages for candidate search terms.

6. The method of claim 5 wherein examining text comprises: examining substantially all text from the one or more pages; and examining meta tags from the one or more pages.

7. The method of claim 5 wherein receiving a website URL comprises:
receiving the advertiser's URL as the web site URL.

8. The method of claim 5 wherein receiving a website URL comprises:
receiving the web site URL from the advertiser.

9. The method of claim 1 wherein computing correlations comprises:
assigning ratings to search terms; and computing a correlation between the advertiser and one or more of the other advertisers using the assigned ratings of advertiser search terms.

10. The method of claim 9 wherein computing an estimated rating comprises:
predicting a likelihood that a search term will be relevant to the advertiser.

11. The method of claim 10 wherein predicting comprises: determining a quality metric for potential search terms; and predicting relevance of the potential search terms based on the quality metric.

12. The method of claim 1 wherein presenting the sorted potential search terms to the new information provider comprises sending the sorted potential search terms with a web page to the output device.

13. A computer network search engine apparatus which includes a database having stored therein a plurality of search listings, each search listing being associated with an information provider, at least one keyword, a money amount and a computer network location and a search engine to identify search listings having a keyword matching a keyword entered by a searcher, to order the identified listings using the money amounts for the respective identified listings, and to generate a result list including at least some of the ordered listings, the apparatus comprising:
an account management server including a processing system which is operative in conjunction with program code to recommend potential search terms to a new information provider adding search listings to the database; collaborative filtering code operable in conjunction with the processing system to compute correlations between potential search terms for the new information provider and search terms of other information providers stored in the database and to compute an estimated rating for the each potential search term for the new information provider;
sorting code operable in conjunction with the processing system and configured to sort the potential search terms according to the computed estimated ratings; spidering code overable in conjunction with the processing system to find initially accepted search terms in a web site by spidering the web site and to include the initially accepted search terms among the sorted potential search terms; an output device configured to provide the sorted potential search terms to the new information provider for review; and an input device configured to receive from the new information provider an indication of accepted search terms, the accepted search terms being stored in the database in association with the new information provider upon receipt of the indication from the new information provider.

14. The computer network search engine apparatus of claim 13 wherein the spidering code is configured to spider a web site of the new information provider.

15. The computer network search apparatus of claim 14 wherein the spidering code is configured to spider a web site specified by the new information provider.

16. The computer network search engine apparatus of claim 13 wherein the spidering code is configured to retrieve pages from the web site of the new information provider, record terms contained in the retrieved pages and score the terms according to a quality metric.

17. The computer network search engine apparatus of claim 16 wherein the spidering code is configured to include terms scoring above a threshold score among the sorted potential search terms.

18. A method for making search term recommendations to an advertiser in a pay for placement market system in which search listings of advertisers may be searched by users entering search terms, the method comprising: receiving from the advertiser a website uniform resource locator (URL); spidering the website associated with the website URL to obtain an initial list of search terms to form a set of potential search terms for the advertiser; computing correlations between the set of potential search terms for the advertiser and search terms of other advertisers stored in a database of the pay for placement market system; computing an estimated rating for each potential search term for the advertiser; sorting the potential search terms according to the estimated ratings; providing the sorted potential search terms to the advertiser; receiving from the advertiser the advertiser's indication of accepted search terms; and storing the accepted search terms in the database for searching by the users.

19. The method of claim 18 further comprising: repeating the acts of computing correlations, computing an estimated rating, sorting and providing the potential search terms and receiving an indication of accepted search terms until the advertiser indicated the process is complete.