US8452798B2 - Query and document topic category transition analysis system and method and query expansion-based information retrieval system and method - Google Patents

Query and document topic category transition analysis system and method and query expansion-based information retrieval system and method Download PDF

Info

Publication number
US8452798B2
US8452798B2 US12/706,826 US70682610A US8452798B2 US 8452798 B2 US8452798 B2 US 8452798B2 US 70682610 A US70682610 A US 70682610A US 8452798 B2 US8452798 B2 US 8452798B2
Authority
US
United States
Prior art keywords
query
topic category
document
transition
topic
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active, expires
Application number
US12/706,826
Other versions
US20110047161A1 (en
Inventor
Sung Hyon Myaeng
Yu Chul Jung
Kyung Min Kim
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Korea Advanced Institute of Science and Technology KAIST
Original Assignee
Korea Advanced Institute of Science and Technology KAIST
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Korea Advanced Institute of Science and Technology KAIST filed Critical Korea Advanced Institute of Science and Technology KAIST
Assigned to KOREA ADVANCED INSTITUTE OF SCIENCE AND TECHNOLOGY reassignment KOREA ADVANCED INSTITUTE OF SCIENCE AND TECHNOLOGY ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: JUNG, YU CHUL, KIM, KYUNG MIN, MYAENG, SUNG HYON
Publication of US20110047161A1 publication Critical patent/US20110047161A1/en
Application granted granted Critical
Publication of US8452798B2 publication Critical patent/US8452798B2/en
Active legal-status Critical Current
Adjusted expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
    • G06F16/33—Querying
    • G06F16/3331—Query processing
    • G06F16/3332—Query translation
    • G06F16/3338—Query expansion
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions

Definitions

  • the present invention relates to an information retrieval system and method, and more particularly, to a query and document topic category transition analysis system and method in which a query topic category of a query input from a user in the form of a set of keywords and a document topic category of a document which a user regards as relevant and selects from information retrieval results are classified to analyze transition between the query topic category and the document topic category, and a query expansion-based information retrieval system and method using query and document topic category transition analysis in which a query input from a user is expanded using a topic category transition analysis result, and corresponding information or documents are retrieved using the expanded query.
  • Conventional techniques for online (Internet) information retrieval services include a document similarity ranking technique for a search engine, a topic category-based document classification technique, and a topic category-based log analysis technique.
  • a search engine statistically analyzes terms of documents and links between documents using document indexes, generates retrieval results suitable for a query input by a user in the form of a ranked list (a set of links indicating documents) and provides the user with the retrieval results through a web page.
  • a process of representing documents in a form suitable for machine learning is performed, and, during the document representing process, selecting appropriate features, and weighting the features are preceded.
  • the document classification process described above may be variously applied to fields such as a voice recognition-based customer center automatic call classification system, a topic category classification system of advertisement contents for keyword advertisements, and an automatic classification system of web sites/patents/academic literature/books.
  • the conventional art described above uses the ODP taxonomy, but has a problem in that it uses only a small number (15) of highest level (coarse-grained) topic categories as topic categories and cannot perform precise (fine-grained) topic category classification based on the ODP taxonomy.
  • the present invention is directed to a query and document topic category transition analysis system and method in which a query topic category of a query input by a user as an information retrieval keyword and a document topic category of a document which a user regards as relevant and selects from information retrieval results are linked to analyze transition between the query topic category and the document topic category, and a query expansion-based information retrieval system and method using query and document topic category transition analysis in which a query input from a user is expanded using a topic category transition analysis result, and an expanded query is used to retrieve information or documents.
  • a method of analyzing transition between a query topic category and a document topic category including: classifying a query input from a user and classifying a document which a user selects from information retrieval results for the input query; deriving a weight with respect to topic category transition between the query topic category and the document topic category; and generating a topic category transition map as a result of analyzing topic category transition between a user query and a relevant document based on the derived weight for the topic category transition.
  • the method may further include: generating corresponding documents as pseudo documents according to each topic category for the user query and the relevant document based on the generated topic category transition map; and extracting at least one representative keyword from the generated pseudo documents and storing the at least one representative keyword.
  • a query expansion-based information retrieval method using query and document topic category transition analysis including: in a state in which a topic category transition map is generated as a result of analyzing topic category transition between a user query and a relevant document, and corresponding documents are generated as pseudo documents according to each topic category for the user query and the relevant document, classifying a corresponding query topic category based on query and document text information for an input query input from a user; allocating a relevant document topic category for the classified query topic category based on the topic category transition map; ranking representative keywords for the query topic category and the relevant document topic category based on the pseudo documents; expanding the input query using the ranked representative keywords; and retrieving corresponding documents using the expanded query.
  • the method may further include providing a user with the retrieved documents according to a ranking of the representative keywords.
  • a computer readable record medium recording a program of implementing the method of one aspect of the present invention or another aspect of the present invention.
  • a system for analyzing transition between a query topic category and a document topic category including: a relevance determined document collection database which stores query and document text information; a topic category classifier which classifies a query topic category for a user query based on the query and document text information and classifies a document topic category for a document selected from information retrieval results for the user query based on the query and document text information; and a topic category transition map database which stores a topic category transition map between a user query and a relevant document which is generated based on a weight for topic category transition between the query topic category and the document topic category classified by the topic category classifier.
  • the system may further include a topic category-classified pseudo document set database in which corresponding documents are generated as pseudo documents according to each topic category for a user query and a relevant document based on the topic category transition map, and at least one representative keyword extracted from the pseudo documents is stored.
  • a query expansion-based information retrieval system using query and document topic category transition analysis including: a topic category transition analysis system which stores a topic category transition map generated as a result of analyzing topic category transition between a user query and a relevant document and corresponding documents generated as pseudo documents according to each topic category for the user query and the relevant document; a query expander which expands a user query, allocates a relevant document for the classified document topic category, and ranks representative keywords for the relevant document topic category based on the pseudo documents, and expands the user query using the ranked representative keywords; and a search engine which retrieves corresponding documents using the user query expanded by the query expander.
  • FIG. 2 is a block diagram of a topic category classifier according to an exemplary embodiment of the present invention.
  • FIG. 3 is a schematic diagram illustrating a topic category transition map according to an exemplary embodiment of the present invention.
  • FIG. 4 is a diagram for describing a result of analyzing topic category transition between a user query and a relevant document according to an exemplary embodiment of the present invention
  • FIG. 5 is a flowchart illustrating a query expansion-based information retrieval method using query and document topic category transition analysis according to an exemplary embodiment of the present invention.
  • FIGS. 6 to 8 illustrate performance evaluation of a retrieval algorithm according to an exemplary embodiment of the present invention.
  • a document represents a web page, but documents such as general news and blogs which include contents and metadata may be included as documents.
  • Contents may include a text, a voice, and a moving picture
  • metadata may include a document language, a document title, a document size, a document identifier (for example, URL information), a document format, a topic category, and other various attributes.
  • a document is preferably interpreted as one which includes any format of data which represents (includes) information on the web (Internet).
  • a query means an information retrieval keyword input from a user
  • a relevant document means a document (for example, a document which is high in relevance to user interest transition) which a user regards as relevant and selects from information retrieval results.
  • a query log includes a user query log of a certain domain as well as a web query log in a web information retrieval service and is not limited to a certain retrieval service filed.
  • an information retrieval system includes a topic category classifier 11 , a relevance determined document collection database 12 (also called a test collection database), a topic category transition map database 13 , and a topic category-classified pseudo document set database 14 .
  • the information retrieval system further includes a retrieval interface 16 , a service server 17 , and a search engine 18 which are required to realize a typical information retrieval service.
  • the information retrieval system includes a retrieval interface for receiving a query (information retrieval keyword) from a user through an online retrieval browser and a service server for ranking relevant documents (including contents) as retrieval results which a search engine retrieves in response to a user query and provides a user with ranked documents through an online web browser.
  • the topic category classifier 11 classifies a query (hereinafter, “user query”) input from a user as an information retrieval keyword according to a query topic category (QC).
  • the topic category classifier 11 classifies a document which a user regards as relevant and selects from information retrieval results according to a document topic category (DC).
  • the relevance determined document collection database 12 stores query and document text information for determining a document (a document which is high in relevance to user interest transition, that is, a relevant document) relevant to a user query.
  • the relevance determined document collection database 12 is also called a test collection database, and a test collection commonly includes a query collection, a news document collection, and mapping information between documents which are relevant or not relevant to a query.
  • the topic category transition map database 13 stores a topic category transition map which is a result of analyzing topic category transition between a user query and a relevant document through a query and document topic category transition analysis technique according to an exemplary embodiment of the present invention.
  • the topic category-classified pseudo document set database 14 stores a set of pseudo documents classified according to a topic category, which will be used in expanding a query input from a user using a topic category transition analysis result.
  • a process of analyzing transition between a query topic category and a document topic category is preferably performed by the topic category classifier 11 after query topic category classification and document topic category classification are performed.
  • a query topic category classification process and a document topic category classification process will be described later in detail with reference to FIG. 2 .
  • the topic category classifier 11 inquires into query and document text information stored in the relevance determined document collection database 12 through a query input from a user, and extracts a corresponding query topic category and classifies a user query as the extracted query topic category (that is, the extracted query topic category is imparted to a user query).
  • the topic category classifier 11 also inquires into query and document text information stored in the relevance determined document collection database 12 through a document which a user regards as relevant and selects from information retrieval results, and extracts a corresponding document topic category and classifies a relevant document as the extracted document topic category (that is, the extracted document topic category is imparted to a relevant document).
  • the topic category classifier 11 determines a category name and a relevance score of the query topic category and a category name and a relevance score of the document topic category, determines a weight between a query topic category and a document topic category based on the relevance scores, and generates (acquires) a topic category transition map (the topic category transition map database 13 ).
  • w xy denotes a weight between a query topic category x of a user query q i and a document topic category of a relevant document d j .
  • a function S(•) denotes a relevance score provided (loaded) by the topic category classifier 11 .
  • a degree of topic category transition from a certain query topic category to a certain document topic category is expressed as an increment in a weight w xy through Formula 1.
  • a weight between a query topic category and a document topic category is derived based on an information retrieval log (a log related to a query input from a user and a log related to a relevant document which a user selects from information retrieval results) and data (query and document text information) of the relevance determined document collection database 12 .
  • a weight between a query topic category and a document topic category which is derived as described above is converted into a transition probability value that a certain query topic category will transition to a certain document topic category or a transition probability value that a certain query will transition to a certain document topic category using Formulas 2 and 3 described below.
  • FIG. 3 illustrates a topic category transition map derived through computation of a transition probability value that a certain query topic category QC j will transition to a certain document topic category DC k .
  • FIG. 3 illustrates an example which uses a transition probability value that a certain query category will transition to a certain document topic category, but it will be apparent to those skilled in the art that the present invention covers a case in which a transition probability value that a certain query will transition to a certain document topic category is used.
  • a jk which represents a weight of a transition occurrence from a query topic category QC j to a document topic category DC k is obtained by adding all of weights (weights between a query topic category and a document topic category) which can be obtained between a query and a document which belong to a query topic category QC j and a document topic category DC k , respectively.
  • a transition probability value that a certain query Q will transition to a certain document topic category DC k is computed by adding all of probability values that each of n query topic categories QC j as which a query Q can be classified will transition to a certain document topic category DC k , that is, a sum of probability values each of which is computed by Formula 2 for each query topic category QC j , as in Formula 3:
  • FIG. 4 illustrates a result of analyzing topic category transition between a user query and a relevant document based on a topic category transition map derived through computation of a transition probability value that a query topic category QC j will transition to a document topic category DC k .
  • FIG. 4 illustrates a representative example of a topic category transition map in which the TREC4 ad hoc search test collection is utilized.
  • the TREC4 ad hoc search test collection is a test collection for evaluating a search engine, which was developed by National Institute of Standards and Technology (NIST), and includes a query collection, a news document collection, and mapping information between documents which are relevant or not relevant to a query.
  • Data used in an exemplary embodiment of the present invention is TREC4 data which includes, for example, 49 queries (average 7.5 words) of an interrogative sentence type and news documents such as AP news and Wall Street Journal from 1988 to 1992 which is commonly known as “Tipster disk 2&3”, wherein the total number of news documents is 567,529, and each query has an average of 133 related news documents.
  • Topic 204 has three document topic categories DC which are highest in probability that a query topic category QC will transition.
  • three document topic categories are topic categories which are determined as relevant to a topic for a user query. That is, three document topic categories of Topic 204 are topic categories for a relevant document.
  • Topic 207 In the case of Topic 207 , a query topic category was wrongly allocated, and a second document topic category was also wrongly allocated.
  • Topic 250 In the case of Topic 250 , a query topic category was somewhat wrongly allocated, but document topic categories are determined as properly allocated.
  • a case in which the number of times that one query topic category transitions to a certain document topic category is small is treated as noise, and a case in which the number of transition times is large is focused, whereby an information retrieval service in which a main tendency of the general public or a certain group is reflected is provided.
  • a topic category of each of a user query and a relevant document is hierarchically classified, and a result of analyzing topic category transition between a user query and a relevant document is derived based on a topic category classification result.
  • Corresponding documents are generated as pseudo documents according to a topic category of each of a user query and a relevant document based on a result of analyzing topic category transition between a user query and a relevant document, which is performed by the topic category classifier 11 as described above, and pseudo documents are stored in the topic category-classified pseudo document set database 14 .
  • a pseudo document set classified according to a topic category is configured using a method of ranking representative keywords in documents collected according to each topic category, for example, a DF-ICF method (which is disclosed in “ Advertising Keyword Suggestion based on Concept Hierarchy” by Yifan Chen et al, Inf. Conf. of Web Search and Data Mining, 2008), and the representative keywords are used in expanding a query input from a user to perform information retrieval (query expansion will be described later with reference to FIG. 5 ).
  • a DF-ICF method which is disclosed in “ Advertising Keyword Suggestion based on Concept Hierarchy” by Yifan Chen et al, Inf. Conf. of Web Search and Data Mining, 2008
  • the topic category classifier 11 includes a taxonomy database 21 , a topic category-classified centroid generator 22 , and a topic category-classified similarity calculator 23 .
  • the taxonomy database 21 may be a certain taxonomy which includes query topic categories and document topic categories which are suitable for an information retrieval service field, for example, to which the present invention is applied, certain information for expressing each query topic category, and certain information for expressing each document topic category.
  • the ODP also known as Directory Mozilla (DMOZ)
  • DMOZ Directory Mozilla
  • fine-grained topic category classification for a user query and a relevant document is performed through the topic category classifier using an external taxonomy such as the ODP, and transition between a query topic category and a document topic category is configured as a topic category transition map based on the fine-grained topic category classification.
  • a topic category for each of a user query and a relevant document is hierarchically classified by the topic category classifier 11 , which will be described later.
  • fine-grained topic category classification for each of a user query and a relevant document is performed using an external taxonomy such as the ODP.
  • a transition probability value for topic category transition between a query topic category and a document topic category is computed based on the fine-grained topic category classification to generate a topic category transition map as a result of analyzing topic category transition between a user query and a relevant document.
  • corresponding documents are collected according to a topic category of each of a pair of (a user query, a relevant document) to generate pseudo documents, and representative keywords are extracted from the pseudo documents and stored in the topic category-classified pseudo document set database 14 .
  • the topic category-classified centroid generator 22 generates a centroid vector using certain information for expressing each topic category according to each topic category (a query topic category and a document topic category) stored in the taxonomy database 21 as in Formula 4. For example, according to an exemplary embodiment of the present invention, a snippet web site address, a title and a description which belong to each topic category of the ODP are used as certain information for expressing each topic category.
  • ⁇ right arrow over (c j ) ⁇ is the centroid for category c j ; s iterates over the snippets in a particular category.
  • snippet web documents s belonging to a topic category c j are collectively used, so that a centroid vector is generated using words belonging to the snippet web documents.
  • the topic category-classified similarity calculator 23 computes cosine similarity according to a query topic category for a query input to the topic category classifier and cosine similarity according to a document topic category for a document input to the topic category classifier, based on a centroid vector generated by the topic category-classified centroid generator 22 .
  • the topic category classifier 11 classifies a query topic category for an input query and a document topic category for an input document based on cosine similarity according to a query topic category computed for an input query and cosine similarity according to a document topic category computed for an input document.
  • a query expansion-based information retrieval method which will be described below with reference to FIG. 5 is performed by an information retrieval system, preferably, the search engine 18 .
  • a search engine is preferably interpreted as a main body for performing information retrieval rather than a module realized by a certain process or device.
  • a query that is, an information retrieval keyword
  • the search engine 18 instructs the topic category classifier 11 , so that the topic classifier 11 inquires into the relevance determined document collection database 12 and classifies a query topic category for a user query (Operation 52 ).
  • the search engine 18 instructs the topic category classifier 11 , so that the topic category classifier 11 inquires into the topic category transition map database 13 and allocates a document topic category (that is, a relevant document topic category of a document which a user regards as relevant and so selects) anticipated from the classified query topic category (Operation 53 ).
  • a document topic category that is, a relevant document topic category of a document which a user regards as relevant and so selects
  • a topic category transition map database 13 a result of analyzing topic category transition between a user query and a relevant document described above is included in a topic category transition map.
  • the search engine 18 instructs the topic category classifier 11 , so that the topic category classifier 11 inquires into the topic category-classified pseudo document set database 14 and ranks representative keywords in documents collected according to each topic category for each of the allocated query topic category and the allocated document topic category, for example, ranks keywords acquired after rearranging using a DF-ICF method (Operation 54 ).
  • keywords are extracted from documents corresponding to a topic category which is allocated to relevant documents, and are ranked based on a DF*ICF weight to thereby configure a representative keyword list.
  • a second highest relevant document topic category and a third highest relevant document topic category as well as a highest relevant document topic category may be included for the allocated query topic category, and a representative keyword list may be configured by ranking keywords thereof.
  • the search engine 18 instructs the topic category classifier 11 , preferably a query expander (not shown) installed at a search engine 18 side, so that the query expander expands a user query (that is, a query input from a user in Operation 51 ), which is initially input, by using the ranked representative keywords (Operation 55 ).
  • the expanded query is configured by synthesizing an initial user query and the ranked representative keywords, and a process of expanding a query may be performed through simple merging or a synthesis method (Rocchio query expansion method) using a synthesis weight.
  • a transition probability value that a query topic category QC j will transition to a document topic category DC k which is computed by Formula 2, may be used as a weight for query expansion.
  • the search engine 18 retrieves all of corresponding documents on the web (Internet) using the expanded query obtained through the query expander (Operation 56 ).
  • the search engine 18 provides a user with retrieved documents at an online retrieval browser side through the service server 17 according to a ranking of the representative keywords (Operation 57 ).
  • the search engine 18 provides a user with retrieved documents as information retrieval results for a user query in the form of a web page including a list (that is, a set of links indicating documents) which is ordered according to a ranking of the representative keywords through the service server 17 .
  • the present algorithm a performance evaluation result for the query expansion-based information retrieval technique (hereinafter, “the present algorithm”) using the query and document topic category transition analysis according to an exemplary embodiment of the present invention will be described with reference to FIGS. 6 to 8 .
  • FIGS. 6 to 8 illustrate performance evaluation of a retrieval algorithm according to an exemplary embodiment of the present invention.
  • FIG. 6 illustrates a performance evaluation result for “Precision@n”
  • FIG. 7 illustrates a performance evaluation result for “Interpolated precision-recall”
  • FIG. 8 illustrates a performance evaluation result for “Overall Performance Comparisons”.
  • TTRF topic transition relevance feedback
  • Precision@n illustrated in FIG. 6 is a criterion for evaluating how preferentially a document relevant to a query is retrieved
  • “Interpolated precision-recall” illustrated in FIG. 7 is a criterion for evaluating how many documents relevant to a query are preferentially retrieved.
  • the algorithm TTRF suggested according to an exemplary embodiment of the present invention is most excellent in performance.
  • performance of the algorithm TTRF according to an exemplary embodiment of the present invention when several document topic categories (4DC in the present experiment) are considered is more excellent than when only one document topic category (1DC) is considered.
  • FIG. 8 A performance improvement ratio of the algorithm TTRF according to an exemplary embodiment of the present invention is illustrated in FIG. 8 .
  • the ERF method is used as a reference for comparing performance improvement ratios of the algorithms.
  • the algorithm TTRF according to an exemplary embodiment of the present invention showed performance improvement of 28%, 48%, and 43% in three measures of mean average precision (MAP), P@5 (precision at 5 retrieved documents), and P@10 (precision at 10 retrieved documents), respectively.
  • MAP mean average precision
  • P@5 precision at 5 retrieved documents
  • P@10 precision at 10 retrieved documents
  • the query and document topic category transition analysis method and the query expansion-based information retrieval method can be implemented as a computer program. Codes and code segments which configure the computer program can be easily inferred by computer programmers skilled in the art.
  • the computer program is stored in a computer readable record medium (information storing medium), and is read and executed in a computer to implement the functions described above.
  • the computer readable record medium includes all types of record media which can be read by a computer.
  • a document topic category relevant to a query topic category for an information retrieval keyword input from a user is anticipated to expand a user query, so that documents in which user interest is reflected can be retrieved. Documents with high relevance are ranked high, and user satisfaction for a retrieval service is high.
  • interest transition of a user is efficiently analyzed using a fine-grained hierarchical topic category structure, and an analysis result can be used for query expansion.
  • an information retrieval service in which a main tendency of the general public or a certain group with respect to a certain topic is reflected based on the number of times that a query topic category transitions to a document topic category can be provided.
  • the present invention when a user query and a document (for example, text contents) relevant to a user query are defined, the present invention can be flexibly applied to various information retrieval service fields.

Abstract

An information retrieval system and method, and more particularly, a query and document topic category transition analysis system and method in which a query topic category of a query input from a user as an information retrieval keyword and a document topic category of a document which a user regards as relevant and selects from information retrieval results are classified to analyze transition between the query topic category and the document topic category, and a query expansion-based information retrieval system and method using query and document topic category transition analysis in which a query input from a user is expanded using a topic category transition analysis result, and corresponding information or documents are retrieved using the expanded query are provided.

Description

CROSS REFERENCE TO RELATED APPLICATION
This application relates to and claims priority to corresponding Korean Patent Application No. 10-2009-0025759, which was filed on Mar. 26, 2009, the entire disclosure of which is incorporated herein by reference.
BACKGROUND
1. Field of the Invention
The present invention relates to an information retrieval system and method, and more particularly, to a query and document topic category transition analysis system and method in which a query topic category of a query input from a user in the form of a set of keywords and a document topic category of a document which a user regards as relevant and selects from information retrieval results are classified to analyze transition between the query topic category and the document topic category, and a query expansion-based information retrieval system and method using query and document topic category transition analysis in which a query input from a user is expanded using a topic category transition analysis result, and corresponding information or documents are retrieved using the expanded query.
2. Discussion of Related Art
Conventional techniques for online (Internet) information retrieval services include a document similarity ranking technique for a search engine, a topic category-based document classification technique, and a topic category-based log analysis technique.
A Document Similarity Ranking Technique for a Search Engine (Hereinafter, “Conventional Art 1”)
In conventional art 1, documents relevant to a query input from a user are retrieved based on a similarity between the document and the query. Most of information retrieval web portal sites which are commercialized (in service) rank various kinds of web contents such as blogs, knowledge, images, news, and shopping information based on retrieval queries through a search engine and provide users with ranked retrieval results.
To this end, all documents on the web have to be indexed in advance, and a search engine statistically analyzes terms of documents and links between documents using document indexes, generates retrieval results suitable for a query input by a user in the form of a ranked list (a set of links indicating documents) and provides the user with the retrieval results through a web page.
However, in information retrieval ranking, usually texts and metadata of documents and relation information (for example, links or topic categories) between documents are used. A method which gives attracting contents from public as high rank is restrictedly used, but there is a problem in that a user preference which depends on a query category of information retrieval keywords entered by a user is excluded from factors for determining information retrieval ranking.
Topic Category-Based Document Classification Technique (Hereinafter, “Conventional Art 2”)
In conventional art 2, in constructing an information retrieval system, a document is classified in advance into one topic category which is previously defined or multiple topic categories which are previously defined.
For example, a document classification process of conventional art 2 is described below.
A process of representing documents in a form suitable for machine learning is performed, and, during the document representing process, selecting appropriate features, and weighting the features are preceded.
Then, in order to accurately allocate a category within an appropriate time, a process of learning a document categorization rule is performed, newly inputted documents are classified according to the learning result.
In particular, in the case in which text-based taxonomy which is already constructed is equipped, a method of extracting input vectors from input documents, generating similarity to vectors representing topic categories which are previously defined, and allocating topic categories to the documents is used.
The document classification process described above may be variously applied to fields such as a voice recognition-based customer center automatic call classification system, a topic category classification system of advertisement contents for keyword advertisements, and an automatic classification system of web sites/patents/academic literature/books.
Meanwhile, a method of automatically identifying a topic category of a user query or a topic category of a document using taxonomy which continuously evolves such as an open directory project (hereinafter, “ODP”) has been attempted, but no research on analyzing transition between a query topic category and a category of relevant documents has been conducted.
Topic Category-Based Log Analysis Technique (Hereinafter, “Conventional Art 3”)
In conventional art 3, based on session information included in a web log related to a query input from a user, session information included in a web log related to retrieval results for the query, and a topic category of a user input query and a user read content, a user navigation path is detected, and navigation path transition is analyzed and used in an information retrieval system.
For example, in “Analysis of Topic Dynamics in Web Search” by Xuehua Shen et al, Int. Conf. of World Wide Web, 2005, an experiment for analyzing and learning topic category transition between web pages which a user queries and then visits according to time and a user (personal/group/general public) based on a Markov model and anticipating a web page which a user will visit later has been conducted. An aspect of user behavior could be somewhat anticipated through an experimental result, and when users were classified into groups of persons with similar behavior and analyzed, it turned out that performance was improved.
However, the conventional art described above anticipates a topic category of a web page which a user will visit without considering a difference between a query input from a user and a web page visited by users.
Also, the conventional art described above uses the ODP taxonomy, but has a problem in that it uses only a small number (15) of highest level (coarse-grained) topic categories as topic categories and cannot perform precise (fine-grained) topic category classification based on the ODP taxonomy.
For the foregoing reasons, there is an urgent need for technology which can more precisely analyze transition of a topic interesting to a user who uses an information retrieval service and classify the user's intention or interesting topic into more detailed query and document topic categories in view of a phenomenon (a propensity or a tendency) in which a topic interesting to a user when an information retrieval keyword is input is different from a topic interesting to the user when the user selects a document which the user regards as relevant from information retrieval results.
There is also an urgent need for technology which can analyze topic category transition between a user query and a relevant document (a document selected by a user) more precisely based on a query and document topic category classification.
There is also an urgent need for technology which automatically extracts a topic transition tendency and expands a user query based on query and document topic category transition analysis and a user log and thus provides information retrieval results with high user satisfaction.
There is also an urgent need for technology which can detect a topic category of a document (content) which is attracting public attention or which a user prefers according to a topic and give documents corresponding to the topic category high rankings among retrieval results.
SUMMARY OF THE INVENTION
The present invention is directed to a query and document topic category transition analysis system and method in which a query topic category of a query input by a user as an information retrieval keyword and a document topic category of a document which a user regards as relevant and selects from information retrieval results are linked to analyze transition between the query topic category and the document topic category, and a query expansion-based information retrieval system and method using query and document topic category transition analysis in which a query input from a user is expanded using a topic category transition analysis result, and an expanded query is used to retrieve information or documents.
According to an aspect of the present invention, there is a method of analyzing transition between a query topic category and a document topic category, including: classifying a query input from a user and classifying a document which a user selects from information retrieval results for the input query; deriving a weight with respect to topic category transition between the query topic category and the document topic category; and generating a topic category transition map as a result of analyzing topic category transition between a user query and a relevant document based on the derived weight for the topic category transition.
The method may further include: generating corresponding documents as pseudo documents according to each topic category for the user query and the relevant document based on the generated topic category transition map; and extracting at least one representative keyword from the generated pseudo documents and storing the at least one representative keyword.
According to another aspect of the present invention, there is a query expansion-based information retrieval method using query and document topic category transition analysis, including: in a state in which a topic category transition map is generated as a result of analyzing topic category transition between a user query and a relevant document, and corresponding documents are generated as pseudo documents according to each topic category for the user query and the relevant document, classifying a corresponding query topic category based on query and document text information for an input query input from a user; allocating a relevant document topic category for the classified query topic category based on the topic category transition map; ranking representative keywords for the query topic category and the relevant document topic category based on the pseudo documents; expanding the input query using the ranked representative keywords; and retrieving corresponding documents using the expanded query.
The method may further include providing a user with the retrieved documents according to a ranking of the representative keywords.
According to still another aspect of the present invention, there is a computer readable record medium recording a program of implementing the method of one aspect of the present invention or another aspect of the present invention.
According to yet another aspect of the present invention, there is a system for analyzing transition between a query topic category and a document topic category, including: a relevance determined document collection database which stores query and document text information; a topic category classifier which classifies a query topic category for a user query based on the query and document text information and classifies a document topic category for a document selected from information retrieval results for the user query based on the query and document text information; and a topic category transition map database which stores a topic category transition map between a user query and a relevant document which is generated based on a weight for topic category transition between the query topic category and the document topic category classified by the topic category classifier.
The system may further include a topic category-classified pseudo document set database in which corresponding documents are generated as pseudo documents according to each topic category for a user query and a relevant document based on the topic category transition map, and at least one representative keyword extracted from the pseudo documents is stored.
According to yet another aspect of the present invention, there is a query expansion-based information retrieval system using query and document topic category transition analysis, including: a topic category transition analysis system which stores a topic category transition map generated as a result of analyzing topic category transition between a user query and a relevant document and corresponding documents generated as pseudo documents according to each topic category for the user query and the relevant document; a query expander which expands a user query, allocates a relevant document for the classified document topic category, and ranks representative keywords for the relevant document topic category based on the pseudo documents, and expands the user query using the ranked representative keywords; and a search engine which retrieves corresponding documents using the user query expanded by the query expander.
BRIEF DESCRIPTION OF THE DRAWINGS
The above and other purposes, characteristics and advantages of the present invention will become more apparent to those of ordinary skill in the art by describing in detail exemplary embodiments thereof with reference to the accompanying drawings, in which:
FIG. 1 illustrates the configuration of a query expansion-based information retrieval system using query and document topic category transition analysis according to an exemplary embodiment of the present invention;
FIG. 2 is a block diagram of a topic category classifier according to an exemplary embodiment of the present invention;
FIG. 3 is a schematic diagram illustrating a topic category transition map according to an exemplary embodiment of the present invention;
FIG. 4 is a diagram for describing a result of analyzing topic category transition between a user query and a relevant document according to an exemplary embodiment of the present invention;
FIG. 5 is a flowchart illustrating a query expansion-based information retrieval method using query and document topic category transition analysis according to an exemplary embodiment of the present invention; and
FIGS. 6 to 8 illustrate performance evaluation of a retrieval algorithm according to an exemplary embodiment of the present invention.
 DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS
Exemplary embodiments of the present invention will be described in detail below with reference to the accompanying drawings. While the present invention is shown and described in connection with exemplary embodiments thereof, it will be apparent to those skilled in the art that various modifications can be made without departing from the spirit and scope of the invention.
FIG. 1 illustrates the configuration of a query expansion-based information retrieval system using query and document topic category transition analysis according to an exemplary embodiment of the present invention, FIG. 2 is a block diagram of a topic category classifier according to an exemplary embodiment of the present invention, FIG. 3 is a schematic diagram illustrating a topic category transition map according to an exemplary embodiment of the present invention, FIG. 4 is a diagram for describing a result of analyzing topic category transition between a user query and a relevant document according to an exemplary embodiment of the present invention, and FIG. 5 is a flowchart illustrating a query expansion-based information retrieval method using query and document topic category transition analysis according to an exemplary embodiment of the present invention.
In order to help understand the present invention, a query expansion-based information retrieval system and method using query and document topic category transition analysis (hereinafter, “information retrieval system and method”) will be first described, and a query and document topic category transition analysis system and method (hereinafter, “topic category transition analysis system and method”) will be described in corresponding portions while describing the information retrieval system and method.
Meanwhile, in an exemplary embodiment of the present invention, a document represents a web page, but documents such as general news and blogs which include contents and metadata may be included as documents. Contents may include a text, a voice, and a moving picture, and metadata may include a document language, a document title, a document size, a document identifier (for example, URL information), a document format, a topic category, and other various attributes. For example, according to an exemplary embodiment of the present invention, a document is preferably interpreted as one which includes any format of data which represents (includes) information on the web (Internet).
Also, according to an exemplary embodiment of the present invention, a query means an information retrieval keyword input from a user, and a relevant document means a document (for example, a document which is high in relevance to user interest transition) which a user regards as relevant and selects from information retrieval results.
Also, according to an exemplary embodiment of the present invention, a query log includes a user query log of a certain domain as well as a web query log in a web information retrieval service and is not limited to a certain retrieval service filed.
Query Expansion-Based Information Retrieval System Using Query and Document Topic Category Transition Analysis.
As shown in FIG. 1, an information retrieval system according to an exemplary embodiment of the present invention includes a topic category classifier 11, a relevance determined document collection database 12 (also called a test collection database), a topic category transition map database 13, and a topic category-classified pseudo document set database 14.
The information retrieval system according to an exemplary embodiment of the present invention further includes a retrieval interface 16, a service server 17, and a search engine 18 which are required to realize a typical information retrieval service. For example, the information retrieval system includes a retrieval interface for receiving a query (information retrieval keyword) from a user through an online retrieval browser and a service server for ranking relevant documents (including contents) as retrieval results which a search engine retrieves in response to a user query and provides a user with ranked documents through an online web browser.
The components of the information retrieval system according to an exemplary embodiment of the present invention will be described in detail. First, the topic category classifier 11 classifies a query (hereinafter, “user query”) input from a user as an information retrieval keyword according to a query topic category (QC). The topic category classifier 11 classifies a document which a user regards as relevant and selects from information retrieval results according to a document topic category (DC).
The relevance determined document collection database 12 stores query and document text information for determining a document (a document which is high in relevance to user interest transition, that is, a relevant document) relevant to a user query. The relevance determined document collection database 12 is also called a test collection database, and a test collection commonly includes a query collection, a news document collection, and mapping information between documents which are relevant or not relevant to a query.
The topic category transition map database 13 stores a topic category transition map which is a result of analyzing topic category transition between a user query and a relevant document through a query and document topic category transition analysis technique according to an exemplary embodiment of the present invention.
The topic category-classified pseudo document set database 14 stores a set of pseudo documents classified according to a topic category, which will be used in expanding a query input from a user using a topic category transition analysis result.
Query and Document Topic Category Transition Analysis Technique.
In an exemplary embodiment of the present invention, a user query and a relevant document are classified as one or more topic categories extracted from among topic categories which are previously defined (that is, one or more topic categories are imparted to a user query and a relevant document), a relevance score for a query topic category of a user query is determined, a relevance score for a document topic category of a relevant document is determined, and transition between a query topic category and a document topic category is analyzed based on a weight between a query topic category and a document topic category.
A process of analyzing transition between a query topic category and a document topic category is preferably performed by the topic category classifier 11 after query topic category classification and document topic category classification are performed. A query topic category classification process and a document topic category classification process will be described later in detail with reference to FIG. 2.
That is, the topic category classifier 11 inquires into query and document text information stored in the relevance determined document collection database 12 through a query input from a user, and extracts a corresponding query topic category and classifies a user query as the extracted query topic category (that is, the extracted query topic category is imparted to a user query).
The topic category classifier 11 also inquires into query and document text information stored in the relevance determined document collection database 12 through a document which a user regards as relevant and selects from information retrieval results, and extracts a corresponding document topic category and classifies a relevant document as the extracted document topic category (that is, the extracted document topic category is imparted to a relevant document).
Then, the topic category classifier 11 determines a category name and a relevance score of the query topic category and a category name and a relevance score of the document topic category, determines a weight between a query topic category and a document topic category based on the relevance scores, and generates (acquires) a topic category transition map (the topic category transition map database 13). A process of determining a weight between a query topic category and a document topic category uses Formula 1:
w xy =S(x,q j)+S(y,d j)  Formula 1
where “wxy” denotes a weight between a query topic category x of a user query qi and a document topic category of a relevant document dj. A function S(•) denotes a relevance score provided (loaded) by the topic category classifier 11.
For example, as input of a pair of (query, document) increases, a degree of topic category transition from a certain query topic category to a certain document topic category is expressed as an increment in a weight wxy through Formula 1.
As described above, a weight between a query topic category and a document topic category is derived based on an information retrieval log (a log related to a query input from a user and a log related to a relevant document which a user selects from information retrieval results) and data (query and document text information) of the relevance determined document collection database 12.
According to an exemplary embodiment of the present invention, a weight between a query topic category and a document topic category which is derived as described above is converted into a transition probability value that a certain query topic category will transition to a certain document topic category or a transition probability value that a certain query will transition to a certain document topic category using Formulas 2 and 3 described below.
That is, according to an exemplary embodiment of the present invention, transition between a query topic category and a document topic category is analyzed in view of both a case in which a certain query exists and a case in which a certain query does not exist in the topic category transition map database 13.
FIG. 3 illustrates a topic category transition map derived through computation of a transition probability value that a certain query topic category QCj will transition to a certain document topic category DCk. FIG. 3 illustrates an example which uses a transition probability value that a certain query category will transition to a certain document topic category, but it will be apparent to those skilled in the art that the present invention covers a case in which a transition probability value that a certain query will transition to a certain document topic category is used.
According to an exemplary embodiment of the present invention, a transition probability value in which a certain query topic category QCj will transition to a certain document topic category DCk is computed by dividing a transition weight from a current query topic category to a certain document topic category by all transition weights derived from a current query topic category as in Formula 2:
P ⁡ ( dc k | qc i ) = A jk ∑ i ⁢ A ji Formula ⁢ ⁢ 2
where Ajk which represents a weight of a transition occurrence from a query topic category QCj to a document topic category DCk is obtained by adding all of weights (weights between a query topic category and a document topic category) which can be obtained between a query and a document which belong to a query topic category QCj and a document topic category DCk, respectively.
Meanwhile, a transition probability value that a certain query Q will transition to a certain document topic category DCk is computed by adding all of probability values that each of n query topic categories QCj as which a query Q can be classified will transition to a certain document topic category DCk, that is, a sum of probability values each of which is computed by Formula 2 for each query topic category QCj, as in Formula 3:
P ⁡ ( d ⁢ ⁢ c k | q ) = ∑ n ⁢ ( A jk ∑ i ⁢ A ji ) . Formula ⁢ ⁢ 3
FIG. 4 illustrates a result of analyzing topic category transition between a user query and a relevant document based on a topic category transition map derived through computation of a transition probability value that a query topic category QCj will transition to a document topic category DCk.
For example, in an exemplary embodiment of the present invention, a “TREC4 ad hoc search test collection” is used as an example of the relevance determined document collection database 12, and FIG. 4 illustrates a representative example of a topic category transition map in which the TREC4 ad hoc search test collection is utilized.
The TREC4 ad hoc search test collection is a test collection for evaluating a search engine, which was developed by National Institute of Standards and Technology (NIST), and includes a query collection, a news document collection, and mapping information between documents which are relevant or not relevant to a query. Data used in an exemplary embodiment of the present invention is TREC4 data which includes, for example, 49 queries (average 7.5 words) of an interrogative sentence type and news documents such as AP news and Wall Street Journal from 1988 to 1992 which is commonly known as “Tipster disk 2&3”, wherein the total number of news documents is 567,529, and each query has an average of 133 related news documents.
As shown in FIG. 4, among three query topics Topic 204, Topic 207 and Topic 250, only Topic 204 has three document topic categories DC which are highest in probability that a query topic category QC will transition. In the case of Topic 204, three document topic categories are topic categories which are determined as relevant to a topic for a user query. That is, three document topic categories of Topic 204 are topic categories for a relevant document.
In the case of Topic 207, a query topic category was wrongly allocated, and a second document topic category was also wrongly allocated.
In the case of Topic 250, a query topic category was somewhat wrongly allocated, but document topic categories are determined as properly allocated.
For example, in an exemplary embodiment of the present invention, a case in which the number of times that one query topic category transitions to a certain document topic category is small is treated as noise, and a case in which the number of transition times is large is focused, whereby an information retrieval service in which a main tendency of the general public or a certain group is reflected is provided.
According to an exemplary embodiment of the present invention, as can be seen from FIG. 4, a topic category of each of a user query and a relevant document is hierarchically classified, and a result of analyzing topic category transition between a user query and a relevant document is derived based on a topic category classification result.
Corresponding documents are generated as pseudo documents according to a topic category of each of a user query and a relevant document based on a result of analyzing topic category transition between a user query and a relevant document, which is performed by the topic category classifier 11 as described above, and pseudo documents are stored in the topic category-classified pseudo document set database 14.
For example, a pseudo document set classified according to a topic category is configured using a method of ranking representative keywords in documents collected according to each topic category, for example, a DF-ICF method (which is disclosed in “Advertising Keyword Suggestion based on Concept Hierarchy” by Yifan Chen et al, Inf. Conf. of Web Search and Data Mining, 2008), and the representative keywords are used in expanding a query input from a user to perform information retrieval (query expansion will be described later with reference to FIG. 5).
Next, a query topic category classification process and a document topic category classification process which are mentioned above will be described in detail with reference to FIG. 2.
Topic Category Classification Technique
As described in FIG. 1, the query and document topic category transition analysis system according to an exemplary embodiment of the present invention includes the topic category classifier 11, the relevance determined document collection database (also called the test collection database) 12, the topic category transition map database 13, and the topic category-classified pseudo document set database 14, and a query topic category classification process and a document topic category classification process will be described below in detail focusing on operation of the topic category classifier 11.
As shown in FIG. 2, the topic category classifier 11 includes a taxonomy database 21, a topic category-classified centroid generator 22, and a topic category-classified similarity calculator 23.
The taxonomy database 21 may be a certain taxonomy which includes query topic categories and document topic categories which are suitable for an information retrieval service field, for example, to which the present invention is applied, certain information for expressing each query topic category, and certain information for expressing each document topic category.
According to an exemplary embodiment of the present invention, the ODP (also known as Directory Mozilla (DMOZ)) which is a web site taxonomy massively constructed by the general public is used as an example of the taxonomy database 21 in order to capture all general interests of users.
That is, according to an exemplary embodiment of the present invention, fine-grained topic category classification for a user query and a relevant document is performed through the topic category classifier using an external taxonomy such as the ODP, and transition between a query topic category and a document topic category is configured as a topic category transition map based on the fine-grained topic category classification. In particular, according to an exemplary embodiment of the present invention, a topic category for each of a user query and a relevant document is hierarchically classified by the topic category classifier 11, which will be described later.
A process of analyzing topic category transition between a user query and a relevant document is summarized below in order to help understand the present invention.
According to an exemplary embodiment of the present invention, fine-grained topic category classification for each of a user query and a relevant document is performed using an external taxonomy such as the ODP.
Then, a transition probability value for topic category transition between a query topic category and a document topic category is computed based on the fine-grained topic category classification to generate a topic category transition map as a result of analyzing topic category transition between a user query and a relevant document.
Based on the result of analyzing topic category transition between a user query and a relevant document, corresponding documents are collected according to a topic category of each of a pair of (a user query, a relevant document) to generate pseudo documents, and representative keywords are extracted from the pseudo documents and stored in the topic category-classified pseudo document set database 14.
The topic category-classified centroid generator 22 generates a centroid vector using certain information for expressing each topic category according to each topic category (a query topic category and a document topic category) stored in the taxonomy database 21 as in Formula 4. For example, according to an exemplary embodiment of the present invention, a snippet web site address, a title and a description which belong to each topic category of the ODP are used as certain information for expressing each topic category.
c j → = 1 c j ⁢ ∑ s -> ∈ C j ⁢ s -> s -> Formula ⁢ ⁢ 4
where {right arrow over (cj)} is the centroid for category cj; s iterates over the snippets in a particular category. For example, as in Formula 4, according to an exemplary embodiment of the present invention, snippet web documents s belonging to a topic category cj are collectively used, so that a centroid vector is generated using words belonging to the snippet web documents.
The topic category-classified similarity calculator 23 computes cosine similarity according to a query topic category for a query input to the topic category classifier and cosine similarity according to a document topic category for a document input to the topic category classifier, based on a centroid vector generated by the topic category-classified centroid generator 22.
The topic category classifier 11 classifies a query topic category for an input query and a document topic category for an input document based on cosine similarity according to a query topic category computed for an input query and cosine similarity according to a document topic category computed for an input document.
Next, a query expansion-based information retrieval method according to an exemplary embodiment of the present invention will be described with reference to FIG. 5. A query expansion-based information retrieval process which will be described below with reference to FIG. 5 is performed by an information retrieval system, preferably, the search engine 18. Here, a search engine is preferably interpreted as a main body for performing information retrieval rather than a module realized by a certain process or device.
Query Expansion-Based Information Retrieval Technique Using Query and Document Topic Category Transition Analysis
First, when a query (that is, an information retrieval keyword) is input to the search engine 18 from a user through an online retrieval browser (that is, the retrieval interface 16) (Operation 51), the search engine 18 instructs the topic category classifier 11, so that the topic classifier 11 inquires into the relevance determined document collection database 12 and classifies a query topic category for a user query (Operation 52).
Then, the search engine 18 instructs the topic category classifier 11, so that the topic category classifier 11 inquires into the topic category transition map database 13 and allocates a document topic category (that is, a relevant document topic category of a document which a user regards as relevant and so selects) anticipated from the classified query topic category (Operation 53). Here, in the topic category transition map database 13, a result of analyzing topic category transition between a user query and a relevant document described above is included in a topic category transition map.
Subsequently, the search engine 18 instructs the topic category classifier 11, so that the topic category classifier 11 inquires into the topic category-classified pseudo document set database 14 and ranks representative keywords in documents collected according to each topic category for each of the allocated query topic category and the allocated document topic category, for example, ranks keywords acquired after rearranging using a DF-ICF method (Operation 54). For example, according to an exemplary embodiment of the present invention, keywords are extracted from documents corresponding to a topic category which is allocated to relevant documents, and are ranked based on a DF*ICF weight to thereby configure a representative keyword list. Also, according to an exemplary embodiment of the present invention, a second highest relevant document topic category and a third highest relevant document topic category as well as a highest relevant document topic category may be included for the allocated query topic category, and a representative keyword list may be configured by ranking keywords thereof.
Then, the search engine 18 instructs the topic category classifier 11, preferably a query expander (not shown) installed at a search engine 18 side, so that the query expander expands a user query (that is, a query input from a user in Operation 51), which is initially input, by using the ranked representative keywords (Operation 55). The expanded query is configured by synthesizing an initial user query and the ranked representative keywords, and a process of expanding a query may be performed through simple merging or a synthesis method (Rocchio query expansion method) using a synthesis weight. In particular, according to an exemplary embodiment of the present invention, a transition probability value that a query topic category QCj will transition to a document topic category DCk, which is computed by Formula 2, may be used as a weight for query expansion.
Then, the search engine 18 retrieves all of corresponding documents on the web (Internet) using the expanded query obtained through the query expander (Operation 56).
Subsequently, the search engine 18 provides a user with retrieved documents at an online retrieval browser side through the service server 17 according to a ranking of the representative keywords (Operation 57). For example, the search engine 18 provides a user with retrieved documents as information retrieval results for a user query in the form of a web page including a list (that is, a set of links indicating documents) which is ordered according to a ranking of the representative keywords through the service server 17.
Next, a performance evaluation result for the query expansion-based information retrieval technique (hereinafter, “the present algorithm”) using the query and document topic category transition analysis according to an exemplary embodiment of the present invention will be described with reference to FIGS. 6 to 8.
FIGS. 6 to 8 illustrate performance evaluation of a retrieval algorithm according to an exemplary embodiment of the present invention. FIG. 6 illustrates a performance evaluation result for “Precision@n”, FIG. 7 illustrates a performance evaluation result for “Interpolated precision-recall”, and FIG. 8 illustrates a performance evaluation result for “Overall Performance Comparisons”.
In an experiment for comparing performance of an existing information retrieval algorithm and the present algorithm, “TREC4 ad hoc search test collect” was used.
As relevance feedback methods used in the experiment, 1) a method using a basic query (a baseline retrieval method), 2) an explicit relevance feedback (ERF) method, 3) a pseudo relevance feedback (PRF) method, 4) a topic relevance feedback (TRF) method, and 5) a topic transition relevance feedback (TTRF) method were used. Of these, 5) the TTRF method is an algorithm suggested according to an exemplary embodiment of the present invention, and the remaining methods are well-known algorithms.
In the experiment, it was tested how many documents relevant to a query are preferentially retrieved with respect to the respective algorithms mentioned above while changing the number (1 to 5 docs) of documents used in relevance feedback, the number (0 to 500) of keywords used for query expansion, and the number (1 to 5 DC) of document topic categories. Cases which were highest in performance in each of the algorithms are used as representative performance of the algorithms and compared.
A result of evaluating performance of the algorithms is as follows.
“Precision@n” illustrated in FIG. 6 is a criterion for evaluating how preferentially a document relevant to a query is retrieved, and “Interpolated precision-recall” illustrated in FIG. 7 is a criterion for evaluating how many documents relevant to a query are preferentially retrieved.
It can be understood from FIGS. 6 and 7 that the algorithm TTRF suggested according to an exemplary embodiment of the present invention is most excellent in performance. In particular, performance of the algorithm TTRF according to an exemplary embodiment of the present invention when several document topic categories (4DC in the present experiment) are considered is more excellent than when only one document topic category (1DC) is considered.
A performance improvement ratio of the algorithm TTRF according to an exemplary embodiment of the present invention is illustrated in FIG. 8.
In the table of FIG. 8, the ERF method is used as a reference for comparing performance improvement ratios of the algorithms. The algorithm TTRF according to an exemplary embodiment of the present invention showed performance improvement of 28%, 48%, and 43% in three measures of mean average precision (MAP), P@5 (precision at 5 retrieved documents), and P@10 (precision at 10 retrieved documents), respectively.
The query and document topic category transition analysis method and the query expansion-based information retrieval method can be implemented as a computer program. Codes and code segments which configure the computer program can be easily inferred by computer programmers skilled in the art. The computer program is stored in a computer readable record medium (information storing medium), and is read and executed in a computer to implement the functions described above. The computer readable record medium includes all types of record media which can be read by a computer.
As described above, according to exemplary embodiments of the present invention, a document topic category relevant to a query topic category for an information retrieval keyword input from a user is anticipated to expand a user query, so that documents in which user interest is reflected can be retrieved. Documents with high relevance are ranked high, and user satisfaction for a retrieval service is high.
Also, according to exemplary embodiments of the present invention, interest transition of a user is efficiently analyzed using a fine-grained hierarchical topic category structure, and an analysis result can be used for query expansion.
Also, according to exemplary embodiments of the present invention, an information retrieval service in which a main tendency of the general public or a certain group with respect to a certain topic is reflected based on the number of times that a query topic category transitions to a document topic category can be provided.
Furthermore, according to exemplary embodiments of the present invention, when a user query and a document (for example, text contents) relevant to a user query are defined, the present invention can be flexibly applied to various information retrieval service fields.
It will be apparent to those skilled in the art that various modifications can be made to the above-described exemplary embodiments of the present invention without departing from the spirit or scope of the invention. Thus, it is intended that the present invention cover all such modifications provided they come within the scope of the appended claims and their equivalents.

Claims (14)

What is claimed is:
1. A method of analyzing transition between a query topic category and a document topic category, comprising:
determining a query topic category based on query and document text information with respect to a query input from a user and determining a document topic category based on query and document text information with respect to a document which a user selects from information retrieval results for the input query;
deriving a weight with respect to topic category transition between the classified query topic category and the classified document topic category;
generating a topic category transition map as a result of analyzing topic category transition between a user query and a relevant document based on the derived weight for the topic category transition;
generating corresponding documents as pseudo documents according to each topic category for the user query and the relevant document based on the generated topic category transition map; and
extracting at least one representative keyword from the generated pseudo documents and storing at least one representative keyword.
2. The method of claim 1, wherein in determining the query topic category and determining the document topic category, query and document text information is acquired using at least one external taxonomy, fine-grained hierarchical query topic category classification is performed, and fine-grained hierarchical document topic category classification is performed.
3. The method of claim 2, wherein the external taxonomy includes an open directory project (ODP) (also known as Directory Mozilla (DMOZ)) in which general interest of a user is included.
4. The method of claim 2, wherein the determining of the query topic category and the determining of the document topic category comprise:
generating a centroid vector using information expressing each topic category according to each topic category (a query topic category and a document topic category) stored in the external taxonomy;
computing similarity according to a corresponding query topic category for the input query and similarity according to a corresponding document topic category for the document, based on the generated centroid vector; and
determining a query topic category for the input query and a document topic category for the document, based on the computed similarity according to a corresponding query topic category and the computed similarity according to a corresponding document topic category.
5. The method of claim 4, wherein in generating the centroid vector, a centroid vector is generated using words included in corresponding snippet web documents belonging to each topic category.
6. The method of claim 1, wherein in deriving the weight, a category name and a relevance score for the classified query topic category are determined, a category name and a relevance score for the classified document topic category are determined, and a weight between a query topic category and a document topic category is derived based on the determined relevance scores.
7. The method of claim 1, wherein the generating of the topic category transition map comprises:
converting the derived weight for the topic category transition into a transition probability value that a certain query topic category will transition to a certain document topic category or a transition probability value that a certain query will transition to a certain document topic category; and
generating a topic category transition map as a result of analyzing topic category transition between a user query and a relevant document based on the converted query or document topic category transition probability value.
8. The method of claim 7, wherein the transition probability value that a certain query topic category will transition to a certain document topic category is computed by dividing a transition weight from a current query topic category to a certain document topic category by all transition weights derived from a current query topic category.
9. The method of claim 7, wherein the transition probability value that a certain query will transition to a certain document topic category is computed by adding all of probability values that each of a predetermined number of query topic categories as which a query can be classified will transition to a certain document topic category.
10. The method of claim 1, wherein in extracting the at least one representative keyword, a method of ranking representative keywords in documents collected according to each topic category is used.
11. A query expansion-based information retrieval method using query and document topic category transition analysis, comprising:
in a state in which a topic category transition map is generated as a result of analyzing topic category transition between a user query and a relevant document, and corresponding documents are generated as pseudo documents according to each topic category for the user query and the relevant document, determining a corresponding query topic category based on query and document text information for an input query input from a user;
allocating a relevant document topic category for the classified query topic category based on the topic category transition map;
ranking representative keywords for the query topic category and the relevant document topic category based on the pseudo documents;
expanding the input query using the ranked representative keywords; and
retrieving corresponding documents using the expanded query.
12. The method of claim 11, further comprising providing a user with the retrieved documents according to a ranking of the representative keywords.
13. The method of claim 11, wherein in ranking the representative keywords, a representative keyword list according to a ranking is generated by including at least one document topic category having relevance for the allocated query topic category.
14. The method of claim 11, wherein in expanding the input query, the input query and the ranked representative keywords are synthesized based on simple merging or a synthesis weight.
US12/706,826 2009-03-26 2010-02-17 Query and document topic category transition analysis system and method and query expansion-based information retrieval system and method Active 2031-11-12 US8452798B2 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
KR1020090025759A KR101078864B1 (en) 2009-03-26 2009-03-26 The query/document topic category transition analysis system and method and the query expansion based information retrieval system and method
KR10-2009-0025759 2009-03-26

Publications (2)

Publication Number Publication Date
US20110047161A1 US20110047161A1 (en) 2011-02-24
US8452798B2 true US8452798B2 (en) 2013-05-28

Family

ID=43129283

Family Applications (1)

Application Number Title Priority Date Filing Date
US12/706,826 Active 2031-11-12 US8452798B2 (en) 2009-03-26 2010-02-17 Query and document topic category transition analysis system and method and query expansion-based information retrieval system and method

Country Status (2)

Country Link
US (1) US8452798B2 (en)
KR (1) KR101078864B1 (en)

Cited By (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130080460A1 (en) * 2011-09-22 2013-03-28 Microsoft Corporation Providing topic based search guidance
US20130144904A1 (en) * 2010-04-14 2013-06-06 Nhn Corporation Method and system for providing query using an image
US20150278366A1 (en) * 2011-06-03 2015-10-01 Google Inc. Identifying topical entities
US9336269B1 (en) 2013-03-14 2016-05-10 Google Inc. Determining question and answer alternatives
US9582543B2 (en) 2014-04-24 2017-02-28 International Business Machines Corporation Temporal proximity query expansion
CN107180111A (en) * 2017-06-13 2017-09-19 深圳市宇数科技有限公司 A kind of information recommendation method, electronic equipment, storage medium and system
US20190066675A1 (en) * 2017-08-23 2019-02-28 Beijing Baidu Netcom Science And Technology Co., Ltd. Artificial intelligence based method and apparatus for classifying voice-recognized text
US11526567B2 (en) * 2018-10-17 2022-12-13 International Business Machines Corporation Contextualizing searches in a collaborative session
US11947604B2 (en) * 2020-03-17 2024-04-02 International Business Machines Corporation Ranking of messages in dialogs using fixed point operations

Families Citing this family (228)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8645137B2 (en) 2000-03-16 2014-02-04 Apple Inc. Fast, language-independent method for user authentication by voice
US8677377B2 (en) 2005-09-08 2014-03-18 Apple Inc. Method and apparatus for building an intelligent automated assistant
US9318108B2 (en) 2010-01-18 2016-04-19 Apple Inc. Intelligent automated assistant
US8977255B2 (en) 2007-04-03 2015-03-10 Apple Inc. Method and system for operating a multi-function portable electronic device using voice-activation
US10002189B2 (en) 2007-12-20 2018-06-19 Apple Inc. Method and apparatus for searching using an active ontology
US9330720B2 (en) 2008-01-03 2016-05-03 Apple Inc. Methods and apparatus for altering audio output signals
US8996376B2 (en) 2008-04-05 2015-03-31 Apple Inc. Intelligent text-to-speech conversion
US10496753B2 (en) 2010-01-18 2019-12-03 Apple Inc. Automatically adapting user interfaces for hands-free interaction
US20100030549A1 (en) 2008-07-31 2010-02-04 Lee Michael M Mobile device having human language translation capability with positional feedback
US8676904B2 (en) 2008-10-02 2014-03-18 Apple Inc. Electronic devices with voice command and contextual data processing capabilities
US9959870B2 (en) 2008-12-11 2018-05-01 Apple Inc. Speech recognition involving a mobile device
US10255566B2 (en) 2011-06-03 2019-04-09 Apple Inc. Generating and processing task items that represent tasks to perform
US10241752B2 (en) 2011-09-30 2019-03-26 Apple Inc. Interface for a virtual digital assistant
US10241644B2 (en) 2011-06-03 2019-03-26 Apple Inc. Actionable reminder entries
US9858925B2 (en) 2009-06-05 2018-01-02 Apple Inc. Using context information to facilitate processing of commands in a virtual assistant
US9431006B2 (en) 2009-07-02 2016-08-30 Apple Inc. Methods and apparatuses for automatic speech recognition
US10553209B2 (en) 2010-01-18 2020-02-04 Apple Inc. Systems and methods for hands-free notification summaries
US10705794B2 (en) 2010-01-18 2020-07-07 Apple Inc. Automatically adapting user interfaces for hands-free interaction
US10276170B2 (en) 2010-01-18 2019-04-30 Apple Inc. Intelligent automated assistant
US10679605B2 (en) 2010-01-18 2020-06-09 Apple Inc. Hands-free list-reading by intelligent automated assistant
US9311392B2 (en) * 2010-02-12 2016-04-12 Nec Corporation Document analysis apparatus, document analysis method, and computer-readable recording medium
US8682667B2 (en) 2010-02-25 2014-03-25 Apple Inc. User profiling for selecting user specific voice input processing information
US8392432B2 (en) * 2010-04-12 2013-03-05 Microsoft Corporation Make and model classifier
CN102236663B (en) 2010-04-30 2014-04-09 阿里巴巴集团控股有限公司 Query method, query system and query device based on vertical search
US10762293B2 (en) 2010-12-22 2020-09-01 Apple Inc. Using parts-of-speech tagging and named entity recognition for spelling correction
US10515147B2 (en) * 2010-12-22 2019-12-24 Apple Inc. Using statistical language models for contextual lookup
US9262612B2 (en) 2011-03-21 2016-02-16 Apple Inc. Device access using voice authentication
US10057736B2 (en) 2011-06-03 2018-08-21 Apple Inc. Active transport based notifications
US8994660B2 (en) 2011-08-29 2015-03-31 Apple Inc. Text correction processing
US8862605B2 (en) * 2011-11-18 2014-10-14 International Business Machines Corporation Systems, methods and computer program products for discovering a text query from example documents
US20130138643A1 (en) * 2011-11-25 2013-05-30 Krishnan Ramanathan Method for automatically extending seed sets
US10134385B2 (en) 2012-03-02 2018-11-20 Apple Inc. Systems and methods for name pronunciation
US9483461B2 (en) 2012-03-06 2016-11-01 Apple Inc. Handling speech synthesis of content for multiple languages
US8719025B2 (en) * 2012-05-14 2014-05-06 International Business Machines Corporation Contextual voice query dilation to improve spoken web searching
US9280610B2 (en) 2012-05-14 2016-03-08 Apple Inc. Crowd sourcing information to fulfill user requests
US10417037B2 (en) 2012-05-15 2019-09-17 Apple Inc. Systems and methods for integrating third party services with a digital assistant
US8805848B2 (en) * 2012-05-24 2014-08-12 International Business Machines Corporation Systems, methods and computer program products for fast and scalable proximal search for search queries
US9721563B2 (en) 2012-06-08 2017-08-01 Apple Inc. Name recognition system
US9495129B2 (en) 2012-06-29 2016-11-15 Apple Inc. Device, method, and user interface for voice-activated navigation and browsing of a document
US10108680B2 (en) 2012-08-02 2018-10-23 Rule 14 Real-time and adaptive data mining
US10198516B2 (en) 2012-08-02 2019-02-05 Rule 14 Real-time and adaptive data mining
US10114872B2 (en) 2012-08-02 2018-10-30 Rule 14 Real-time and adaptive data mining
US10114899B2 (en) 2012-08-02 2018-10-30 Rule 14 Real-time and adaptive data mining
US10108724B2 (en) 2012-08-02 2018-10-23 Rule 14 Real-time and adaptive data mining
US10108725B2 (en) 2012-08-02 2018-10-23 Rule 14 Real-time and adaptive data mining
US10102257B2 (en) 2012-08-02 2018-10-16 Rule 14 Real-time and adaptive data mining
US10108679B2 (en) 2012-08-02 2018-10-23 Rule 14 Real-time and adaptive data mining
US11048712B2 (en) * 2012-08-02 2021-06-29 Rule 14 Real-time and adaptive data mining
US10108678B2 (en) 2012-08-02 2018-10-23 Rule 14 Real-time and adaptive data mining
US10114870B2 (en) 2012-08-02 2018-10-30 Rule 14 Real-time and adaptive data mining
US9229977B2 (en) * 2012-08-02 2016-01-05 Rule 14 Real-time and adaptive data mining
US10114871B2 (en) 2012-08-02 2018-10-30 Rule 14 Real-time and adaptive data mining
US10120911B2 (en) 2012-08-02 2018-11-06 Rule 14 Real-time and adaptive data mining
US10108713B2 (en) 2012-08-02 2018-10-23 Rule 14 Real-time and adaptive data mining
US10108723B2 (en) 2012-08-02 2018-10-23 Rule 14 Real-time and adaptive data mining
US9576574B2 (en) 2012-09-10 2017-02-21 Apple Inc. Context-sensitive handling of interruptions by intelligent digital assistant
US9547647B2 (en) 2012-09-19 2017-01-17 Apple Inc. Voice-based media searching
US9582572B2 (en) * 2012-12-19 2017-02-28 Intel Corporation Personalized search library based on continual concept correlation
KR20230137475A (en) 2013-02-07 2023-10-04 애플 인크. Voice trigger for a digital assistant
US10652394B2 (en) 2013-03-14 2020-05-12 Apple Inc. System and method for processing voicemail
US9368114B2 (en) 2013-03-14 2016-06-14 Apple Inc. Context-sensitive handling of interruptions
AU2014233517B2 (en) 2013-03-15 2017-05-25 Apple Inc. Training an at least partial voice command system
US10748529B1 (en) 2013-03-15 2020-08-18 Apple Inc. Voice activated device for use with a voice-based digital assistant
WO2014144579A1 (en) 2013-03-15 2014-09-18 Apple Inc. System and method for updating an adaptive speech recognition model
CN103164537B (en) * 2013-04-09 2016-01-13 浙江鸿程计算机系统有限公司 A kind of method of search engine logs data mining of user oriented information requirement
US9582608B2 (en) 2013-06-07 2017-02-28 Apple Inc. Unified ranking with entropy-weighted information for phrase-based semantic auto-completion
WO2014197336A1 (en) 2013-06-07 2014-12-11 Apple Inc. System and method for detecting errors in interactions with a voice-based digital assistant
WO2014197334A2 (en) 2013-06-07 2014-12-11 Apple Inc. System and method for user-specified pronunciation of words for speech synthesis and recognition
WO2014197335A1 (en) 2013-06-08 2014-12-11 Apple Inc. Interpreting and acting upon commands that involve sharing information with remote devices
EP3937002A1 (en) 2013-06-09 2022-01-12 Apple Inc. Device, method, and graphical user interface for enabling conversation persistence across two or more instances of a digital assistant
US10176167B2 (en) 2013-06-09 2019-01-08 Apple Inc. System and method for inferring user intent from speech inputs
US9646062B2 (en) 2013-06-10 2017-05-09 Microsoft Technology Licensing, Llc News results through query expansion
AU2014278595B2 (en) 2013-06-13 2017-04-06 Apple Inc. System and method for emergency calls initiated by voice command
DE112014003653B4 (en) 2013-08-06 2024-04-18 Apple Inc. Automatically activate intelligent responses based on activities from remote devices
CN103455564B (en) * 2013-08-15 2018-11-13 复旦大学 It is a kind of that the diversified method of inquiry lexical item is made according to topic information in wikipedia
CN104516903A (en) * 2013-09-29 2015-04-15 北大方正集团有限公司 Keyword extension method and system and classification corpus labeling method and system
US10296160B2 (en) 2013-12-06 2019-05-21 Apple Inc. Method for extracting salient dialog usage from live data
US9965521B1 (en) * 2014-02-05 2018-05-08 Google Llc Determining a transition probability from one or more past activity indications to one or more subsequent activity indications
US9881010B1 (en) * 2014-05-12 2018-01-30 Google Inc. Suggestions based on document topics
US9620105B2 (en) 2014-05-15 2017-04-11 Apple Inc. Analyzing audio input for efficient speech and music recognition
US10592095B2 (en) 2014-05-23 2020-03-17 Apple Inc. Instantaneous speaking of content on touch devices
US9502031B2 (en) 2014-05-27 2016-11-22 Apple Inc. Method for supporting dynamic grammars in WFST-based ASR
US9785630B2 (en) 2014-05-30 2017-10-10 Apple Inc. Text prediction using combined word N-gram and unigram language models
US10289433B2 (en) 2014-05-30 2019-05-14 Apple Inc. Domain specific language for encoding assistant dialog
AU2015266863B2 (en) 2014-05-30 2018-03-15 Apple Inc. Multi-command single utterance input method
US9842101B2 (en) 2014-05-30 2017-12-12 Apple Inc. Predictive conversion of language input
US9734193B2 (en) 2014-05-30 2017-08-15 Apple Inc. Determining domain salience ranking from ambiguous words in natural speech
US10078631B2 (en) 2014-05-30 2018-09-18 Apple Inc. Entropy-guided text prediction using combined word and character n-gram language models
US9715875B2 (en) 2014-05-30 2017-07-25 Apple Inc. Reducing the need for manual start/end-pointing and trigger phrases
US10170123B2 (en) 2014-05-30 2019-01-01 Apple Inc. Intelligent assistant for home automation
US9760559B2 (en) 2014-05-30 2017-09-12 Apple Inc. Predictive text input
US9430463B2 (en) 2014-05-30 2016-08-30 Apple Inc. Exemplar-based natural language processing
US9633004B2 (en) 2014-05-30 2017-04-25 Apple Inc. Better resolution when referencing to concepts
US9338493B2 (en) 2014-06-30 2016-05-10 Apple Inc. Intelligent automated assistant for TV user interactions
US10659851B2 (en) 2014-06-30 2020-05-19 Apple Inc. Real-time digital assistant knowledge updates
US10592539B1 (en) * 2014-07-11 2020-03-17 Twitter, Inc. Trends in a messaging platform
US10601749B1 (en) 2014-07-11 2020-03-24 Twitter, Inc. Trends in a messaging platform
US10446141B2 (en) 2014-08-28 2019-10-15 Apple Inc. Automatic speech recognition based on user feedback
WO2016036345A1 (en) * 2014-09-02 2016-03-10 Hewlett-Packard Development Company, L. P. External resource identification
US9818400B2 (en) 2014-09-11 2017-11-14 Apple Inc. Method and apparatus for discovering trending terms in speech requests
US10789041B2 (en) 2014-09-12 2020-09-29 Apple Inc. Dynamic thresholds for always listening speech trigger
US9606986B2 (en) 2014-09-29 2017-03-28 Apple Inc. Integrated word N-gram and class M-gram language models
US9886432B2 (en) 2014-09-30 2018-02-06 Apple Inc. Parsimonious handling of word inflection via categorical stem + suffix N-gram language models
US10127911B2 (en) 2014-09-30 2018-11-13 Apple Inc. Speaker identification and unsupervised speaker adaptation techniques
US9646609B2 (en) 2014-09-30 2017-05-09 Apple Inc. Caching apparatus for serving phonetic pronunciations
US10074360B2 (en) 2014-09-30 2018-09-11 Apple Inc. Providing an indication of the suitability of speech recognition
US9668121B2 (en) 2014-09-30 2017-05-30 Apple Inc. Social reminders
US10552013B2 (en) 2014-12-02 2020-02-04 Apple Inc. Data detection
US9711141B2 (en) 2014-12-09 2017-07-18 Apple Inc. Disambiguating heteronyms in speech synthesis
US9865280B2 (en) 2015-03-06 2018-01-09 Apple Inc. Structured dictation using intelligent automated assistants
US10152299B2 (en) 2015-03-06 2018-12-11 Apple Inc. Reducing response latency of intelligent automated assistants
US9721566B2 (en) 2015-03-08 2017-08-01 Apple Inc. Competing devices responding to voice triggers
US9886953B2 (en) 2015-03-08 2018-02-06 Apple Inc. Virtual assistant activation
US10567477B2 (en) 2015-03-08 2020-02-18 Apple Inc. Virtual assistant continuity
US9899019B2 (en) 2015-03-18 2018-02-20 Apple Inc. Systems and methods for structured stem and suffix language models
US9842105B2 (en) 2015-04-16 2017-12-12 Apple Inc. Parsimonious continuous-space phrase representations for natural language processing
US9424321B1 (en) * 2015-04-27 2016-08-23 Altep, Inc. Conceptual document analysis and characterization
US10460227B2 (en) 2015-05-15 2019-10-29 Apple Inc. Virtual assistant in a communication session
US10200824B2 (en) 2015-05-27 2019-02-05 Apple Inc. Systems and methods for proactively identifying and surfacing relevant content on a touch-sensitive device
US10083688B2 (en) 2015-05-27 2018-09-25 Apple Inc. Device voice control for selecting a displayed affordance
US10127220B2 (en) 2015-06-04 2018-11-13 Apple Inc. Language identification from short strings
US10101822B2 (en) 2015-06-05 2018-10-16 Apple Inc. Language input correction
US9578173B2 (en) 2015-06-05 2017-02-21 Apple Inc. Virtual assistant aided communication with 3rd party service in a communication session
US10186254B2 (en) 2015-06-07 2019-01-22 Apple Inc. Context-based endpoint detection
US11025565B2 (en) 2015-06-07 2021-06-01 Apple Inc. Personalized prediction of responses for instant messaging
US10255907B2 (en) 2015-06-07 2019-04-09 Apple Inc. Automatic accent detection using acoustic models
CN105094315B (en) * 2015-06-25 2018-03-06 百度在线网络技术（北京）有限公司 The method and apparatus of human-machine intelligence's chat based on artificial intelligence
US20160378747A1 (en) 2015-06-29 2016-12-29 Apple Inc. Virtual assistant for media playback
WO2017018736A1 (en) * 2015-07-24 2017-02-02 Samsung Electronics Co., Ltd. Method for automatically generating dynamic index for content displayed on electronic device
US10671428B2 (en) 2015-09-08 2020-06-02 Apple Inc. Distributed personal assistant
US10747498B2 (en) 2015-09-08 2020-08-18 Apple Inc. Zero latency digital assistant
US10331312B2 (en) 2015-09-08 2019-06-25 Apple Inc. Intelligent automated assistant in a media environment
US10740384B2 (en) 2015-09-08 2020-08-11 Apple Inc. Intelligent automated assistant for media search and playback
US9697820B2 (en) 2015-09-24 2017-07-04 Apple Inc. Unit-selection text-to-speech synthesis using concatenation-sensitive neural networks
US10366158B2 (en) 2015-09-29 2019-07-30 Apple Inc. Efficient word encoding for recurrent neural network language models
US11010550B2 (en) 2015-09-29 2021-05-18 Apple Inc. Unified language modeling framework for word prediction, auto-completion and auto-correction
US11587559B2 (en) 2015-09-30 2023-02-21 Apple Inc. Intelligent device identification
US10691473B2 (en) 2015-11-06 2020-06-23 Apple Inc. Intelligent automated assistant in a messaging environment
US10956666B2 (en) 2015-11-09 2021-03-23 Apple Inc. Unconventional virtual assistant interactions
US10049668B2 (en) 2015-12-02 2018-08-14 Apple Inc. Applying neural network language models to weighted finite state transducers for automatic speech recognition
US10223066B2 (en) 2015-12-23 2019-03-05 Apple Inc. Proactive assistance based on dialog communication between devices
US10446143B2 (en) 2016-03-14 2019-10-15 Apple Inc. Identification of voice inputs providing credentials
CN107193832A (en) * 2016-03-15 2017-09-22 北京京东尚科信息技术有限公司 Similarity method for digging and device
US9934775B2 (en) 2016-05-26 2018-04-03 Apple Inc. Unit-selection text-to-speech synthesis based on predicted concatenation parameters
US9972304B2 (en) 2016-06-03 2018-05-15 Apple Inc. Privacy preserving distributed evaluation framework for embedded personalized systems
US11227589B2 (en) 2016-06-06 2022-01-18 Apple Inc. Intelligent list reading
US10249300B2 (en) 2016-06-06 2019-04-02 Apple Inc. Intelligent list reading
US10049663B2 (en) 2016-06-08 2018-08-14 Apple, Inc. Intelligent automated assistant for media exploration
DK179588B1 (en) 2016-06-09 2019-02-22 Apple Inc. Intelligent automated assistant in a home environment
US10192552B2 (en) 2016-06-10 2019-01-29 Apple Inc. Digital assistant providing whispered speech
US10586535B2 (en) 2016-06-10 2020-03-10 Apple Inc. Intelligent digital assistant in a multi-tasking environment
US10490187B2 (en) 2016-06-10 2019-11-26 Apple Inc. Digital assistant providing automated status report
US10509862B2 (en) 2016-06-10 2019-12-17 Apple Inc. Dynamic phrase expansion of language input
US10067938B2 (en) 2016-06-10 2018-09-04 Apple Inc. Multilingual word prediction
DK179415B1 (en) 2016-06-11 2018-06-14 Apple Inc Intelligent device arbitration and control
DK201670540A1 (en) 2016-06-11 2018-01-08 Apple Inc Application integration with a digital assistant
DK179343B1 (en) 2016-06-11 2018-05-14 Apple Inc Intelligent task discovery
DK179049B1 (en) 2016-06-11 2017-09-18 Apple Inc Data driven natural language event detection and classification
US10318562B2 (en) 2016-07-27 2019-06-11 Google Llc Triggering application information
US10474753B2 (en) 2016-09-07 2019-11-12 Apple Inc. Language identification using recurrent neural networks
US10043516B2 (en) 2016-09-23 2018-08-07 Apple Inc. Intelligent automated assistant
US20180137178A1 (en) * 2016-11-11 2018-05-17 International Business Machines Corporation Accessing data and performing a data processing command on the data with a single user input
US11250074B2 (en) * 2016-11-30 2022-02-15 Microsoft Technology Licensing, Llc Auto-generation of key-value clusters to classify implicit app queries and increase coverage for existing classified queries
US11281993B2 (en) 2016-12-05 2022-03-22 Apple Inc. Model and ensemble compression for metric learning
US10593346B2 (en) 2016-12-22 2020-03-17 Apple Inc. Rank-reduced token representation for automatic speech recognition
US11204787B2 (en) 2017-01-09 2021-12-21 Apple Inc. Application integration with a digital assistant
US10417266B2 (en) 2017-05-09 2019-09-17 Apple Inc. Context-aware ranking of intelligent response suggestions
DK201770383A1 (en) 2017-05-09 2018-12-14 Apple Inc. User interface for correcting recognition errors
US10726832B2 (en) 2017-05-11 2020-07-28 Apple Inc. Maintaining privacy of personal information
DK201770439A1 (en) 2017-05-11 2018-12-13 Apple Inc. Offline personal assistant
US10395654B2 (en) 2017-05-11 2019-08-27 Apple Inc. Text normalization based on a data-driven learning network
DK179496B1 (en) 2017-05-12 2019-01-15 Apple Inc. USER-SPECIFIC Acoustic Models
DK201770429A1 (en) 2017-05-12 2018-12-14 Apple Inc. Low-latency intelligent automated assistant
US11301477B2 (en) 2017-05-12 2022-04-12 Apple Inc. Feedback analysis of a digital assistant
DK179745B1 (en) 2017-05-12 2019-05-01 Apple Inc. SYNCHRONIZATION AND TASK DELEGATION OF A DIGITAL ASSISTANT
DK201770432A1 (en) 2017-05-15 2018-12-21 Apple Inc. Hierarchical belief states for digital assistants
DK201770431A1 (en) 2017-05-15 2018-12-20 Apple Inc. Optimizing dialogue policy decisions for digital assistants using implicit feedback
US10303715B2 (en) 2017-05-16 2019-05-28 Apple Inc. Intelligent automated assistant for media exploration
US20180336892A1 (en) 2017-05-16 2018-11-22 Apple Inc. Detecting a trigger of a digital assistant
US10403278B2 (en) 2017-05-16 2019-09-03 Apple Inc. Methods and systems for phonetic matching in digital assistant services
US10311144B2 (en) 2017-05-16 2019-06-04 Apple Inc. Emoji word sense disambiguation
DK179560B1 (en) 2017-05-16 2019-02-18 Apple Inc. Far-field extension for digital assistant services
US10657328B2 (en) 2017-06-02 2020-05-19 Apple Inc. Multi-task recurrent neural network architecture for efficient morphology handling in neural language modeling
US10445429B2 (en) 2017-09-21 2019-10-15 Apple Inc. Natural language understanding using vocabularies with compressed serialized tries
US10755051B2 (en) 2017-09-29 2020-08-25 Apple Inc. Rule-based natural language processing
KR102042896B1 (en) 2017-10-19 2019-11-27 주식회사 인텔리콘 연구소 System and method for searching electronic information using visualization of related terms with user interaction and storage means
US10636424B2 (en) 2017-11-30 2020-04-28 Apple Inc. Multi-turn canned dialog
US10733982B2 (en) 2018-01-08 2020-08-04 Apple Inc. Multi-directional dialog
US10733375B2 (en) 2018-01-31 2020-08-04 Apple Inc. Knowledge-based framework for improving natural language understanding
US10789959B2 (en) 2018-03-02 2020-09-29 Apple Inc. Training speaker recognition models for digital assistants
US10592604B2 (en) 2018-03-12 2020-03-17 Apple Inc. Inverse text normalization for automatic speech recognition
US10818288B2 (en) 2018-03-26 2020-10-27 Apple Inc. Natural assistant interaction
US10909331B2 (en) 2018-03-30 2021-02-02 Apple Inc. Implicit identification of translation payload with neural machine translation
CN110516033A (en) * 2018-05-04 2019-11-29 北京京东尚科信息技术有限公司 A kind of method and apparatus calculating user preference
US10928918B2 (en) 2018-05-07 2021-02-23 Apple Inc. Raise to speak
US11145294B2 (en) 2018-05-07 2021-10-12 Apple Inc. Intelligent automated assistant for delivering content from user experiences
US10984780B2 (en) 2018-05-21 2021-04-20 Apple Inc. Global semantic word embeddings using bi-directional recurrent neural networks
DK180639B1 (en) 2018-06-01 2021-11-04 Apple Inc DISABILITY OF ATTENTION-ATTENTIVE VIRTUAL ASSISTANT
US10892996B2 (en) 2018-06-01 2021-01-12 Apple Inc. Variable latency device coordination
DK201870355A1 (en) 2018-06-01 2019-12-16 Apple Inc. Virtual assistant operation in multi-device environments
DK179822B1 (en) 2018-06-01 2019-07-12 Apple Inc. Voice interaction at a primary device to access call functionality of a companion device
US11386266B2 (en) 2018-06-01 2022-07-12 Apple Inc. Text correction
US10504518B1 (en) 2018-06-03 2019-12-10 Apple Inc. Accelerated task performance
US11010561B2 (en) 2018-09-27 2021-05-18 Apple Inc. Sentiment prediction from textual data
US11170166B2 (en) 2018-09-28 2021-11-09 Apple Inc. Neural typographical error modeling via generative adversarial networks
US10839159B2 (en) 2018-09-28 2020-11-17 Apple Inc. Named entity normalization in a spoken dialog system
US11462215B2 (en) 2018-09-28 2022-10-04 Apple Inc. Multi-modal inputs for voice commands
US11475898B2 (en) 2018-10-26 2022-10-18 Apple Inc. Low-latency multi-speaker speech recognition
US11638059B2 (en) 2019-01-04 2023-04-25 Apple Inc. Content playback on multiple devices
CN109829104B (en) * 2019-01-14 2022-12-16 华中师范大学 Semantic similarity based pseudo-correlation feedback model information retrieval method and system
US11132358B2 (en) * 2019-02-19 2021-09-28 International Business Machines Corporation Candidate name generation
US11226972B2 (en) 2019-02-19 2022-01-18 International Business Machines Corporation Ranking collections of document passages associated with an entity name by relevance to a query
US11348573B2 (en) 2019-03-18 2022-05-31 Apple Inc. Multimodality in digital assistant systems
US11307752B2 (en) 2019-05-06 2022-04-19 Apple Inc. User configurable task triggers
US11423908B2 (en) 2019-05-06 2022-08-23 Apple Inc. Interpreting spoken requests
US11475884B2 (en) 2019-05-06 2022-10-18 Apple Inc. Reducing digital assistant latency when a language is incorrectly determined
DK201970509A1 (en) 2019-05-06 2021-01-15 Apple Inc Spoken notifications
US11140099B2 (en) 2019-05-21 2021-10-05 Apple Inc. Providing message response suggestions
US11289073B2 (en) 2019-05-31 2022-03-29 Apple Inc. Device text to speech
DK180129B1 (en) 2019-05-31 2020-06-02 Apple Inc. User activity shortcut suggestions
DK201970511A1 (en) 2019-05-31 2021-02-15 Apple Inc Voice identification in digital assistant systems
US11496600B2 (en) 2019-05-31 2022-11-08 Apple Inc. Remote execution of machine-learned models
US11360641B2 (en) 2019-06-01 2022-06-14 Apple Inc. Increasing the relevance of new available information
KR102304050B1 (en) * 2019-09-23 2021-09-24 네이버 주식회사 Method for providing Question and Answer service and web server using the same
WO2021056255A1 (en) 2019-09-25 2021-04-01 Apple Inc. Text detection using global geometry estimators
US11038934B1 (en) 2020-05-11 2021-06-15 Apple Inc. Digital assistant hardware abstraction
US11755276B2 (en) 2020-05-12 2023-09-12 Apple Inc. Reducing description length based on confidence
US11880660B2 (en) * 2021-02-22 2024-01-23 Microsoft Technology Licensing, Llc Interpreting text classifier results with affiliation and exemplification

Citations (14)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6182066B1 (en) 1997-11-26 2001-01-30 International Business Machines Corp. Category processing of query topics and electronic document content topics
US6701305B1 (en) * 1999-06-09 2004-03-02 The Boeing Company Methods, apparatus and computer program products for information retrieval and document classification utilizing a multidimensional subspace
US20070005646A1 (en) * 2005-06-30 2007-01-04 Microsoft Corporation Analysis of topic dynamics of web search
KR20070008991A (en) 2005-07-14 2007-01-18 주식회사 케이티 Text category classification apparatus and its method
KR20070040162A (en) 2005-10-11 2007-04-16 주식회사 코리아 와이즈넛 System and method for offering searching service based on topics
US20070239707A1 (en) * 2006-04-03 2007-10-11 Collins John B Method of searching text to find relevant content
KR20080096887A (en) 2007-04-30 2008-11-04 주식회사 온네트 Ranking system based on user's attention and the method thereof
KR20090000010A (en) 2006-12-14 2009-01-07 엔에이치엔(주) Method for classifing category of keyword and system for executing the method
KR20090010752A (en) 2007-07-24 2009-01-30 엔에이치엔(주) System and method for generating relating data class
KR20090013367A (en) 2007-08-01 2009-02-05 주식회사 다음커뮤니케이션 System and method for recommending a keyword according to each category
US20090125505A1 (en) * 2007-11-13 2009-05-14 Kosmix Corporation Information retrieval using category as a consideration
US20100161605A1 (en) * 2008-12-23 2010-06-24 Yahoo! Inc. Context transfer in search advertising
US7769751B1 (en) * 2006-01-17 2010-08-03 Google Inc. Method and apparatus for classifying documents based on user inputs
US7814085B1 (en) * 2004-02-26 2010-10-12 Google Inc. System and method for determining a composite score for categorized search results

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2005267095A (en) * 2004-03-17 2005-09-29 Nippon Telegr & Teleph Corp <Ntt> Information display method and device, and information display program

Patent Citations (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6182066B1 (en) 1997-11-26 2001-01-30 International Business Machines Corp. Category processing of query topics and electronic document content topics
US6701305B1 (en) * 1999-06-09 2004-03-02 The Boeing Company Methods, apparatus and computer program products for information retrieval and document classification utilizing a multidimensional subspace
US7814085B1 (en) * 2004-02-26 2010-10-12 Google Inc. System and method for determining a composite score for categorized search results
US20070005646A1 (en) * 2005-06-30 2007-01-04 Microsoft Corporation Analysis of topic dynamics of web search
WO2007005465A2 (en) 2005-06-30 2007-01-11 Microsoft Corporation Analysis of topic dynamics of web search
KR20070008991A (en) 2005-07-14 2007-01-18 주식회사 케이티 Text category classification apparatus and its method
KR20070040162A (en) 2005-10-11 2007-04-16 주식회사 코리아 와이즈넛 System and method for offering searching service based on topics
US7769751B1 (en) * 2006-01-17 2010-08-03 Google Inc. Method and apparatus for classifying documents based on user inputs
US20070239707A1 (en) * 2006-04-03 2007-10-11 Collins John B Method of searching text to find relevant content
KR20090000010A (en) 2006-12-14 2009-01-07 엔에이치엔(주) Method for classifing category of keyword and system for executing the method
KR20080096887A (en) 2007-04-30 2008-11-04 주식회사 온네트 Ranking system based on user's attention and the method thereof
KR20090010752A (en) 2007-07-24 2009-01-30 엔에이치엔(주) System and method for generating relating data class
KR20090013367A (en) 2007-08-01 2009-02-05 주식회사 다음커뮤니케이션 System and method for recommending a keyword according to each category
US20090125505A1 (en) * 2007-11-13 2009-05-14 Kosmix Corporation Information retrieval using category as a consideration
US20100161605A1 (en) * 2008-12-23 2010-06-24 Yahoo! Inc. Context transfer in search advertising

Non-Patent Citations (28)

* Cited by examiner, † Cited by third party
Title
"Advertising Keyword Suggestion based on Concept Hierarchy" by Yifan Chen et al, Inf. Conf. of Web Search and Data Mining, 2008); 20 pp.
"Analysis of Topic Dynamics in Web Search" by Xuehua Shen et al, Int. Conf. of World Wide Web, 2005; 2 pp.
Andre et al., "From X-Rays to Silly Putty via Uranus: Serendipity and its Role in Web Seaerch", CHI 2009, Apr. 4-9, 2009, 4 pgs, Abstract.
Baeza-Yates et al., "Extracting semantic relations from query logs", In Proc. of SIGKDD'07, pp. 76-85, 2007, Abstract.
Baeza-Yates et al., "The anatomy of a large query graph", J. of Physics A: Mathematical and Theoretical, 2008, Abstract.
Baeza-Yates et al., "The Intention Behind Web Queries", In Proc. of SPIRE'06, pp. 98-109, 2006. Abstract.
Broder et al., "A semantic approach to contextual advertising", In Proc. of ACM SIGIR'07, pp. 559-566, 2007, Abstract.
Broder, "A taxonomy of web serach", ACM SIGIR Forum vol (36), Isswue (2), pp. 3-10, 2002, Abstract.
Chen et al., "Advertising keyword suggestion based on concept hierarchy", In Proc. of ACM WSDM'08, pp. 251-260, 2008, Abstract.
Cho et al., "Impact of Search Engines on Page Popularity" WWW2004, May 17-22, 2004, 10 pgs.
Craswell et al., "Random Walks on teh Click Graph", In Proc. of ACM SIGIR'07, pp. 239-246, 2007.
Cui et al., "Probabilistic query expansion using query logs", In Proc. of WWW'02, pp. 325-332, 2002, Abstract.
Fang et al., "An exploration of axiomatic approaches to information retrieval", In Proc. of ACM SIGIR'05, pp. 480-487, 2005, Abstract.
Fonseca et al., "Concept-based interactive query expansion", In Proc. of CIKM'05, pp. 696-703, 2005, Abstract.
Kawamae et al., "Query and Content Suggestion Based on Latent Interest and Topic Class", WWW 2004, May 17-22, 2004, pp. 350-351, Abstract.
Li et al., "Learning query intent from regularized click graphs", In Proc. of ACM SIGIR'08, pp. 339-346, 2008, Abstract.
Ma et al., "Learning latent semantic relations from clickthrough data for query suggestion", In Proc. of ACM CIKM'08, pp. 709-718, 2008, Abstract.
Nettleton, et al., "Analysis of Web Search Engine Query Session and Clicked Documents", In Proc. of WebKDD'06, pp. 207-226, Abstract, 2007.
Parikh et al., "Inferring semantic query relations from collective user behavior", In Proc. of CIKM'08, pp. 349-358, 2008, Abstract.
Robust classification of rare queries using web knowledge, ACM Portal, 2007, pp. 231-238, Abstract.
Schonhofen, "Identifying Document Topics Using the Wikipedia Category Network", ACM Portal, IEEE Computer Society, 2006, pp. 456-462, Abstract.
Sebastiani, "Machine learning in automated text categorization", ACM Computing Survey, 34(1):1-47, 2002, Abstract.
Shen et al., "Analysis of Topic Dynamics in Web Search", In Proc. of WWW'05, pp. 1102-1103, 2005.
Shen et al., "Building bridges for web query classification", In Proc. of ACM SIGIR'06, pp. 131-138, 2006, Abstract.
Xing et al., "Deep classifier: Automatically Categorizing Search Results into Large-Scale Hierarchies", In Proc. of ACM WSDM'08, pp. 139-148, 2008, Abstract.
Xue et al., "Deep Classificaiton in Large-scale Text Hierarchies", In Proc. of ACM SIRIR'08, pp. 619-626, Jul. 20-24, 2008.
Zhang et al., "Query Expansion Based on Topics", In Proc. of FSDK'08, pp. 610-614, 2008, Abstract.
Zobel, "How reliable are the results of large-scale information retrieval experiments?" In Proc. of ACM SIGIR'98, pp. 307-314, 1998, Abstract.

Cited By (16)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130144904A1 (en) * 2010-04-14 2013-06-06 Nhn Corporation Method and system for providing query using an image
US9672282B2 (en) * 2010-04-14 2017-06-06 Naver Corporation Method and system for providing query using an image
US20150278366A1 (en) * 2011-06-03 2015-10-01 Google Inc. Identifying topical entities
US10068022B2 (en) * 2011-06-03 2018-09-04 Google Llc Identifying topical entities
US9043350B2 (en) * 2011-09-22 2015-05-26 Microsoft Technology Licensing, Llc Providing topic based search guidance
US20130080460A1 (en) * 2011-09-22 2013-03-28 Microsoft Corporation Providing topic based search guidance
US10346415B1 (en) 2013-03-14 2019-07-09 Google Inc. Determining question and answer alternatives
US9336269B1 (en) 2013-03-14 2016-05-10 Google Inc. Determining question and answer alternatives
US9582543B2 (en) 2014-04-24 2017-02-28 International Business Machines Corporation Temporal proximity query expansion
US9582544B2 (en) 2014-04-24 2017-02-28 International Business Machines Corporation Temporal proximity query expansion
CN107180111A (en) * 2017-06-13 2017-09-19 深圳市宇数科技有限公司 A kind of information recommendation method, electronic equipment, storage medium and system
CN107180111B (en) * 2017-06-13 2019-10-25 深圳市宇数科技有限公司 A kind of information recommendation method, electronic equipment, storage medium and system
US20190066675A1 (en) * 2017-08-23 2019-02-28 Beijing Baidu Netcom Science And Technology Co., Ltd. Artificial intelligence based method and apparatus for classifying voice-recognized text
US10762901B2 (en) * 2017-08-23 2020-09-01 Beijing Baidu Netcom Science And Technology Co., Ltd. Artificial intelligence based method and apparatus for classifying voice-recognized text
US11526567B2 (en) * 2018-10-17 2022-12-13 International Business Machines Corporation Contextualizing searches in a collaborative session
US11947604B2 (en) * 2020-03-17 2024-04-02 International Business Machines Corporation Ranking of messages in dialogs using fixed point operations

Also Published As

Publication number Publication date
US20110047161A1 (en) 2011-02-24
KR101078864B1 (en) 2011-11-02
KR20100107610A (en) 2010-10-06

Similar Documents

Publication Publication Date Title
US8452798B2 (en) Query and document topic category transition analysis system and method and query expansion-based information retrieval system and method
Atefeh et al. A survey of techniques for event detection in twitter
US8051080B2 (en) Contextual ranking of keywords using click data
US8219577B2 (en) Apparatus and method product for presenting recommended information
US8108405B2 (en) Refining a search space in response to user input
US20090292685A1 (en) Video search re-ranking via multi-graph propagation
US20150095300A1 (en) System and method for mark-up language document rank analysis
Dang et al. A review of text mining techniques associated with various application areas
EP2307951A1 (en) Method and apparatus for relating datasets by using semantic vectors and keyword analyses
KR100954842B1 (en) Method and System of classifying web page using category tag information and Recording medium using by the same
Godoy et al. Hybrid content and tag-based profiles for recommendation in collaborative tagging systems
Nakatsuji et al. Detecting innovative topics based on user-interest ontology
JP4879775B2 (en) Dictionary creation method
JP3921837B2 (en) Information discrimination support device, recording medium storing information discrimination support program, and information discrimination support method
US10380244B2 (en) Server and method for providing content based on context information
Moumtzidou et al. Discovery of environmental nodes in the web
Chang et al. Identifying user goals from Web search results
Rajkumar et al. Users’ click and bookmark based personalization using modified agglomerative clustering for web search engine
KR101110026B1 (en) Apparatus and method for searching local information
JP2003208447A (en) Device, method and program for retrieving document, and medium recorded with program for retrieving document
KR101057075B1 (en) Computer-readable recording media containing information retrieval methods and programs capable of performing the information
Potey et al. Personalization approaches for ranking: A review and research experiments
Gaou et al. Search Engine Optimization to detect user's intent
Kanhabua Time-aware approaches to information retrieval
KR101513660B1 (en) Historical information retrieval system based on period query

Legal Events

Date Code Title Description
AS Assignment

Owner name: KOREA ADVANCED INSTITUTE OF SCIENCE AND TECHNOLOGY

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:MYAENG, SUNG HYON;JUNG, YU CHUL;KIM, KYUNG MIN;REEL/FRAME:023945/0081

Effective date: 20090423

STCF Information on status: patent grant

Free format text: PATENTED CASE

FEPP Fee payment procedure

Free format text: PAYOR NUMBER ASSIGNED (ORIGINAL EVENT CODE: ASPN); ENTITY STATUS OF PATENT OWNER: SMALL ENTITY

FPAY Fee payment

Year of fee payment: 4

MAFP Maintenance fee payment

Free format text: PAYMENT OF MAINTENANCE FEE, 8TH YR, SMALL ENTITY (ORIGINAL EVENT CODE: M2552); ENTITY STATUS OF PATENT OWNER: SMALL ENTITY

Year of fee payment: 8