EP2734967A1 - A recommendation engine that processes data including user data to provide recommendations and explanations for the recommendations to a user - Google Patents

A recommendation engine that processes data including user data to provide recommendations and explanations for the recommendations to a user

Info

Publication number
EP2734967A1
EP2734967A1 EP12815207.1A EP12815207A EP2734967A1 EP 2734967 A1 EP2734967 A1 EP 2734967A1 EP 12815207 A EP12815207 A EP 12815207A EP 2734967 A1 EP2734967 A1 EP 2734967A1
Authority
EP
European Patent Office
Prior art keywords
relevant
user
users
data
server computer
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Withdrawn
Application number
EP12815207.1A
Other languages
German (de)
French (fr)
Other versions
EP2734967A4 (en
Inventor
Christopher Eric Shogo BERNER
Jeremy Ryan SCHIFF
Corey Layne REESE
Paul Kenneth TWOHEY
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
OpenTable Inc
Original Assignee
Ness Computing LLC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Ness Computing LLC filed Critical Ness Computing LLC
Publication of EP2734967A1 publication Critical patent/EP2734967A1/en
Publication of EP2734967A4 publication Critical patent/EP2734967A4/en
Withdrawn legal-status Critical Current

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06QINFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q30/00Commerce
    • G06Q30/06Buying, selling or leasing transactions
    • G06Q30/0601Electronic shopping [e-shopping]
    • G06Q30/0631Item recommendations

Definitions

  • the present invention relates to recommendation systems in general and in particular to recommendation system that collect data to sift through and generate personalized
  • the Internet has become a useful source of information relating to goods and services, but because so many users and companies are connected to the Internet and providing information of their own, it is difficult to find relevant information among all available data. As individuals and groups are now able to connect to the Internet and retrieve information from websites and via apps, and do so perhaps while in transit towards a geographic area of interest, they come to expect the ability to find information about goods and services being offered. [0006]
  • One source of information may be in the form- of websites hosted by or for providers.
  • website refers to a collection of one or more web pages displayable by a browser or other web client, the one or more web pages being hosted on a web server (i.e., a computer coupled to the Internet, or other network over which communication occurs) that responds to requests for pages hosted thereon.
  • a web server i.e., a computer coupled to the Internet, or other network over which communication occurs
  • An example might be a restaurant that generates information about its hours, pictures of its dining room, contact information, history, menus and other offerings, then puts that information into a suitable form (such as HTML pages, dynamically generated pages, etc.), stores those pages in electronic storage accessible by the web server (hosted by the restaurant or perhaps a company that specializes in web presence that provides such services to the restaurant), and then serves those pages as client devices request pages.
  • the information might also be presented in another form, such as data supplied to an application (i.e., app) on the client device rather than a generic browser on the client device.
  • social media websites that provide information to a user regarding the user's friends may also be another source that the user refers to in gathering information and searching for relevant recommendations.
  • social media websites allow users to provide a wide variety of information that may or may not be related to what a user is searching for (e.g., status updates unrelated to a vendor of goods or services), and it is also difficult to cross-reference the recommendations of friends through social media websites with general websites providing reviews, or websites for particular businesses, in a user's search for relevant recommendations. [0009] An improved recommendations system is desirable.
  • Embodiments of the invention relate to an improved computer-implemented method for generating personalized recommendations and providing explanation data related to the personalized recommendations.
  • Existing methods for generating personalized recommendations to a target user may not provide explaining factors and processes used to determine the personalized recommendations. Thus, if the target user is dissatisfied with the personalized recommendations, it is difficult for the target user to feed back to a personalized
  • providing the target user with explanation data also informs the target user of a source from which the personalized recommendation was determined, which may influence the target user's decision to select the personalized recommendation or not.
  • a target user may receive a personalized recommendation to go to Restaurant C and also receive explanation data that Restaurant C was recommended to the target user because a "friend" of the target user, User C, goes there regularly.
  • User C is an ex-boyfriend of the target user, the target user may not want to go to Restaurant C for fear of running into User C, even though it may be likely that the target user would have enjoyed the food at Restaurant C.
  • Embodiments of the invention relate to a method for providing explanation data from a personalized recommendations process for a primary user based at least on stored data about the primary user.
  • the method comprises a server computer obtaining data related to one or more users who are relevant to the primary user, wherein the one or more users relevant to the primary user are similar to the primary user.
  • the server computer determines at least one group of users relevant to the primary user based on the data related to the one or more users, each relevant group including one or more relevant users to the primary user. Then, the server computer also obtains data related to one or more entities.
  • the method further comprises determining, at the server computer, one or more entities relevant to the primary user based on the data related to the one or more entities, and then associating the at least one relevant group of users with the one or more relevant entities.
  • the server computer then generates one or more potential candidate factors based on at least the relevant group of users or the one or more relevant entities.
  • the method may further comprise selecting, by the server computer, a set of factors from the one or more potential candidate factors and displaying the set of selected factors to the primary user, wherein the potential candidate factors are used to determine recommendations or explanations to the primary user.
  • Another embodiment of the invention relates to a server computer comprising a processor and a non-transitory computer readable medium, the non-transitory computer readable medium comprising code executable by the processor to implement a method for generating personalized recommendations for a primary user based at least on stored data about the primary user.
  • the method comprises a server computer obtaining data related to one or more users who are relevant to the primary user, wherein the one or more users relevant to the primary user are similar to the primary user.
  • the server computer determines at least one group of users relevant to the primary user based on the data related to the one or more users, each relevant group including one or more relevant users to the primary user. Then, the server computer also obtains data related to one or more entities.
  • the method further comprises determining, at the server computer, one or more entities relevant to the primary user based on the data related to the one or more entities, and then associating the at least one relevant group of users with the one or more relevant entities.
  • the server computer then generates one or more potential candidate factors based on at least the relevant group of users or the one or more relevant entities.
  • the method performed by the server computer may further comprise selecting, by the server computer, a set of factors from the one or more potential candidate factors and displaying the set of selected factors to the primary user, wherein the potential candidate factors are used to determine recommendations to the primary user.
  • FIG. 1 is a block diagram of a personalized recommendation system according to aspects of the present invention.
  • FIG. 2 is a block diagram of an exemplary flow that may be performed by the personalized recommendation system to explain recommendations according to aspects of the present invention.
  • FIG. 3 is a block diagram of a modularized personalized recommendations system according to aspects of the present invention.
  • FIG. 4 is an illustration of a user display of selected factors to explain recommendations according to aspects of the present invention.
  • FIG. 5 is an illustration of a user display of selected factors and user input to explain and provide recommendations according to aspects of the present invention.
  • systems which recommend entities may use a variety of algorithmic approaches to produce predicted evaluations or recommendations, hereafter referred to as recommendations.
  • recommendations Prior to discussing embodiments of the invention, some terms used herein are described below:
  • Entities may be anything that can be represented or referred to in a computer system.
  • entities are providers of goods and services, such as restaurants or nightclubs.
  • entities are particular goods or services or events, such as a particular dish at a restaurant or a one-time or repeating event such as a concert.
  • entities are documents or sections of documents, such as text pages or musical compositions, to be presented to a user.
  • an entity may be a relationship or potential relationship between two people.
  • a dating recommendation system may recommend in favor of or against a potential relationship between one user and another.
  • entities may be individuals.
  • Recommendations may be along a single or multiple dimensions, and/or over a discrete or continuous domain. Recommendations may be presented in a numerical, textual, pictorial, auditory, video, or tactile fashion. Recommendations may be include any of these characteristics, either alone or in combination. For example, a user may rate a restaurant on a scale of 1 , 2, 3, 4, or 5 stars, with 1 star being the lowest rating, and 5 stars being the highest rating. In another realization, a user may instead rate a restaurant on a 1 -5 star scale for taste rating, 1 -4 "$" scale for price rating, and add an optional one word description.
  • recommendations might be called “evaluations” or used in a similar manner. For example, an evaluation might be done by a team of doctors choosing a picture that best represents the mood or situation of a clinical patient, in effect, "recommending" which is the "best” image to used.
  • Another example of an evaluation or recommendation may include citizen journalists recording a five second video to make a review or recommendation, or an audio summary of a protest or event.
  • recommendations may use three dimensional sensing technologies to have dancers record their expressions in response to listening to a composition.
  • Another example may include a user operating an application on a mobile device, and dragging pictorial representations of dishes into different groups or piles categorized as "tasty", "yucky", and "only sometimes".
  • a recommendation may be inferred from aspects of a user's behavior such as whether the user visits a particular web page, looks in one direction or another, or whether the user walks or turns in one way or another.
  • An "affinity graph” is a model of the affinity between users or entities and users or entities, either in general or with respect to a given attribute or cluster of attributes of the users or entities, as a graph.
  • Affinity graphs may represent relationships between users and entities, users and other users, or entities and other entities, and may be used to determine recommendations.
  • An example of an affinity graph is a social graph wherein nodes represent users of a social interaction system and edges between nodes represent "friend" links. It should be understood, however, that some embodiments may not represent users explicitly and might not require that a person interacting with the application be identified or authenticated.
  • Attributes provide additional information about a node in the graph.
  • an attribute may be the zoom level of a camcorder type or the primary cuisine type of a restaurant.
  • Entities may be represented by nodes, with individual users being one type of entity.
  • Edges represent relationships between nodes.
  • Directional edges can model unidirectional relationships, such as if person A follows person B on Twitter.
  • a symmetric relationship, like friends on Facebook, may be represented as two directional edges, or a unidirectional edge.
  • evaluation data for a user may be, according to embodiments of the invention, inputs into a recommendation service provided by a personalized recommendation system described herein.
  • evaluation data may include ratings on a discrete star scale for the quality of a restaurant for a restaurant discovery website/product or answers to personality questions for a match making website/service.
  • Evaluation data may be associated with one entity or a set of entities. For example, a user may rate Jane's Sushi 90/100 and may rate sushi restaurants in general 82/100.
  • a "recommendations provider” is a service that generates recommendations and predictions of evaluation data of entities for users. The recommendations provider may be on the same device or different devices from the device displaying the recommendations. The recommendations provider may be operated by the same system collecting the evaluation data, or may be operated on a separate system as the system collecting the evaluation data.
  • An "explanations provider” is a service that generates explanations of the
  • the explanations provider may be on the same device or different devices from the device displaying explanations.
  • the explanations provider and the recommendations provider may be operated on the same or on different systems from each other.
  • the explanations provider may be operated on the same system or a separate system as the system collecting the evaluation data.
  • an "unknown user” is a user of the recommendations provider or other service known to the recommendations provider for which the service has access to evaluation data but is not connected to a primary user or group of users viewing a recommendation by an appropriate link or links in the affinity graph. For example, if the affinity graph is a family tree, then Jane and her mother, Joan, would be connected, but Jane's best friend Becky would be an unknown user in evaluating recommendations for Jane and Joan as primary users. Depending on the system or the primary user, a user may be known or unknown in different contexts.
  • a "known user” is a user of the recommendations provider or another service known to the recommendations provider for which the recommender knows evaluation data and which is connected to the user (group of users) viewing a recommendation by an appropriate link or links in the affinity graph. For example Joan is a known user to Jane from the example above.
  • An "explained entity” is a specific entity being recommended to a user or group of users for which recommendations and recommendation explanations are being generated.
  • the explained entity may or may not have associated evaluation data.
  • a "known entity” is an entity with sufficient associated evaluation data to show a user how the known entity has been evaluated. For example, in a system for recommending theater performances, a user may have given the category "Musicals" two thumbs up to indicate approval, thereby making the category a known entity (because the thumbs up are evaluation data that can be shown to a user).
  • Technicolor Dreamcoat is an explained entity (because the previous ratings could be used to create a recommendation for "Joseph and the Amazing Technicolor Dreamcoat”).
  • An "evaluation description” is the language or messaging used to describe evaluation data for an entity. For example, in a system with 1 to 5 star ratings, 1 star might be “I hated it,” 2 stars might be “1 disliked it,” 3 stars might be “I'm okay with it,” 4 stars might be “1 liked it,” and 5 stars might be “I loved it.”
  • the descriptions need not be a direct mapping from an evaluation to prose and may be dependent on the context in which the evaluation description is viewed.
  • Evaluation descriptions may include textual information as well as richer media sources in the same or similar manner as recommendation explanations.
  • a "predicted evaluation description” is the language or messaging used to make predictions about what known or unknown users would evaluate an explained entity. For example, when a recommendations provider for kitchen tiles has an affinity graph that models followers and a primary user following Joe, then when the primary user is viewing an entity describing marble tile that the recommendations provider believes Joe would rate 89/90 on beauty and 75/80 on long term value, a predicted evaluation description might be, "Joe would probably love the marble tile.” Predicted evaluation descriptions may include textual information as well as richer media sources in the same or similar manner as recommendation explanations. For example, with "Joe would probably love the marble tile," a picture of Joe may appear above his name and both the picture and his name may be hyperlinked to a page describing his preferences.
  • the personalized recommendations system 160 may be coupled to a plurality of users, such as a primary user 100, secondary user 102, or any number of users 104. There may be relationships between specific users, for example, the primary user 100 and secondary user 102 may be friends, family members, or co-workers.
  • the plurality of users may be coupled to the personalized recommendations system 160 through associated user devices 1 10, 1 12, and 1 14.
  • User devices may include mobile phones (e.g., a BlackberryTM phone, an iPhoneTM device, an AndroidTM device), tablets (e.g., an iPadTM tablet), or other electronic devices.
  • the personalized recommendations system 160 may obtain inputs from any number of users 100, 102, 104, registered with the personalized recommendations system 160.
  • the users 100, 102, and 104 may be interconnected and have relationships with one another.
  • the inputs may be transmitted through the user devices 1 10, 1 12, 1 14 via an application.
  • the user devices 1 10, 1 12, 1 14 may be connected to the personalized recommendations system 160 through the internet, a wireless network, a telecommunications network, or any suitable communications network.
  • the primary user 100 may use the primary user device 1 10 to download and install an application issued by the personalized recommendations system 160.
  • the primary user 100 may edit personal ratings for entities, such as venues and restaurants, and other personal preferences, such as cuisine, type of food, price range, location, distance from current location, etc.
  • the primary user 100 may also provide feedback items or data through the application to respond to recommendations, to add or remove entities, and/or rate an experience at an entity.
  • entities may include merchants and service providers.
  • Additional inputs to the personalized recommendations system 160 may come from external data sources, for example, a social network computer system such as the FacebookTM computer system 140(A), a public messaging system such as the TwitterTM messaging system 140(B), and/or any number of other data sources 1 40(N) via the user accounts 150(A), 150(B), 152(A), 152(B) with the data sources and/or user devices 1 10, 1 12.
  • the external data sources may also provide raw location data of the users, for example, through a "check-in" through system 140(A), system 140(B), or other social networking provider system capable of providing raw location data.
  • the other data sources 140( ) may themselves have associated user accounts, which may map directly or indirectly to user accounts of other systems.
  • the personalized recommendations system or provider may use affinity graphs as models of the affinity between users or entities and users or entities, either in general or with respect to a given attribute or cluster of attributes of the users or entities, as a graph.
  • Affinity graphs may represent relationships between users and entities, users and other users, or entities and other entities, and may be used to determine recommendations. Attributes provide additional information about a node in the graph. For example, an attribute may be the zoom level of a camcorder type or the primary cuisine type of a restaurant.
  • Entities may be represented by nodes, with individual users being one type of entity.
  • Edges represent relationships between nodes. Directional edges can model unidirectional relationships, such as if person A follows person B on Twitter.
  • a symmetric relationship like friends on Facebook, may be represented as two directional edges, or a unidirectional edge.
  • These relationships may be about different associations other than social affiliations, such as an expertise graph where an edge from user A to user B indicates that user A bel ieves user B to be an expert.
  • There may also be weights on these edges, which for example, might represent the confidence in another user's expertise, or represent how familiar two users are with each other.
  • Affinity graph nodes may be further augmented with additional information such as short form textual messages, pictures, check-ins, other explicit or impl icit annotations that are used to denote a user has visited or prefers a location, or other explicitly tagged content.
  • the affinity graph may be constructed by users choosing to link themselves to other nodes, or by aggregation and analysis methods which infer the links, weights, and/or nodes.
  • a link may be inferred from who sends messages to whom in a network, or links may be taken as given in the form of user-determined friendship relationships and weights on the links inferred from messages, or information may be inferred by who or what a user looks at.
  • the recommendations provider operated on the personal ized recommendations system may also use evaluation data, such as inputs into the recommendation service provided by the personalized recommendation system described herein.
  • evaluation data may include ratings on a discrete star scale for the quality of a restaurant for a restaurant discovery website/product or answers to personality questions for a match making website/service.
  • Evaluation data may be associated with one entity or a set of entities. For example, a user may rate Jane's Sushi 90/ 100 and may rate sushi restaurants in general 82/100.
  • the recommendations provider operated by personalized recommendations system may generate recommendations along a single or multiple dimensions, and/or over a discrete or continuous domain.
  • Recommendations may be numerical, textual, pictorial, auditory, or tactile in fashion.
  • Recommendations may be include any of these characteristics, either alone or in combination. For example, a user may rate a restaurant on a scale of 1 , 2, 3, 4, or 5 stars, with 1 star being the lowest rating, and 5 stars being the highest rating. In another realization, a user may instead rate a restaurant on a 1 -5 star scale for taste rating, 1 -4 "$" scale for price rating, and add an optional one word description.
  • a recommendation involves having a team of doctors choosing a picture that best represents the mood of a clinical patient.
  • Other examples may include recording a five second video to make a review or recommendation, an audio summary of a protest or event, where a user looks or how longs the user looks at something, or in a mobile application, dragging pictorial representations of entities into rating categories.
  • a recommendations provider may construct recommendations using statistical techniques that are known in the literatures of statistics, artificial intelligence, and machine learning.
  • An example of such a technique is using cosine similarity in an application where the ratings are on a 1 -5 star rating scale to determine how similar one entity's set of ratings is to another entity' s set of ratings, and to provide recommendations of the first entity based on the similarity.
  • Another example of such a technique is using latent Dirichlet allocation in an application where the ratings are combinations of one or more words like "yucky,” "tasty,” etc., to model the ratings as words associated with topics that an entity is associated with in a topic model.
  • Another example of such a technique is using natural language processing to extract sentiment information from free-form textual statements related to an entity and using a collaborative filtering technique with the information thus extracted.
  • Example recommendations may be, but are not limited to being, textual (e.g., a short 20 word comment), continuous (e.g., a ratings slider), discrete as in star ratings, pictorial (e.g., a representative picture, drawing, or photograph which most closely resembles the entity), auditory (e.g.. a volume measurement or review), gestural (e.g., a swipe up or down), visual (e.g., an interpretive dance for the entity), or some combination thereof.
  • Recommendations may involve adding edges or weights, either individually or some combination to one or more affinity graphs.
  • the personalized recommendations system presents one or many explanations for why an entity has been included in a recommendation to a primary user or set of users.
  • An explanation of the recommendation may or may not reflect computations performed to generate the recommendation or factors used to perform the computations.
  • Recommendation explanations may be considered as a set of plausible reasons that a given entity or entities would be a good choice for a specific user or group of users.
  • a recommendation is given to a user that is derived from a statistical technique that incorporates large quantities of information, and that uses features of the information that are learned by machine and are not human-interpretable, no aspect of the decision to make the recommendation may be clearly explainable to the user. Instead, the system might construct an explanation by, for example, calculating a similarity score between an entity being recommended and another entity the user has already rated, and reporting on the similarity.
  • FIG. 2 A conceptual architecture of an exemplary system according to an embodiment of an invention is diagrammed in FIG. 2.
  • the personalized recommendations system may have a user account database storing data (e.g., evaluation data) related to all users, including the primary user.
  • the recommendation for the primary user includes
  • the personalized recommendation system may have an entity database of stored data for explained entities 204, such as restaurants, hotels, and other vendors providing goods and/or services to users 202.
  • Explained entities 204 may be entities that are recommended to a primary user or group of users 202 for which recommendations and recommendation explanations are being generated.
  • the explained entity may or may not have associated evaluation data, in other words, it may not be a known entity, which is an entity with sufficient associated evaluation data to show a user how the known entity has been evaluated.
  • the personalized recommendations system may perform similar user and entity selection.
  • Data related to primary user or primary user group 202 and data related to explained entities 204 may be collected, and the personalized recommendations system may use this data to determine similar users 206 and similar entities 208.
  • Similar users and similar entities are identified as relevant groups of users and entities, respectively, relative to the primary user or primary group of users 202 and the entities to be explained 204. In some embodiments, only relevant entities are determined and used to generate personalized recommendations and/or explanations, and data related to similar or relevant users is not used.
  • step S2 the primary user 202, the entities to be explained 204, the similar users 206, and similar entities 208 are used alone or in combination in a process to generate candidate factors for recommendation explanations. From these potential explanation factors the system may determine and select, in step S3, a subset to show to the user. Finally, in step S4, the system crafts a display of the explanation factors or recommendation explanations to present to the primary user.
  • the personalized recommendations explanation system may decide to generate and process the English language HTML explanations simultaneously while generating explanation factors, so that there is no need for display generation after factor selection is done.
  • step S4 may occur simultaneously with step S2 instead of
  • step S 1 locating similar users 206 and similar entities 208 may produce many subsets of all the users and entities known to the personalized
  • recommendations explanation system may produce “user and user,” “user and entity,” and “entity and entity” pairings with optional subsets or supersets of any combination, such as pairing a group of users with a set of groups of users.
  • Locating similar users and entities may take into account attributes of the entities or users previously stored in the entity database and user account database, respectively. Attributes may also be data collected in real-time at the time of a recommendation request of the primary user, such as the time of day, and location information of the primary user, which may be used either alone or in combination with each other or with previously stored data.
  • the user database and entity database could be NoSQL databases, an SQL database, another information retrieval system such as a computer filesystem, or an in-memory data store.
  • the system could have a temporary piece of data obtained from a third-party on demand, such that data does not need to be statically or locally stored.
  • a database could consist of multiple storage facilities, such as a cache backed by a database or a normalized authoritative database and denormalized in-memory stores.
  • Exemplary attributes may include demographic information, user gender, user height, user weight, user income, entity revenues, the cost of items, cuisine type, previous evaluations done by the user, check-ins, other explicit or implicit annotations that are used to denote whether a user has visited or prefers a location or entity, or relations in an affinity graph(s).
  • Demographic data may include gender, height, weight, income, or pre-determined attributes.
  • Additional user attributes may also include explicitly stated preferences.
  • the personalized recommendations explanation system recommending restaurants may allow a user to declare they are one or more of pescetarian, vegetarian, vegan, kosher, lactose intolerant, and wheat gluten intolerant.
  • two restaurants might be similar if they both serve the same or related cuisines, e.g., the system may consider cupcakes and desserts being similar cuisines, while Japanese and Chinese are not.
  • a restaurant recommendation system may also deem two restaurants similar if their respective head chefs both apprenticed at the same restaurant, or if both restaurants are in a zip code with the same median family income and have been open for more than 3 years. Al l these forms of information may be learned or explicit. For example, user attributes such as gender might be inferred from user behavior;,whether a restaurant is Japanese may be inferred or may be received from an external database, and whether Japanese or Chinese should be considered similar or dissimilar may be learned or preprogrammed.
  • the generation and selection of recommendation explanation factors may be computed using a variety of metrics incorporating different spatial, temporal, social, demographic, and user interaction based signals, either alone or in combination.
  • Other examples of data user in selecting recommendation explanation factors include, but are not l imited to, explicit user suggestions, demographic data, location data at the time a user or group of users is seeing an explanation, the time of day a primary user or group of users is requesting a recommendation and viewing recommendation explanation, explicit user suggestion of an explanation or part of an explanation, voting or other group selection of expl icit user suggestions or system generated suggestions of an explanation or part of an explanation, table driven matching.
  • historical data such as query history including click through and other interest actions to a search or discovery service, browsing history, purchase history, the location of the primary user or group of users over time, and information in third party services from previous visits or evaluations, may be used in generating and selecting recommendation explanation factors.
  • the selection of factors may also involve psychological models of behavior which may or may not be culturally dependent. For example, it could be the case that American users respond to explanations better when referred to as "users," while British prefer to be referred to as "patrons.”
  • the selection of factors may also involve functions to be maximized such as a prediction of profitability, user retention, or conversion rate (of a number viewers or visitors into the number of users who take some action, such as making a purchase). A function to be maximized may be predicted for this purpose using statistical techniques (for example, regression analysis).
  • Evaluation descriptions describe evaluation data for an entity, relating to a user' s experience or relationship with that entity. For example, in a system with 1 to 5 star ratings for a restaurant, 1 star indicate that the user "hates” it, 2 stars might indicate that the user “dislikes” it, 3 stars might indicate that the user is “okay” or indifferent to it, 4 stars might indicate that the user “likes” it, and 5 stars might indicate that the user "loves” it.
  • the descriptions may not be a direct mapping from an evaluation to prose and may be dependent on the context in which the evaluation description is viewed.
  • Evaluation descriptions may include textual information as well as richer media sources in the same or similar manner as recommendation explanations.
  • Predicted evaluation descriptions may also be used to make predictions about what known or unknown users would evaluate an explained entity. For example, a recommendations provider for kitchen tiles may have an affinity graph modeling followers and a primary user who is following Joe.
  • a predicted evaluation description might be, "Joe would probably love the marble tile.”
  • Predicted evaluation descriptions may include textual information as wel l as richer media sources in the same or similar manner as recommendation explanations. For example, with "Joe would probably love the marble tile," a picture of Joe may appear above his name and both the picture and his name may be hyperlinked to a page describing his preferences.
  • FIG. 3 shows a block diagram of an exemplary modular personalized recommendations system 160 according to an embodiment of the invention, where the personalized
  • recommendations system 160 is configured to provide explanations of the personalized recommendations.
  • an explanations provider may be separate from the recommendations provider, and may be operated by different systems.
  • the personalized recommendations system 160 operates as the recommendations provider and explanation (i.e., recommendations explanation) provider.
  • the personalized recommendations system 160 may be a server computer 700, which may be a single computer, a cluster of computers, a server, or a super computer. Processes and methods implemented by or at the server computer could also be implemented by or at a client, in various embodiments.
  • the server computer 700 may comprise a processor 740, and a non- transitory computer readable medium 750, on which code is stored.
  • There may be several databases to store data for example, an entity database 730(a), a factor database 730(b), a user database 730(c), and a media object database 730(d).
  • Stored in the entity database 730(a) may be evaluation data relating to entities, including entity meta-data (e.g., location, hours of operation, specialty, rating).
  • Entity meta-data may include any sub-category of data related to the entity.
  • the factor database 730(b) may store previously determined and selected factors from a plurality of users.
  • the user database 730(c) may store user data obtained from a plurality of users, such as real-time data from social networking sites (e.g., tweets, status updates), and stored data related to a plurality of users, such as user preferences and settings (e.g., gender, dietary restrictions).
  • the real-time data may be explicit or implicit, and may be inferred or its significance or relevance may be inferred, and may include text, images, video, audio, or other media.
  • the user data may be directly obtained from the user or may be obtained from external data sources, such as a social networking site associated with the user.
  • the personalized recommendations system 160 may also include a media object database 730(d), storing media objects, such as high resolution photos.
  • the media objects may be associated with entities, recommendations, and/or the target user, and may be used in the display of recommendations and/or recommendation explanations.
  • the association between media objects and entities, recommendations, or a user may be inferred or may be preprogrammed or derived from a preexisting database.
  • the code stored on the non-transitory computer readable medium 750 may be in modules, and is executable by the processor 740 to perform functions, which are implemented as processing steps described above by the personalized recommendations system providing recommendation explanations.
  • the non-transitory computer-readable medium 750 may comprise software modules, such as a similar user selection module 701 , a similar entity selection module 702, a factor generation module 703, a factor selection module 704, a factor display module 705, and an explanation module 706. The functions of these modules are described in the process flows described above and in FIG. 2.
  • the personalized recommendations system 160 may include a network interface 720.
  • the network interface 720 includes hardware and software that enables the personalized recommendations system 160 to transmit and receive data over the Internet or any other communications network (e.g., telecommunications network, cable, radio).
  • the recommendations and recommendation explanations may be displayed.
  • the display of recommendation explanations performed in step S4 may be in the form of textual information on a screen of a mobile device.
  • the means by which information is communicated is not limited to textual information or to a device screen, but may include instead or in addition richer media sources or forms of communication, such as two or three dimensional models, pictures, a heads-up display, video, hyperlinked text, web pages, styled text, recorded sounds, computer generated sounds, vibrations of a tactile device, pressure exerted on the user by haptic technology, electric shocks or pulses, the control of remote devices, and the generation of scents.
  • the system may explain a
  • recommendation for an Italian restaurant by indicating that five of a primary user's friends go to the restaurant and then providing pictures of the five friends, where each picture is linked to more information from each friend, such as possibly including the friend's evaluation (e.g., review) of the restaurant.
  • the display of recommendation explanations may be visible or not depending on external context, for example, not showing recommendations about user's cuisine preferences to other users until that primary user has rated at least ten places on a 1 -5 scale.
  • the explanation of display generation may be computed using a variety of metrics incorporating different spatial, temporal, social, demographic, and user interaction based signals, either alone or in combination.
  • data used in explaining display generation include, but are not limited to, explicit user suggestions, demographic data, location data at the time a user or group of users is seeing an explanation, the time of day a primary user or group of users is requesting a recommendation and viewing recommendation explanation, explicit user suggestion of an explanation or part of an explanation, voting or other group selection of explicit user suggestions or system generated suggestions of an explanation or part of an explanation, table driven matching.
  • historical data such as query history (including click through and other interest actions to a search or discovery service), browsing history, purchase history, the location of the primary user or group of users over time, and information in third party services from previous visits or evaluations, may be used in generating and selecting recommendation explanation factors.
  • the selection of factors may also involve psychological models of behavior which may or may not be cultural ly dependent. For example, it could be the case that American users respond to explanations better when referred to as "users,” while British prefer to be referred to as "patrons.”
  • the personalized recommendations system may count or aggregate a number of users with a similar actual or predicted evaluation. When generating explanations, incorporating this number the number may be shown directly to the user(s) or a description or summary may be shown instead.
  • the language or messaging used is known as a numeric descriptor and may include textual information as well as richer media sources in the same or similar manner as discussed earlier in regards to recommendation explanations.
  • FIG. 4 shows an illustration of a user display 400 in which a recommendation system for restaurants is displayed explained entities.
  • the predicted rating 402 is 3.5 stars, and shows a percentage 404 of 84%, indicating to the primary user, "You'll love it.”
  • the text in 406 describes the explained entity, Kaygetsu (a sushi restaurant), and also indicates that there exists sufficient evaluation data for Kirala (a different sushi restaurant), and there are over 10 people who like Kirala and Kaygetsu.
  • the explanation may be: "Many people who like Kirala also like Kaygetsu,” where hovering over "many” brings up a slideshow of photographs 408 of a subset of the people's faces who like both restaurants where the photos are selected according to a model of the user(s)'s affinity for facial features.
  • the slide of photographs of relevant users to the primary user or textual description may indicate users who have similar tastes to the primary user, thus predicting and suggesting that the primary user may like the same restaurants as the relevant user (e.g., Gordo Taqueria, The Counter, Bouchon), or dislike others (e.g., McDonald's).
  • one packaging of explanation factors may incorporate a numeric descriptor of unknown users, an evaluation description for a known entity A, a known entity A, an evaluation description for an unknown entity B, and an unknown entity B.
  • a numeric descriptor of unknown users For example, "A few people who like Kirala also like Kaygetsu” 406 or "People who love Bi Rite Creamery also love Mitchell's Ice Cream”.
  • another packaging of explanation factors may incorporate an objective statistic and a relation on the objective statistic (possibly involving one or more affinity graphs). For example, "Several people who have similar tastes to you love Bouchon” 406, “People who have similar tastes to you hate Plutos", and “Many people with similar taste preferences to you are okay with II Fornaio” may all be such explanation factors.
  • objective statistics may include counting, or other forms of aggregation and cosine similarity in a vector space, either alone or in combination.
  • Explicit or implicit annotations such as a check-in or status update on a social media website, may indicate that a user has visited or prefers a location or entity. These explicit or implicit annotations may be used by the system to determine and specify narrowing relationships between users and entities.
  • a check-in on Facebook is an example of such an annotation, and may be used to determine a number of a primary user's relevant users (e.g., Facebook friends, Twitter follows) that have gone there in the past and/or the frequency at which the primary user's relevant users visit the entity.
  • Narrowing relationships may involve location data that may be obtained implicitly through a user's mobile device or other means (e.g.. GPS, location services). For example, “ oi is the best match for your tastes in San Francisco”, “Denny's is the best match for your tastes in Sunnyvale that is open now” 416, “The Counter is the best Burger Joint for you in Palo Alto”, and “Burger King is the best Burger Joint for you in Cupertino open now” all explain a restaurant choice in terms of location with optional extra descriptions based on preferences or restaurant attributes like hours of operation cross-referenced with the current time. Location data may also be used to compute statistics for geographic areas which can be explanation factors.
  • the recommendations explanation may call attention to the 6 and elide the other 34.
  • the evaluation data related to users that are more similar to the primary user may be weighted more heavily than evaluation data from users that are not as similar to the primary user.
  • FIG. 5 shows an exemplary user display 500 which presents a list of cosmetic surgery options (e.g., nose job, face lift, calf implants, liposuction), along with a percentage ranking of how much the system recommends the given surgery to the primary user, see section 506.
  • cosmetic surgery options e.g., nose job, face lift, calf implants, liposuction
  • the display may also include other descriptions (either textual or other rich media) of relevant information 502, such as the primary user's insurance or other information that the primary user may find interesting.
  • the display may also include photographs 504 or other media.
  • Recommendation explanations may also be used to explain the ranking of entities as part of a browsing, search, or other information retrieval experience.
  • the recommendation explanations associated with a list of bars near a group of users could include text to explain why one bar was ranked higher than another, examples being "15 Romelo has the best cocktails in the city” and "Cocktails at BarOne are second only to those at 15 Romelo”.
  • the system may monitor and track various metrics, and optionally incorporate those while selecting recommendation explanation factors for their display. For example, explicit collections or lists of entities a user has annotated, frequency at which users select an entity with a given type of explanation optionally adjusting for user location, the time of day the user is viewing the recommendation explanation, present and past purchase history for the user or group of users, and comparisons between the primary user and other users; may all be used either alone or in combination, in generating and selecting recommendation explanations, and the display of the selected recommendation explanations.
  • Embodiments of the invention may be extended such that an explanation of a recommendation relates to a target user's friend (e.g., on Facebook) or follower/followee (e.g., Twitter).
  • Location information may be used in the context of an affinity graph to determine recommendations.
  • Explanations of recommendations may also relate to a prediction of what the target user might do, or how they might think or evaluate an entity.
  • the per-user data in the affinity graph about the user e.g., a user's followees/friends may lead to an inference about the user) to determine what explanation is likely to be most useful, valuable, and relevant.
  • the explanation may involve similarity between users.
  • the explanations may be offered to the user as a set of selections, which involves what explanations have been helpful in the past for the same or similar users.
  • Some embodiments may provide methods of operation in which they do not need to know who a user is; for example, there may be a kiosk mode in which members of the public can use the system without identifying themselves.
  • the description provides preferred exemplary embodiment(s) only, and is not intended to limit the scope, applicabi lity or configuration of the invention.
  • the embodiments may be described as a process which is depicted as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although a flowchart may describe the operations as a sequential process, many of the operations may be performed in parallel or concurrently. In addition, the order of the operations may be re-arranged. A process is terminated when its operations are completed, but could have additional steps not included in the figure. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, its term ination corresponds to a return of the function to the calling function or the main function.
  • the term “storage medium” may represent one or more devices for storing data, including read only memory (ROM), random access memory (RAM), magnetic RAM, core memory, magnetic disk storage mediums, optical storage mediums, flash memory devices and/or other machine readable mediums for storing information.
  • ROM read only memory
  • RAM random access memory
  • magnetic RAM magnetic RAM
  • core memory magnetic disk storage mediums
  • optical storage mediums flash memory devices and/or other machine readable mediums for storing information.
  • computer-readable medium includes, but is not limited to portable or fixed storage devices, optical storage devices, wireless channels and various other mediums capable of storing, containing or carrying instruction(s) and/or data.
  • embodiments may be implemented by hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof.
  • the program code or code segments to perform the necessary tasks may be stored in a machine readable medium such as storage medium.
  • a processor(s) may perform the necessary tasks.
  • a code segment may represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a class, or any combination of instructions, data structures, or program statements.
  • a code segment may be coupled to another code segment or a hardware circuit by passing and/or receiving information, data, arguments, parameters, or memory contents.
  • Information, arguments, parameters, data, etc. may be passed, forwarded, or transmitted via any suitable means including memory sharing, message passing, token passing, network

Abstract

Embodiments of the invention relate to a computer-implemented method for generating explanatory data from a personalized recommendations process for a primary user based at least on stored data about the primary user. The method comprises a server computer obtaining data related to one or more users who are relevant to the primary user, then determining at least one group of users relevant to the primary user. The server computer also obtains data related to one or more entities, determines one or more entities relevant to the primary user, and associates the at least one relevant group of users with the one or more relevant entities. One or more potential candidate factors are generated. A set of factors are selected from the one or more potential candidate factors, wherein the potential candidate factors are used as explanatory data to determine recommendations to the primary user.

Description

A RECOMMENDATION ENGINE THAT PROCESSES DATA INCLUDING USER DATA TO PROVIDE RECOMMENDATIONS AND EXPLANATIONS FOR THE RECOMMENDATIONS TO A USER
CROSS-REFERENCES TO RELATED APPLICATIONS
[0001] This application claims priority from co-pending U.S. Provisional Patent Application No. 61/509,999 filed July 20, 201 1 entitled "Method and Apparatus for Explaining
Recommendations".
[0002] The present disclosure may be related to the following commonly assigned U.S.
Provisional Patent Applications: U.S. Provisional Patent Application No. 61 /509,995 filed July 20, 201 1 , entitled
"Method and Apparatus for Category Based Navigation",
U.S. Provisional Patent Application No. 61/510,003 filed July 20, 201 1 entitled "Method and Apparatus for Allowing Users to Augment Searches", and
U.S. Provisional Patent Application No. 61/510,004 filed July 20, 201 1 entitled "Method and Apparatus for Quickly Evaluating Entities".
[0003] The respective disclosures of these applications/patents are incorporated herein by reference, as if set forth in full in this document, for all purposes.
FIELD OF THE INVENTION
[0004] The present invention relates to recommendation systems in general and in particular to recommendation system that collect data to sift through and generate personalized
recommendations and also explain personalized recommendations for users, via user devices, in order to present recommendation explanation factors used in providing informative and intuitive recommendations.
BACKGROUND OF THE INVENTION
[0005] The Internet has become a useful source of information relating to goods and services, but because so many users and companies are connected to the Internet and providing information of their own, it is difficult to find relevant information among all available data. As individuals and groups are now able to connect to the Internet and retrieve information from websites and via apps, and do so perhaps while in transit towards a geographic area of interest, they come to expect the ability to find information about goods and services being offered. [0006] One source of information may be in the form- of websites hosted by or for providers. As used herein, "website" refers to a collection of one or more web pages displayable by a browser or other web client, the one or more web pages being hosted on a web server (i.e., a computer coupled to the Internet, or other network over which communication occurs) that responds to requests for pages hosted thereon. An example might be a restaurant that generates information about its hours, pictures of its dining room, contact information, history, menus and other offerings, then puts that information into a suitable form (such as HTML pages, dynamically generated pages, etc.), stores those pages in electronic storage accessible by the web server (hosted by the restaurant or perhaps a company that specializes in web presence that provides such services to the restaurant), and then serves those pages as client devices request pages. The information might also be presented in another form, such as data supplied to an application (i.e., app) on the client device rather than a generic browser on the client device.
[0007] However, while interested users may get information directly from the restaurant, the users are still limited to information sourced by that restaurant. A number of websites have been set up to collect information from other customers of a vendor of goods or services and provide reviews, comments and other details to other customers or potential customers. For example, there might be a website that focuses on hotels, another on restaurants, and another on home repair, or a general one. However, these websites may have too much information to be useful to a user searching for relevant recommendations.
[0008] Additionally, with the increase of users participating in social media websites, users are also interested in getting information about their friends, and the opinions of their friends. For example, a user may find out that a number of their friends on a social media website highly recommend a restaurant, which may be more persuasive to that user than the recommendations of strangers or high rankings on general rating websites. Thus, social media websites that provide information to a user regarding the user's friends may also be another source that the user refers to in gathering information and searching for relevant recommendations. However, social media websites allow users to provide a wide variety of information that may or may not be related to what a user is searching for (e.g., status updates unrelated to a vendor of goods or services), and it is also difficult to cross-reference the recommendations of friends through social media websites with general websites providing reviews, or websites for particular businesses, in a user's search for relevant recommendations. [0009] An improved recommendations system is desirable.
BRIEF SUMMARY OF THE INVENTION
[0010] Embodiments of the invention relate to an improved computer-implemented method for generating personalized recommendations and providing explanation data related to the personalized recommendations. Existing methods for generating personalized recommendations to a target user may not provide explaining factors and processes used to determine the personalized recommendations. Thus, if the target user is dissatisfied with the personalized recommendations, it is difficult for the target user to feed back to a personalized
recommendations system how to improve the generation of personalized recommendations for the target user. Further, providing the target user with explanation data also informs the target user of a source from which the personalized recommendation was determined, which may influence the target user's decision to select the personalized recommendation or not. For example, a target user may receive a personalized recommendation to go to Restaurant C and also receive explanation data that Restaurant C was recommended to the target user because a "friend" of the target user, User C, goes there regularly. However, if User C is an ex-boyfriend of the target user, the target user may not want to go to Restaurant C for fear of running into User C, even though it may be likely that the target user would have enjoyed the food at Restaurant C.
[0011] Thus, providing explanation data of personalized recommendations to the target user is advantageous and allows users to understand the logic and processes behind generating their personalized recommendations. Additionally, personalized recommendation systems are improved in this manner by becoming more intelligent about generating personalized recommendations to its users.
[0012] Embodiments of the invention relate to a method for providing explanation data from a personalized recommendations process for a primary user based at least on stored data about the primary user. The method comprises a server computer obtaining data related to one or more users who are relevant to the primary user, wherein the one or more users relevant to the primary user are similar to the primary user. The server computer then determines at least one group of users relevant to the primary user based on the data related to the one or more users, each relevant group including one or more relevant users to the primary user. Then, the server computer also obtains data related to one or more entities. The method further comprises determining, at the server computer, one or more entities relevant to the primary user based on the data related to the one or more entities, and then associating the at least one relevant group of users with the one or more relevant entities. The server computer then generates one or more potential candidate factors based on at least the relevant group of users or the one or more relevant entities. [0013] The method may further comprise selecting, by the server computer, a set of factors from the one or more potential candidate factors and displaying the set of selected factors to the primary user, wherein the potential candidate factors are used to determine recommendations or explanations to the primary user.
[0014] Another embodiment of the invention relates to a server computer comprising a processor and a non-transitory computer readable medium, the non-transitory computer readable medium comprising code executable by the processor to implement a method for generating personalized recommendations for a primary user based at least on stored data about the primary user. The method comprises a server computer obtaining data related to one or more users who are relevant to the primary user, wherein the one or more users relevant to the primary user are similar to the primary user. The server computer then determines at least one group of users relevant to the primary user based on the data related to the one or more users, each relevant group including one or more relevant users to the primary user. Then, the server computer also obtains data related to one or more entities. The method further comprises determining, at the server computer, one or more entities relevant to the primary user based on the data related to the one or more entities, and then associating the at least one relevant group of users with the one or more relevant entities. The server computer then generates one or more potential candidate factors based on at least the relevant group of users or the one or more relevant entities.
[0015] The method performed by the server computer may further comprise selecting, by the server computer, a set of factors from the one or more potential candidate factors and displaying the set of selected factors to the primary user, wherein the potential candidate factors are used to determine recommendations to the primary user. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] A further understanding of the nature and advantages of the present invention may be realized by reference to the following drawings. In the appended figures, similar components or features may have the same reference label, with distinct instances of the component or feature distinguished by parenthetical indices. If only the first reference label is used in the
specification, the description is applicable to any one of the similar components having the same first reference label irrespective of the second reference label.
[0017] FIG. 1 is a block diagram of a personalized recommendation system according to aspects of the present invention. [0018] FIG. 2 is a block diagram of an exemplary flow that may be performed by the personalized recommendation system to explain recommendations according to aspects of the present invention.
[0019] FIG. 3 is a block diagram of a modularized personalized recommendations system according to aspects of the present invention. [0020] FIG. 4 is an illustration of a user display of selected factors to explain recommendations according to aspects of the present invention.
[0021] FIG. 5 is an illustration of a user display of selected factors and user input to explain and provide recommendations according to aspects of the present invention.
DETAILED DESCRIPTION OF THE INVENTION
[0022] An improved recommendation system is described herein. In order to provide a context for describing embodiments of the present invention, embodiments of the invention will be described herein with reference to a user device that receives user recommendations. Those skilled in the art will recognize, however, that the present invention is not limited to such embodiments. [0023] It should be understood that, given the complexity of the operations performed, the invention is implemented using computing elements and may also include communication elements that convey data between disparate locations.
[0024] In various embodiments, systems which recommend entities may use a variety of algorithmic approaches to produce predicted evaluations or recommendations, hereafter referred to as recommendations. Prior to discussing embodiments of the invention, some terms used herein are described below:
[0025] "Entities" according to embodiments of the invention may be anything that can be represented or referred to in a computer system. In some embodiments, entities are providers of goods and services, such as restaurants or nightclubs. In other embodiments, entities are particular goods or services or events, such as a particular dish at a restaurant or a one-time or repeating event such as a concert. In others entities are documents or sections of documents, such as text pages or musical compositions, to be presented to a user. In other embodiments, an entity may be a relationship or potential relationship between two people. For example, a dating recommendation system may recommend in favor of or against a potential relationship between one user and another. In another example, entities may be individuals.
[0026] "Recommendations" according to embodiments of the invention may be along a single or multiple dimensions, and/or over a discrete or continuous domain. Recommendations may be presented in a numerical, textual, pictorial, auditory, video, or tactile fashion. Recommendations may be include any of these characteristics, either alone or in combination. For example, a user may rate a restaurant on a scale of 1 , 2, 3, 4, or 5 stars, with 1 star being the lowest rating, and 5 stars being the highest rating. In another realization, a user may instead rate a restaurant on a 1 -5 star scale for taste rating, 1 -4 "$" scale for price rating, and add an optional one word description. [0027] In some implementations, recommendations might be called "evaluations" or used in a similar manner. For example, an evaluation might be done by a team of doctors choosing a picture that best represents the mood or situation of a clinical patient, in effect, "recommending" which is the "best" image to used.
[0028] Another example of an evaluation or recommendation may include citizen journalists recording a five second video to make a review or recommendation, or an audio summary of a protest or event. In another example, recommendations may use three dimensional sensing technologies to have dancers record their expressions in response to listening to a composition. Another example may include a user operating an application on a mobile device, and dragging pictorial representations of dishes into different groups or piles categorized as "tasty", "yucky", and "only sometimes". In another example, a recommendation may be inferred from aspects of a user's behavior such as whether the user visits a particular web page, looks in one direction or another, or whether the user walks or turns in one way or another.
[0029] An "affinity graph" according to embodiments of the invention is a model of the affinity between users or entities and users or entities, either in general or with respect to a given attribute or cluster of attributes of the users or entities, as a graph. Affinity graphs may represent relationships between users and entities, users and other users, or entities and other entities, and may be used to determine recommendations. An example of an affinity graph is a social graph wherein nodes represent users of a social interaction system and edges between nodes represent "friend" links. It should be understood, however, that some embodiments may not represent users explicitly and might not require that a person interacting with the application be identified or authenticated.
[0030] Attributes provide additional information about a node in the graph. For example, an attribute may be the zoom level of a camcorder type or the primary cuisine type of a restaurant. Entities may be represented by nodes, with individual users being one type of entity. Edges represent relationships between nodes. Directional edges can model unidirectional relationships, such as if person A follows person B on Twitter. A symmetric relationship, like friends on Facebook, may be represented as two directional edges, or a unidirectional edge.
[0031] "Evaluation data" for a user may be, according to embodiments of the invention, inputs into a recommendation service provided by a personalized recommendation system described herein. For example, evaluation data may include ratings on a discrete star scale for the quality of a restaurant for a restaurant discovery website/product or answers to personality questions for a match making website/service. Evaluation data may be associated with one entity or a set of entities. For example, a user may rate Jane's Sushi 90/100 and may rate sushi restaurants in general 82/100. [0032] A "recommendations provider" is a service that generates recommendations and predictions of evaluation data of entities for users. The recommendations provider may be on the same device or different devices from the device displaying the recommendations. The recommendations provider may be operated by the same system collecting the evaluation data, or may be operated on a separate system as the system collecting the evaluation data. [0033] An "explanations provider" is a service that generates explanations of the
recommendations (i.e., recommendation explanations) provided by the recommendations provider. The explanations provider may be on the same device or different devices from the device displaying explanations. The explanations provider and the recommendations provider may be operated on the same or on different systems from each other. The explanations provider may be operated on the same system or a separate system as the system collecting the evaluation data.
[0034] As used herein, an "unknown user" is a user of the recommendations provider or other service known to the recommendations provider for which the service has access to evaluation data but is not connected to a primary user or group of users viewing a recommendation by an appropriate link or links in the affinity graph. For example, if the affinity graph is a family tree, then Jane and her mother, Joan, would be connected, but Jane's best friend Becky would be an unknown user in evaluating recommendations for Jane and Joan as primary users. Depending on the system or the primary user, a user may be known or unknown in different contexts. For example, in a system with an affinity graph linking users with their favorite celebrities and an affinity graph linking users and their friends, the celebrity linkage might be appropriate for movie or fashion recommendation explanations for a primary user, but not appropriate for dentist recommendations. The celebrity's dentist may not be relevant to the primary user, but the dentist of the primary user's coworker may be more relevant and useful to the primary user. [0035] As used herein, a "known user" is a user of the recommendations provider or another service known to the recommendations provider for which the recommender knows evaluation data and which is connected to the user (group of users) viewing a recommendation by an appropriate link or links in the affinity graph. For example Joan is a known user to Jane from the example above. [0036] An "explained entity" is a specific entity being recommended to a user or group of users for which recommendations and recommendation explanations are being generated. The explained entity may or may not have associated evaluation data. A "known entity" is an entity with sufficient associated evaluation data to show a user how the known entity has been evaluated. For example, in a system for recommending theater performances, a user may have given the category "Musicals" two thumbs up to indicate approval, thereby making the category a known entity (because the thumbs up are evaluation data that can be shown to a user). As a further example, when viewing a page about "Joseph and the Amazing Technicolor Dreamcoat," where the user has not given a thumbs up or thumbs down to that musical, that entity would not be a known entity because the system would not have associated evaluation data (the thumbs up or down) that could be shown to the user. Nevertheless, the user's previous thumbs down ratings for two other Andrew Lloyd Webber plays could indicate that "Joseph and the Amazing
Technicolor Dreamcoat" is an explained entity (because the previous ratings could be used to create a recommendation for "Joseph and the Amazing Technicolor Dreamcoat").
[0037] An "evaluation description" is the language or messaging used to describe evaluation data for an entity. For example, in a system with 1 to 5 star ratings, 1 star might be "I hated it," 2 stars might be "1 disliked it," 3 stars might be "I'm okay with it," 4 stars might be "1 liked it," and 5 stars might be "I loved it." The descriptions need not be a direct mapping from an evaluation to prose and may be dependent on the context in which the evaluation description is viewed. From example, in a system with 1 to 5 star ratings where various entities are being compared, those with 4 and 5 stars may be described and categorized as "enjoyed" and 1 and 2 stars as "Did not care for." Evaluation descriptions may include textual information as well as richer media sources in the same or similar manner as recommendation explanations.
[0038] A "predicted evaluation description" is the language or messaging used to make predictions about what known or unknown users would evaluate an explained entity. For example, when a recommendations provider for kitchen tiles has an affinity graph that models followers and a primary user following Joe, then when the primary user is viewing an entity describing marble tile that the recommendations provider believes Joe would rate 89/90 on beauty and 75/80 on long term value, a predicted evaluation description might be, "Joe would probably love the marble tile." Predicted evaluation descriptions may include textual information as well as richer media sources in the same or similar manner as recommendation explanations. For example, with "Joe would probably love the marble tile," a picture of Joe may appear above his name and both the picture and his name may be hyperlinked to a page describing his preferences.
Recommendations
[0039] In FIG. 1 , an exemplary personalized recommendations system according to an embodiment of the invention is shown. The personalized recommendations system 160 may be coupled to a plurality of users, such as a primary user 100, secondary user 102, or any number of users 104. There may be relationships between specific users, for example, the primary user 100 and secondary user 102 may be friends, family members, or co-workers. The plurality of users may be coupled to the personalized recommendations system 160 through associated user devices 1 10, 1 12, and 1 14. User devices may include mobile phones (e.g., a Blackberry™ phone, an iPhone™ device, an Android™ device), tablets (e.g., an iPad™ tablet), or other electronic devices.
[0040] The personalized recommendations system 160 may obtain inputs from any number of users 100, 102, 104, registered with the personalized recommendations system 160. The users 100, 102, and 104 may be interconnected and have relationships with one another. The inputs may be transmitted through the user devices 1 10, 1 12, 1 14 via an application. The user devices 1 10, 1 12, 1 14 may be connected to the personalized recommendations system 160 through the internet, a wireless network, a telecommunications network, or any suitable communications network. For example, the primary user 100 may use the primary user device 1 10 to download and install an application issued by the personalized recommendations system 160. Through the application, the primary user 100 may edit personal ratings for entities, such as venues and restaurants, and other personal preferences, such as cuisine, type of food, price range, location, distance from current location, etc. The primary user 100 may also provide feedback items or data through the application to respond to recommendations, to add or remove entities, and/or rate an experience at an entity. In other embodiments of the invention, entities may include merchants and service providers.
[0041] Additional inputs to the personalized recommendations system 160 may come from external data sources, for example, a social network computer system such as the Facebook™ computer system 140(A), a public messaging system such as the Twitter™ messaging system 140(B), and/or any number of other data sources 1 40(N) via the user accounts 150(A), 150(B), 152(A), 152(B) with the data sources and/or user devices 1 10, 1 12. The external data sources may also provide raw location data of the users, for example, through a "check-in" through system 140(A), system 140(B), or other social networking provider system capable of providing raw location data. The other data sources 140( ) may themselves have associated user accounts, which may map directly or indirectly to user accounts of other systems. [0042] The personalized recommendations system or provider may use affinity graphs as models of the affinity between users or entities and users or entities, either in general or with respect to a given attribute or cluster of attributes of the users or entities, as a graph. Affinity graphs may represent relationships between users and entities, users and other users, or entities and other entities, and may be used to determine recommendations. Attributes provide additional information about a node in the graph. For example, an attribute may be the zoom level of a camcorder type or the primary cuisine type of a restaurant. Entities may be represented by nodes, with individual users being one type of entity. Edges represent relationships between nodes. Directional edges can model unidirectional relationships, such as if person A follows person B on Twitter. A symmetric relationship, like friends on Facebook, may be represented as two directional edges, or a unidirectional edge. [0043] These relationships may be about different associations other than social affiliations, such as an expertise graph where an edge from user A to user B indicates that user A bel ieves user B to be an expert. There may also be weights on these edges, which for example, might represent the confidence in another user's expertise, or represent how familiar two users are with each other. Affinity graph nodes may be further augmented with additional information such as short form textual messages, pictures, check-ins, other explicit or impl icit annotations that are used to denote a user has visited or prefers a location, or other explicitly tagged content. The affinity graph may be constructed by users choosing to link themselves to other nodes, or by aggregation and analysis methods which infer the links, weights, and/or nodes. In embodiments, a link may be inferred from who sends messages to whom in a network, or links may be taken as given in the form of user-determined friendship relationships and weights on the links inferred from messages, or information may be inferred by who or what a user looks at.
[0044] The recommendations provider operated on the personal ized recommendations system may also use evaluation data, such as inputs into the recommendation service provided by the personalized recommendation system described herein. For example, evaluation data may include ratings on a discrete star scale for the quality of a restaurant for a restaurant discovery website/product or answers to personality questions for a match making website/service.
Evaluation data may be associated with one entity or a set of entities. For example, a user may rate Jane's Sushi 90/ 100 and may rate sushi restaurants in general 82/100.
[0045] Using affinity graphs, evaluation data, and/or other algorithms and data, the recommendations provider operated by personalized recommendations system may generate recommendations along a single or multiple dimensions, and/or over a discrete or continuous domain. Recommendations may be numerical, textual, pictorial, auditory, or tactile in fashion. Recommendations may be include any of these characteristics, either alone or in combination. For example, a user may rate a restaurant on a scale of 1 , 2, 3, 4, or 5 stars, with 1 star being the lowest rating, and 5 stars being the highest rating. In another realization, a user may instead rate a restaurant on a 1 -5 star scale for taste rating, 1 -4 "$" scale for price rating, and add an optional one word description. Another example of a recommendation (e.g., evaluation) involves having a team of doctors choosing a picture that best represents the mood of a clinical patient. Other examples may include recording a five second video to make a review or recommendation, an audio summary of a protest or event, where a user looks or how longs the user looks at something, or in a mobile application, dragging pictorial representations of entities into rating categories.
[0046] In embodiments, a recommendations provider may construct recommendations using statistical techniques that are known in the literatures of statistics, artificial intelligence, and machine learning. An example of such a technique is using cosine similarity in an application where the ratings are on a 1 -5 star rating scale to determine how similar one entity's set of ratings is to another entity' s set of ratings, and to provide recommendations of the first entity based on the similarity. Another example of such a technique is using latent Dirichlet allocation in an application where the ratings are combinations of one or more words like "yucky," "tasty," etc., to model the ratings as words associated with topics that an entity is associated with in a topic model. Another example of such a technique is using natural language processing to extract sentiment information from free-form textual statements related to an entity and using a collaborative filtering technique with the information thus extracted.
[0047] Example recommendations may be, but are not limited to being, textual (e.g., a short 20 word comment), continuous (e.g., a ratings slider), discrete as in star ratings, pictorial (e.g., a representative picture, drawing, or photograph which most closely resembles the entity), auditory (e.g.. a volume measurement or review), gestural (e.g., a swipe up or down), visual (e.g., an interpretive dance for the entity), or some combination thereof. Recommendations may involve adding edges or weights, either individually or some combination to one or more affinity graphs.
[0048] Some recommendations may not be intuitive to the primary user viewing the recommendation. The personalized recommendations system according to embodiments of the invention presents one or many explanations for why an entity has been included in a recommendation to a primary user or set of users. An explanation of the recommendation may or may not reflect computations performed to generate the recommendation or factors used to perform the computations. Recommendation explanations may be considered as a set of plausible reasons that a given entity or entities would be a good choice for a specific user or group of users.
[0049] For example, if a recommendation is given to a user that is derived from a statistical technique that incorporates large quantities of information, and that uses features of the information that are learned by machine and are not human-interpretable, no aspect of the decision to make the recommendation may be clearly explainable to the user. Instead, the system might construct an explanation by, for example, calculating a similarity score between an entity being recommended and another entity the user has already rated, and reporting on the similarity.
Recommendation Explanations
[00S0] A conceptual architecture of an exemplary system according to an embodiment of an invention is diagrammed in FIG. 2. There may be a primary user or primary group of users 202 in which a recommendation is to be provided to. The personalized recommendations system may have a user account database storing data (e.g., evaluation data) related to all users, including the primary user. The recommendation for the primary user includes
recommendations to entities within a group of explained entities 204. For example, the personalized recommendation system may have an entity database of stored data for explained entities 204, such as restaurants, hotels, and other vendors providing goods and/or services to users 202.
[0051] Explained entities 204 may be entities that are recommended to a primary user or group of users 202 for which recommendations and recommendation explanations are being generated. The explained entity may or may not have associated evaluation data, in other words, it may not be a known entity, which is an entity with sufficient associated evaluation data to show a user how the known entity has been evaluated.
[0052] In step S I , the personalized recommendations system may perform similar user and entity selection. Data related to primary user or primary user group 202 and data related to explained entities 204 may be collected, and the personalized recommendations system may use this data to determine similar users 206 and similar entities 208. Similar users and similar entities are identified as relevant groups of users and entities, respectively, relative to the primary user or primary group of users 202 and the entities to be explained 204. In some embodiments, only relevant entities are determined and used to generate personalized recommendations and/or explanations, and data related to similar or relevant users is not used. [0053] Next, in step S2, the primary user 202, the entities to be explained 204, the similar users 206, and similar entities 208 are used alone or in combination in a process to generate candidate factors for recommendation explanations. From these potential explanation factors the system may determine and select, in step S3, a subset to show to the user. Finally, in step S4, the system crafts a display of the explanation factors or recommendation explanations to present to the primary user.
[0054] Any number or part of the steps may be el ided or combined. Such combinations may be necessary to improve the speed of computation, ease the maintainability of an
implementation, or allow customization by a primary user. For example, an implementation for a horse recommendation in Montana, the personalized recommendations explanation system may decide to generate and process the English language HTML explanations simultaneously while generating explanation factors, so that there is no need for display generation after factor selection is done. Thus, step S4 may occur simultaneously with step S2 instead of
chronologically after step S3. This may be done to improve performance (e.g., performing steps in parallel to improve speed) or to simplify the implementation, among other possible reasons. [0055] As depicted in FIG. 2, in step S 1 , locating similar users 206 and similar entities 208 may produce many subsets of all the users and entities known to the personalized
recommendations explanation system. For example, it may produce "user and user," "user and entity," and "entity and entity" pairings with optional subsets or supersets of any combination, such as pairing a group of users with a set of groups of users. Locating similar users and entities may take into account attributes of the entities or users previously stored in the entity database and user account database, respectively. Attributes may also be data collected in real-time at the time of a recommendation request of the primary user, such as the time of day, and location information of the primary user, which may be used either alone or in combination with each other or with previously stored data. The user database and entity database could be NoSQL databases, an SQL database, another information retrieval system such as a computer filesystem, or an in-memory data store. In other embodiments, the system could have a temporary piece of data obtained from a third-party on demand, such that data does not need to be statically or locally stored. A database could consist of multiple storage facilities, such as a cache backed by a database or a normalized authoritative database and denormalized in-memory stores.
[0056] Exemplary attributes may include demographic information, user gender, user height, user weight, user income, entity revenues, the cost of items, cuisine type, previous evaluations done by the user, check-ins, other explicit or implicit annotations that are used to denote whether a user has visited or prefers a location or entity, or relations in an affinity graph(s). Demographic data may include gender, height, weight, income, or pre-determined attributes. Additional user attributes may also include explicitly stated preferences. The personalized recommendations explanation system recommending restaurants may allow a user to declare they are one or more of pescetarian, vegetarian, vegan, kosher, lactose intolerant, and wheat gluten intolerant. In an example of a restaurant recommendation system, two restaurants might be similar if they both serve the same or related cuisines, e.g., the system may consider cupcakes and desserts being similar cuisines, while Japanese and Chinese are not. Extending the example, a restaurant recommendation system may also deem two restaurants similar if their respective head chefs both apprenticed at the same restaurant, or if both restaurants are in a zip code with the same median family income and have been open for more than 3 years. Al l these forms of information may be learned or explicit. For example, user attributes such as gender might be inferred from user behavior;,whether a restaurant is Japanese may be inferred or may be received from an external database, and whether Japanese or Chinese should be considered similar or dissimilar may be learned or preprogrammed.
[0057] In steps S2 and S3, the generation and selection of recommendation explanation factors may be computed using a variety of metrics incorporating different spatial, temporal, social, demographic, and user interaction based signals, either alone or in combination. Other examples of data user in selecting recommendation explanation factors include, but are not l imited to, explicit user suggestions, demographic data, location data at the time a user or group of users is seeing an explanation, the time of day a primary user or group of users is requesting a recommendation and viewing recommendation explanation, explicit user suggestion of an explanation or part of an explanation, voting or other group selection of expl icit user suggestions or system generated suggestions of an explanation or part of an explanation, table driven matching. Additionally, historical data, such as query history including click through and other interest actions to a search or discovery service, browsing history, purchase history, the location of the primary user or group of users over time, and information in third party services from previous visits or evaluations, may be used in generating and selecting recommendation explanation factors. The selection of factors may also involve psychological models of behavior which may or may not be culturally dependent. For example, it could be the case that American users respond to explanations better when referred to as "users," while British prefer to be referred to as "patrons." The selection of factors may also involve functions to be maximized such as a prediction of profitability, user retention, or conversion rate (of a number viewers or visitors into the number of users who take some action, such as making a purchase). A function to be maximized may be predicted for this purpose using statistical techniques (for example, regression analysis).
[0058] In generation and selection of recommendation explanation factors for
recommendations of explained entities 204 for primary user or primary user groups 202, the system may access evaluation descriptions of the explained entities 204 and/or similar entities 208. Evaluation descriptions describe evaluation data for an entity, relating to a user' s experience or relationship with that entity. For example, in a system with 1 to 5 star ratings for a restaurant, 1 star indicate that the user "hates" it, 2 stars might indicate that the user "dislikes" it, 3 stars might indicate that the user is "okay" or indifferent to it, 4 stars might indicate that the user "likes" it, and 5 stars might indicate that the user "loves" it. The descriptions may not be a direct mapping from an evaluation to prose and may be dependent on the context in which the evaluation description is viewed. For example, in a system with 1 to 5 star ratings where various entities are being compared, those with 4 or 5 stars may be grouped together and categorized as "Enjoyed." and 1 or 2 stars as "Did not care for." Evaluation descriptions may include textual information as well as richer media sources in the same or similar manner as recommendation explanations. [0059] Predicted evaluation descriptions may also be used to make predictions about what known or unknown users would evaluate an explained entity. For example, a recommendations provider for kitchen tiles may have an affinity graph modeling followers and a primary user who is following Joe. When the primary user is viewing an entity describing marble tile which the recommendations provider believes Joe would rate 89/90 on beauty and 75/80 on long term value, a predicted evaluation description might be, "Joe would probably love the marble tile." Predicted evaluation descriptions may include textual information as wel l as richer media sources in the same or similar manner as recommendation explanations. For example, with "Joe would probably love the marble tile," a picture of Joe may appear above his name and both the picture and his name may be hyperlinked to a page describing his preferences.
[0060] As another example, consider a system in which bars are evaluated on a 1 to 5 star rating scale, which has an affinity graph which models friendships, in which Abby is viewing a bar Formosa. When the recommendations provider predicts a rating for Abby's friend, Sarah, the predicted evaluation description might be: "would probably hate" for a 1 star prediction, "would dislike" for a 2 star prediction, "would be okay with" for a 3 star prediction, "would like" for a 4 star prediction and "would love" for a 5 star prediction. There are many other ways in which such a description could be phrased; for instance, a 1 star prediction could give a description of "would probably rate 1 star." Note that Abby's and Sarah's predicted ratings may be the same or different.
[0061] FIG. 3 shows a block diagram of an exemplary modular personalized recommendations system 160 according to an embodiment of the invention, where the personalized
recommendations system 160 is configured to provide explanations of the personalized recommendations. In other embodiments of the invention, an explanations provider may be separate from the recommendations provider, and may be operated by different systems. In the example shown in FIG. 3, the personalized recommendations system 160 operates as the recommendations provider and explanation (i.e., recommendations explanation) provider.
[0062] The personalized recommendations system 160 may be a server computer 700, which may be a single computer, a cluster of computers, a server, or a super computer. Processes and methods implemented by or at the server computer could also be implemented by or at a client, in various embodiments. The server computer 700 may comprise a processor 740, and a non- transitory computer readable medium 750, on which code is stored. There may be several databases to store data, for example, an entity database 730(a), a factor database 730(b), a user database 730(c), and a media object database 730(d). Stored in the entity database 730(a) may be evaluation data relating to entities, including entity meta-data (e.g., location, hours of operation, specialty, rating). Entity meta-data may include any sub-category of data related to the entity. The factor database 730(b) may store previously determined and selected factors from a plurality of users. The user database 730(c) may store user data obtained from a plurality of users, such as real-time data from social networking sites (e.g., tweets, status updates), and stored data related to a plurality of users, such as user preferences and settings (e.g., gender, dietary restrictions). The real-time data may be explicit or implicit, and may be inferred or its significance or relevance may be inferred, and may include text, images, video, audio, or other media. The user data may be directly obtained from the user or may be obtained from external data sources, such as a social networking site associated with the user. The personalized recommendations system 160 may also include a media object database 730(d), storing media objects, such as high resolution photos. The media objects may be associated with entities, recommendations, and/or the target user, and may be used in the display of recommendations and/or recommendation explanations. The association between media objects and entities, recommendations, or a user may be inferred or may be preprogrammed or derived from a preexisting database.
[0063] The code stored on the non-transitory computer readable medium 750 may be in modules, and is executable by the processor 740 to perform functions, which are implemented as processing steps described above by the personalized recommendations system providing recommendation explanations. The non-transitory computer-readable medium 750 may comprise software modules, such as a similar user selection module 701 , a similar entity selection module 702, a factor generation module 703, a factor selection module 704, a factor display module 705, and an explanation module 706. The functions of these modules are described in the process flows described above and in FIG. 2.
[0064] The personalized recommendations system 160, to communicate with external data sources, social networking sites, and users, may include a network interface 720. The network interface 720 includes hardware and software that enables the personalized recommendations system 160 to transmit and receive data over the Internet or any other communications network (e.g., telecommunications network, cable, radio).
Displaying Recommendations and Recommendation Explanations
[0065] Once recommendations have been determined and their explaining factors have been selected, the recommendations and recommendation explanations may be displayed. The display of recommendation explanations performed in step S4 may be in the form of textual information on a screen of a mobile device. However, the means by which information is communicated is not limited to textual information or to a device screen, but may include instead or in addition richer media sources or forms of communication, such as two or three dimensional models, pictures, a heads-up display, video, hyperlinked text, web pages, styled text, recorded sounds, computer generated sounds, vibrations of a tactile device, pressure exerted on the user by haptic technology, electric shocks or pulses, the control of remote devices, and the generation of scents. As an example of a restaurant recommendation system, the system may explain a
recommendation for an Italian restaurant by indicating that five of a primary user's friends go to the restaurant and then providing pictures of the five friends, where each picture is linked to more information from each friend, such as possibly including the friend's evaluation (e.g., review) of the restaurant. The display of recommendation explanations may be visible or not depending on external context, for example, not showing recommendations about user's cuisine preferences to other users until that primary user has rated at least ten places on a 1 -5 scale. [0066] Similar to the generation and selection of recommendation explanation factors, the explanation of display generation may be computed using a variety of metrics incorporating different spatial, temporal, social, demographic, and user interaction based signals, either alone or in combination. Other examples of data used in explaining display generation include, but are not limited to, explicit user suggestions, demographic data, location data at the time a user or group of users is seeing an explanation, the time of day a primary user or group of users is requesting a recommendation and viewing recommendation explanation, explicit user suggestion of an explanation or part of an explanation, voting or other group selection of explicit user suggestions or system generated suggestions of an explanation or part of an explanation, table driven matching. Additionally, historical data, such as query history (including click through and other interest actions to a search or discovery service), browsing history, purchase history, the location of the primary user or group of users over time, and information in third party services from previous visits or evaluations, may be used in generating and selecting recommendation explanation factors. The selection of factors may also involve psychological models of behavior which may or may not be cultural ly dependent. For example, it could be the case that American users respond to explanations better when referred to as "users," while British prefer to be referred to as "patrons."
[0067] In explaining a predicted evaluation, the personalized recommendations system may count or aggregate a number of users with a similar actual or predicted evaluation. When generating explanations, incorporating this number the number may be shown directly to the user(s) or a description or summary may be shown instead. The language or messaging used is known as a numeric descriptor and may include textual information as well as richer media sources in the same or similar manner as discussed earlier in regards to recommendation explanations.
[0068] FIG. 4 shows an illustration of a user display 400 in which a recommendation system for restaurants is displayed explained entities. The predicted rating 402 is 3.5 stars, and shows a percentage 404 of 84%, indicating to the primary user, "You'll love it." The text in 406 describes the explained entity, Kaygetsu (a sushi restaurant), and also indicates that there exists sufficient evaluation data for Kirala (a different sushi restaurant), and there are over 10 people who like Kirala and Kaygetsu. Therefore, the explanation may be: "Many people who like Kirala also like Kaygetsu," where hovering over "many" brings up a slideshow of photographs 408 of a subset of the people's faces who like both restaurants where the photos are selected according to a model of the user(s)'s affinity for facial features. The slide of photographs of relevant users to the primary user or textual description may indicate users who have similar tastes to the primary user, thus predicting and suggesting that the primary user may like the same restaurants as the relevant user (e.g., Gordo Taqueria, The Counter, Bouchon), or dislike others (e.g., McDonald's).
[0069] In an embodiment of the invention shown in FIG. 4, one packaging of explanation factors may incorporate a numeric descriptor of unknown users, an evaluation description for a known entity A, a known entity A, an evaluation description for an unknown entity B, and an unknown entity B. For example, "A few people who like Kirala also like Kaygetsu" 406 or "People who love Bi Rite Creamery also love Mitchell's Ice Cream".
[0070J In another embodiment of the invention shown in FIG. 4, another packaging of explanation factors may incorporate an objective statistic and a relation on the objective statistic (possibly involving one or more affinity graphs). For example, "Several people who have similar tastes to you love Bouchon" 406, "People who have similar tastes to you hate Plutos", and "Many people with similar taste preferences to you are okay with II Fornaio" may all be such explanation factors. Examples of objective statistics may include counting, or other forms of aggregation and cosine similarity in a vector space, either alone or in combination.
[0071] Explicit or implicit annotations, such as a check-in or status update on a social media website, may indicate that a user has visited or prefers a location or entity. These explicit or implicit annotations may be used by the system to determine and specify narrowing relationships between users and entities. A check-in on Facebook is an example of such an annotation, and may be used to determine a number of a primary user's relevant users (e.g., Facebook friends, Twitter follows) that have gone there in the past and/or the frequency at which the primary user's relevant users visit the entity. For example, "A few people you follow came here last month" 410, "Abby (one of your followers) who also likes Plutos checked in here", "Michael Preysman (a person you follow) checked in at Baretta and has also checked in here" 412, "Many people who have checked in at Delfina have checked in here" 412, "You checked in at Cafe Borrone six months ago" 412, "You checked in at Plutos six times in the last two weeks", and "Paul came here recently and loved it" 414 are all example explanations generated from such a narrowing.
[0072] Narrowing relationships may involve location data that may be obtained implicitly through a user's mobile device or other means (e.g.. GPS, location services). For example, " oi is the best match for your tastes in San Francisco", "Denny's is the best match for your tastes in Sunnyvale that is open now" 416, "The Counter is the best Burger Joint for you in Palo Alto", and "Burger King is the best Burger Joint for you in Cupertino open now" all explain a restaurant choice in terms of location with optional extra descriptions based on preferences or restaurant attributes like hours of operation cross-referenced with the current time. Location data may also be used to compute statistics for geographic areas which can be explanation factors. For example, "Berkeley is known for sushi, try Kirala" 416 and "New Orleans is known for Po Boys, try them at Mahony's". Location may also include facts of general interest or interest to the user, for example: "Cafe Du Monde is the home of the Beignet, try one!" [0073] While the preceding examples and FIG. 4 are related to restaurants, the preceding explanation factors may also be extended to narrowing relationships which may involve one or more affinity graphs, and in other areas. For example, in a song recommendation system, "Many people you follow hate Madonna", "Abby (one of your followers) who has similar music preferences to you loves Pink", "Sting would love Meatloaf ', and "Eminem who has similar music preferences to you would like Mozart" all provide more specific information than that computed by an objective statistic alone. One class of narrowing relations involves selecting a subset of users who have each rated a subset of entities and showing them as explanation factors. For example, in a restaurant recommendation system, there may be 40 ratings from 40 users about a given dish, with 6 users rating it 5 stars, and 34 rating it 1 star. If the primary user whose recommendations are being explained to are more similar to the 6 users than the other 34, according to an objective statistic, like cosine similarity, the recommendations explanation may call attention to the 6 and elide the other 34. In other words, the evaluation data related to users that are more similar to the primary user may be weighted more heavily than evaluation data from users that are not as similar to the primary user.
[0074] Recommendation explanations may be displayed in a variety of contexts which may or may not include the recommendation being explained. For example, FIG. 5 shows an exemplary user display 500 which presents a list of cosmetic surgery options (e.g., nose job, face lift, calf implants, liposuction), along with a percentage ranking of how much the system recommends the given surgery to the primary user, see section 506. When the primary user selects surgery options from the list, they are presented with a textual description of why the surgery was ranked that way for the primary user, thus explaining its ranking recommendation. The display may also include other descriptions (either textual or other rich media) of relevant information 502, such as the primary user's insurance or other information that the primary user may find interesting. The display may also include photographs 504 or other media.
[0075] Recommendation explanations may also be used to explain the ranking of entities as part of a browsing, search, or other information retrieval experience. For example, the recommendation explanations associated with a list of bars near a group of users could include text to explain why one bar was ranked higher than another, examples being "15 Romelo has the best cocktails in the city" and "Cocktails at BarOne are second only to those at 15 Romelo".
[0076] The system may monitor and track various metrics, and optionally incorporate those while selecting recommendation explanation factors for their display. For example, explicit collections or lists of entities a user has annotated, frequency at which users select an entity with a given type of explanation optionally adjusting for user location, the time of day the user is viewing the recommendation explanation, present and past purchase history for the user or group of users, and comparisons between the primary user and other users; may all be used either alone or in combination, in generating and selecting recommendation explanations, and the display of the selected recommendation explanations.
[0077] Embodiments of the invention may be extended such that an explanation of a recommendation relates to a target user's friend (e.g., on Facebook) or follower/followee (e.g., Twitter). Location information may be used in the context of an affinity graph to determine recommendations. Explanations of recommendations may also relate to a prediction of what the target user might do, or how they might think or evaluate an entity. Then, the per-user data in the affinity graph about the user (e.g., a user's followees/friends may lead to an inference about the user) to determine what explanation is likely to be most useful, valuable, and relevant. The explanation may involve similarity between users. The explanations may be offered to the user as a set of selections, which involves what explanations have been helpful in the past for the same or similar users. In some embodiments, there may be multiple simultaneous target users, or there may be one target user and several associated users for whom searches should also be relevant. Input data from such multiple users can be presented synchronously or asynchronously. Some embodiments may provide methods of operation in which they do not need to know who a user is; for example, there may be a kiosk mode in which members of the public can use the system without identifying themselves. [0078] The description provides preferred exemplary embodiment(s) only, and is not intended to limit the scope, applicabi lity or configuration of the invention. Rather, the description of the preferred exemplary embodiment(s) will provide those ski lled in the art with an enabling description for implementing a preferred exemplary embodiment of the invention. It is to be understood that various changes may be made in the function and arrangement of elements without departing from the spirit and scope of the invention as set forth in the appended claims.
[0079] Specific details are given in the description to provide a thorough understanding of the embodiments. However, it will be understood by one of ordinary skill in the art that the embodiments may be practiced without these specific details. For example, systems may be shown in block diagrams in order not to obscure the embodiments in unnecessary detail. In other instances, well-known processes, structures and techniques may be shown without unnecessary detail in order to avoid obscuring the embodiments.
[0080] Also, it is noted that the embodiments may be described as a process which is depicted as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although a flowchart may describe the operations as a sequential process, many of the operations may be performed in parallel or concurrently. In addition, the order of the operations may be re-arranged. A process is terminated when its operations are completed, but could have additional steps not included in the figure. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, its term ination corresponds to a return of the function to the calling function or the main function. [0081] Moreover, as disclosed herein, the term "storage medium" may represent one or more devices for storing data, including read only memory (ROM), random access memory (RAM), magnetic RAM, core memory, magnetic disk storage mediums, optical storage mediums, flash memory devices and/or other machine readable mediums for storing information. The term "computer-readable medium" includes, but is not limited to portable or fixed storage devices, optical storage devices, wireless channels and various other mediums capable of storing, containing or carrying instruction(s) and/or data.
[0082] Furthermore, embodiments may be implemented by hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware or microcode, the program code or code segments to perform the necessary tasks may be stored in a machine readable medium such as storage medium. A processor(s) may perform the necessary tasks. A code segment may represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a class, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and/or receiving information, data, arguments, parameters, or memory contents.
Information, arguments, parameters, data, etc. may be passed, forwarded, or transmitted via any suitable means including memory sharing, message passing, token passing, network
transmission, etc.
[0083] Further embodiments may be envisioned to one of ordinary skill in the art after reading this disclosure. In other embodiments, combinations or sub-combinations of the above disclosed invention may be advantageously made. The example arrangements of components are shown for purposes of illustration and it should be understood that combinations, additions, rearrangements, and the like are contemplated in alternative embodiments of the present invention. Thus, while the invention has been described with respect to exemplary embodiments, one skilled in the art will recognize that numerous modifications are possible. [0084] For example, the processes described herein may be implemented using hardware components, software components, and/or any combination thereof. The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense. It will, however, be evident that various modifications and changes may be made thereunto without departing from the broader spirit and scope of the invention as set forth in the claims and that the invention is intended to cover all modifications and equivalents within the scope of the following claims.

Claims

WHAT IS CLAIMED IS: 1. A computer-implemented method for generating explanatory data from a personalized recommendations process for a primary user based at least on stored data about the primary user, the method comprising:
determining the primary user for whom explanatory data is to be generated, the primary user being one of a plurality of users over which recommendations are considered;
obtaining data related to a set of relevant users, wherein the set of relevant users comprises one or more users in the plurality of users that are deemed relevant to the primary user, wherein relevancy of a user in the set of relevant users to the primary user is based, at least in part, on similarities, according to stored user data, between the relevant user and the primary user;
obtaining, at the server computer, entity data from a plurality of data sources, wherein the entity data is associated with an entity in a plurality of entities;
storing the entity data at the server computer, wherein the entity data associated with the plurality of entities is stored in an entity database;
determining, at the server computer, one or more entities relevant to the primary user based on the data related to the one or more entities;
determining, by the server computer, one or more potential candidate factors based on at least the one or more relevant entities; and
generating, by the server computer, explanation data based on the one or more potential candidate factors used in the personalized recommendations process.
2. The method of claim 1 , further comprising:
selecting, by the server computer, a set of factors from the one or more potential candidate factors; and
communicating, by the server computer, the set of selected factors to the primary user, wherein the potential candidate factors are used to determine recommendations to the primary user.
3. The method of claim 1 , further comprising:
determining, at the server computer, at least one group of users relevant to the primary user based on the data related to the one or more users, each relevant group including one or more relevant users to the primary user; associating, by the server computer, the at least one relevant group of users with the one or more relevant entities;
determining, by the server computer, one or more potential candidate factors based on at least the relevant group of users; and
generating, by the server computer, explanation data based on the one or more potential candidate factors used in the personalized recommendations process.
4. The method of claim 1 , wherein associating the at least one relevant group of users with the one or more relevant entities includes determining a relationship between a relevant user with a relevant entity, a relationship between a relevant entity with another relevant entity, or a relationship between a relevant user with another relevant user.
5. The method of claim 1 , wherein data related to one or more relevant users includes account information stored in a database of users, wherein the account information includes demographic data.
6. The method of claim 1 , wherein the data related to one or more relevant users includes current location information of the user.
7. The method of claim 5, wherein the pre-determined attributes includes cuisine preferences or food allergies.
8. The method of claim 1 , wherein the current location information is associated with data related to one or more relevant entities, wherein the data related to one of relevant entities includes previous data from the one or more relevant users related to the one or more relevant entities.
9. The method of claim 3, wherein selecting the set of factors is based on at least one of, or combinations of, spatial, temporal, social, demographic, or user history data previously stored in a database.
10. The method of claim 9, wherein selecting the set of factors is based on at least one of, or combinations of, spatial, temporal, social, demographic, or user current data collected in real-time.
1 1. The method of claim 9, wherein the user history data includes at least one of, or combinations of, query history, browsing history, or purchase history of the primary user.
12. The method of claim 9, wherein selecting the set of factors is based on table driven matching, wherein the tables are stored in the database.
13. The method of claim 10, wherein selecting the set of factors is based on psychological models of behavior of a demographic of the primary user.
14. The method of claim 3, further comprising:
determining a set of media objects associated with the set of selected factors, wherein the set of media objects includes at least one of, or combinations of, textual information, pictures, video, hyperlinked text, webpages, styled text, and sounds; and
commnicating the set of media objects with the set of selected factors for the
recommendation explanation.
15. The method of claim 3, wherein communicating the set selected factors includes multi-dimensional models, vibrations, electric shocks, pulses, or scents.
16. A server computer comprising a processor and a non-transitory computer readable medium, the non-transitory computer readable medium comprising code executable by the processor to implement a computer-implemented method for generating explanatory data from a personalized recommendations process for a primary user based at least on stored data about the primary user, the method comprising:
determining the primary user for whom explanatory data is to be generated, the primary user being one of a plurality of users over which recommendations are considered;
obtaining, at a server computer, data related to a set of relevant users, wherein the set of relevant users comprises one or more users in the plurality of users that are deemed relevant to the primary user, wherein relevancy of a user in the set of relevant users to the primary user is based, at least on part, on similarities, according to stored user data, between the relevant user and the primary user;
obtaining, at the server computer, entity data from a plurality of data sources, wherein the entity data is associated with an entity in a plurality of entities; 86 storing the entity data at the server computer, wherein the entity data associated with the
87 plurality of entities is stored in an entity database;
88 determining, at the server computer, one or more entities relevant to the primary user based
89 on the data related to the one or more entities;
90 determining, by the server computer, one or more potential candidate factors based on at
91 least the one or more relevant entities; and
92 generating, by the server computer, explanation data based on the one or more potential
93 candidate factors used in the personalized recommendations process.
94 17. The server computer of claim 16, the method further comprising:
95 selecting, by the server computer, a set of factors from the one or more potential candidate
96 factors; and
97 communicating, by the server computer, the set of selected factors to the primary user,
98 wherein the potential candidate factors are used to determine recommendations to the
99 primary user.
100 18. The server computer of claim 16, the method further comprising:
101 determining, at the server computer, at least one group of users relevant to the primary user
102 based on the data related to the one or more users, each relevant group including one or
103 more relevant users to the primary user;
104 associating, by the server computer, the at least one relevant group of users with the one or
105 more relevant entities;
106 determining, by the server computer, one or more potential candidate factors based on at
107 least the relevant group of users; and
108 generating, by the server computer, explanation data based on the one or more potential
109 candidate factors used in the personalized recommendations process.
1 10 19. The server computer of claim 16, wherein associating the at least one
1 1 1 relevant group of users with the one or more relevant entities includes determining a relationship
1 12 between a relevant user with a relevant entity, a relationship between a relevant entity with
1 13 another relevant entity, or a relationship between a relevant user with another relevant user.
20. The server computer of claim 16, wherein data related to one or more relevant users includes account information stored in a database of users, wherein the account information includes demographic data.
21. The server computer of claim 16, wherein the data related to one or more relevant users includes current location information of the user.
22. The server computer of claim 20, wherein the pre-determined attributes includes cuisine preferences or food allergies.
23. The server computer of claim 16, wherein the current location information is associated with data related to one or more relevant entities, wherein the data related to one of relevant entities includes previous data from the one or more relevant users related to the one or more relevant entities.
24. The method of claim 18, wherein selecting the set of factors is based on at least one of, or combinations of, spatial, temporal, social, demographic, or user history data previously stored in a database.
25. The method of claim 24, wherein selecting the set of factors is based on at least one of, or combinations of, spatial, temporal, social, demographic, or user current data collected in real-time.
26. The method of claim 24, wherein the user history data includes at least one of, or combinations of, query history, browsing history, or purchase history of the primary user.
27. T he method of claim 24, wherein selecting the set of factors is based on table driven matching, wherein the tables are stored in the database.
28. The method of claim 25, wherein selecting the set of factors is based on psychological models of behavior of a demographic of the primary user.
29. The method of claim 18, further comprising:
determining a set of media objects associated with the set of selected factors, wherein the set of media objects includes at least one of, or combinations of, textual information, pictures, video, hyperlinked text, webpages, styled text, and sounds; and 141 communicating the set of media objects with the set of selected factors for the
142 recommendation explanation.
143 30. The method of claim 18, wherein communicating the set selected factors
144 includes multi-dimensional models, vibrations, electric shocks, pulses, or scents.
1
EP12815207.1A 2011-07-20 2012-07-19 A recommendation engine that processes data including user data to provide recommendations and explanations for the recommendations to a user Withdrawn EP2734967A4 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US201161509999P 2011-07-20 2011-07-20
PCT/US2012/047476 WO2013013091A1 (en) 2011-07-20 2012-07-19 A recommendation engine that processes data including user data to provide recommendations and explanations for the recommendations to a user

Publications (2)

Publication Number Publication Date
EP2734967A1 true EP2734967A1 (en) 2014-05-28
EP2734967A4 EP2734967A4 (en) 2015-04-22

Family

ID=47558487

Family Applications (1)

Application Number Title Priority Date Filing Date
EP12815207.1A Withdrawn EP2734967A4 (en) 2011-07-20 2012-07-19 A recommendation engine that processes data including user data to provide recommendations and explanations for the recommendations to a user

Country Status (4)

Country Link
EP (1) EP2734967A4 (en)
AU (1) AU2012283930B2 (en)
CA (1) CA2842255C (en)
WO (1) WO2013013091A1 (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10394919B2 (en) 2013-03-08 2019-08-27 Opentable, Inc. Context-based queryless presentation of recommendations

Families Citing this family (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9792640B2 (en) 2010-08-18 2017-10-17 Jinni Media Ltd. Generating and providing content recommendations to a group of users
US20120095862A1 (en) 2010-10-15 2012-04-19 Ness Computing, Inc. (a Delaware Corportaion) Computer system and method for analyzing data sets and generating personalized recommendations
US9123335B2 (en) 2013-02-20 2015-09-01 Jinni Media Limited System apparatus circuit method and associated computer executable code for natural language understanding and semantic content discovery
WO2014177955A1 (en) * 2013-04-28 2014-11-06 Jinni Media Ltd. A system apparatus circuit method and associated computer executable code for generating and providing content recommendations to a group of users
CA2914939A1 (en) * 2013-06-10 2014-12-18 Opentable, Inc. Providing personalized recommendations relating to group actions
US9754306B2 (en) 2014-03-03 2017-09-05 Invent.ly LLC Recommendation engine with profile analysis
US9430476B2 (en) * 2014-03-13 2016-08-30 GM Global Technology Operations LLC Method and apparatus of user recommendation system for in-vehicle apps
US20160342288A1 (en) * 2015-05-19 2016-11-24 Ebay Inc. Intelligent highlighting of item listing features
US10068284B1 (en) 2017-08-29 2018-09-04 Snowfall Technologies Llc Graphical user interface having scrollable, visual representation of historical product searches and direct shopping cart links
US20190295124A1 (en) * 2018-03-26 2019-09-26 DoorDash, Inc. Dynamic predictive similarity grouping based on vectorization of merchant data
US20190340537A1 (en) * 2018-05-07 2019-11-07 Google Llc Personalized Match Score For Places
US11301513B2 (en) 2018-07-06 2022-04-12 Spotify Ab Personalizing explainable recommendations with bandits
US11144983B2 (en) 2019-08-09 2021-10-12 Virgin Cruises Intermediate Limited Systems and methods for computer generated recommendations with improved accuracy and relevance
US11386301B2 (en) 2019-09-06 2022-07-12 The Yes Platform Cluster and image-based feedback system
US20210374851A1 (en) * 2020-05-28 2021-12-02 Td Ameritrade Ip Company, Inc. Hierarchical Node-Based Display Architecture

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20070118498A1 (en) * 2005-11-22 2007-05-24 Nec Laboratories America, Inc. Methods and systems for utilizing content, dynamic patterns, and/or relational information for data analysis
US20080214162A1 (en) * 2005-09-14 2008-09-04 Jorey Ramer Realtime surveying within mobile sponsored content
US20090177644A1 (en) * 2008-01-04 2009-07-09 Ronald Martinez Systems and methods of mapping attention

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6370513B1 (en) * 1997-08-08 2002-04-09 Parasoft Corporation Method and apparatus for automated selection, organization, and recommendation of items
US8494978B2 (en) * 2007-11-02 2013-07-23 Ebay Inc. Inferring user preferences from an internet based social interactive construct
US20090234664A1 (en) * 2008-03-11 2009-09-17 Schaffnit Wayne A System and method for recommending entertainment venues for specific occasions

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20080214162A1 (en) * 2005-09-14 2008-09-04 Jorey Ramer Realtime surveying within mobile sponsored content
US20070118498A1 (en) * 2005-11-22 2007-05-24 Nec Laboratories America, Inc. Methods and systems for utilizing content, dynamic patterns, and/or relational information for data analysis
US20090177644A1 (en) * 2008-01-04 2009-07-09 Ronald Martinez Systems and methods of mapping attention

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of WO2013013091A1 *

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10394919B2 (en) 2013-03-08 2019-08-27 Opentable, Inc. Context-based queryless presentation of recommendations

Also Published As

Publication number Publication date
AU2012283930B2 (en) 2016-02-04
WO2013013091A1 (en) 2013-01-24
AU2012283930A1 (en) 2014-02-06
CA2842255A1 (en) 2013-01-24
EP2734967A4 (en) 2015-04-22
CA2842255C (en) 2020-02-18

Similar Documents

Publication Publication Date Title
CA2842255C (en) A recommendation engine that processes data including user data to provide recommendations and explanations for the recommendations to a user
US11709851B2 (en) Method and apparatus for quickly evaluating entities
US20130024464A1 (en) Recommendation engine that processes data including user data to provide recommendations and explanations for the recommendations to a user
US20220292575A1 (en) Computer System and Method for Analyzing Data Sets and Generating Personalized Recommendations
US20190370676A1 (en) Interestingness recommendations in a computing advice facility
US11263543B2 (en) Node bootstrapping in a social graph
US9396492B2 (en) Computer system and method for analyzing data sets and providing personalized recommendations
CA2955330C (en) Geographically localized recommendations in a computing advice facility
Hong et al. Mymoviehistory: social recommender system by discovering social affinities among users
WO2012051586A1 (en) A computer system and method for analyzing data sets and generating personalized recommendations
AU2012283928B2 (en) Method and apparatus for category based navigation
AU2012283929B2 (en) Method and apparatus for allowing users to augment searches

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

17P Request for examination filed

Effective date: 20140117

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

RAP1 Party data changed (applicant data changed or rights of an application transferred)

Owner name: NESS COMPUTING, LLC

RAP1 Party data changed (applicant data changed or rights of an application transferred)

Owner name: OPENTABLE, INC.

DAX Request for extension of the european patent (deleted)
RA4 Supplementary search report drawn up and despatched (corrected)

Effective date: 20150320

RIC1 Information provided on ipc code assigned before grant

Ipc: G06Q 30/00 20120101AFI20150316BHEP

Ipc: G06Q 50/00 20120101ALI20150316BHEP

Ipc: G06Q 30/06 20120101ALN20150316BHEP

Ipc: G06Q 30/02 20120101ALI20150316BHEP

17Q First examination report despatched

Effective date: 20170124

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE APPLICATION IS DEEMED TO BE WITHDRAWN

18D Application deemed to be withdrawn

Effective date: 20170607