US9747896B2

US9747896B2 - System and method for providing follow-up responses to prior natural language inputs of a user

Info

Publication number: US9747896B2
Application number: US14/884,030
Authority: US
Inventors: Michael R. Kennewick, JR.; Michael R. Kennewick, SR.
Original assignee: VoiceBox Technologies Corp
Current assignee: VoiceBox Technologies Corp
Priority date: 2014-10-15
Filing date: 2015-10-15
Publication date: 2017-08-29
Anticipated expiration: 2035-10-15
Also published as: US10229673B2; CN107003999A; CN107003999B; WO2016061309A1; US20170345417A1; EP3207467A1; EP3207467A4; US20160110347A1

Abstract

In certain implementations, follow-up responses may be provided for prior natural language inputs of a user. As an example, a natural language input associated with a user may be received at a computer system. A determination of whether information sufficient for providing an adequate response to the natural language input is currently accessible to the computer system may be effectuated. A first response to the natural language input (that indicates that a follow-up response will be provided) may be provided based on a determination that information sufficient for providing an adequate response to the natural language input is not currently accessible. Information sufficient for providing an adequate response to the natural language input may be received. A second response to the natural language input may then be provided based on the received sufficient information.

Description

CROSS REFERENCE TO RELATED APPLICATION

This application claims the benefit of U.S. Provisional Patent Application Ser. No. 62/064,329 filed Oct. 15, 2014 entitled “System and Method for Providing Follow-up Responses to Prior Natural Language Inputs of a User”, the entirety of which is incorporated herein by reference.

FIELD OF THE INVENTION

The invention relates to systems and methods of providing responses to natural language inputs.

BACKGROUND OF THE INVENTION

Electronic user devices have emerged to become nearly ubiquitous in the everyday lives of many people. One of the reasons for this increased use is the convenience of receiving information with a user device, for example, via personal assistant software capable of processing natural language input. In many cases, however, information for providing an adequate response to a natural language request from a user may not be immediately available or accessible to the personal assistant software (e.g., because of poor network connectivity, service outage, etc.). In such cases, the personal assistant is unable to provide an adequate response and the user has to remember to provide the query at a later time to receive an adequate response. These and other drawbacks exist.

SUMMARY OF THE INVENTION

The invention relates to systems and methods for providing responses to natural language inputs. As an example, system responses to natural language inputs (e.g., utterances, gestures, etc.) may be facilitated by enabling follow-up responses (or real-time updates in the form of follow-up responses) to prior natural language inputs of a user (or the user's prior requests). These follow-up responses may, for instance, be provided for presentation to the user without the user having to specify a need for a follow-up. As such, in an implementation, a user need not provide a user request specifying one or more system actions to be performed at a later time for those system actions to be subsequently performed.

In one scenario, for example, a user may ask a personal assistant application to set a thermostat in his/her house to a certain temperature (e.g., “72 degrees or other temperature”). However, a service associated with the thermostat may be temporarily down, the personal assistant application may not be able to immediately satisfy the user's command. As such, as a result of its inability to receive confirmation of a connection with the associated service, the personal assistant application may alert the user of its inability to currently connect to the user's thermostat and indicate that it will follow-up with the user when it is able to set the user's thermostat to the requested temperature. The personal assistant application may thereafter periodically check with the associated service to see if access to set the thermostat is available. When the access becomes available (and without further user input regarding setting of the thermostat after the user's initial command) the personal assistant application may transmit instructions to set the thermostat to the requested temperature. Upon receiving confirmation of the requested setting from the associated service, the personal assistant application may provide a follow-up response for presentation to the user that informs the user that the thermostat has been set to the requested temperature.

In another scenario, a user may ask a personal assistant application to inform him/her of nearby pizza restaurants (e.g., “Are there pizza restaurant around here?”). However, if the personal assistant application is hosted on a user device of the user, and the user device does not currently have network access (e.g., no cellular or WiFi connection), the personal assistant application may inform the user of the lack of network access and that it will follow-up with the user when it is able to perform the requested query. At a subsequent time, upon detection of network access (and without further user input regarding queries for pizza restaurants), the personal assistant application may initiate the requested query. When the results of the requested query are received, the personal assistant application may formulate a response for presentation to the user based on the query results.

In an implementation, a follow-up response to a prior natural language input may be provided for presentation to a user during an interactive session (e.g., between a personal assistant application and the user) other than the interactive session in which the user provided the prior natural language input. As such, in some cases, one or more subsequent natural language inputs may be received from the user and/or system responses to the subsequent natural language inputs may be provided to the user before the follow-up response is provided. In further cases, the subsequent user inputs and their corresponding responses may not be related to the prior natural language input for which the follow-up response is provided.

In an implementation, context information associated with an interactive session between a system (or a personal assistant application thereof) and a user may be utilized to tailor a follow-up response provided during the interactive session. As an example, if the topic of the current interactive session (e.g., a subsequent dialog session) is different than a topic of the prior natural language input (or its related user request), the system may utilize the context information associated with the interactive session to generate a follow-up response to the prior natural language input that may “fit” better within the current interactive session. In one use case, for instance, if the prior natural language input comprises a query for a friend's birthday, and the information regarding the friend's birthday is obtained during a dialog between the system and the user about the user's anniversary, the follow-up response that is generated so that the response is related (at least in part) to the current dialog (e.g., “Speaking of important dates, you asked me earlier about your friend's birthday; it's next Friday”).

Various other aspects of the invention will be apparent through the detailed description of the invention and the drawings attached hereto. It is also to be understood that both the foregoing general description and the following detailed description are exemplary and not restrictive of the scope of the invention. As used in the specification and in the claims, the singular form of “a”, “an”, and “the” include plural referents unless the context clearly dictates otherwise. In addition, as used in the specification and the claims, the term “or” means “and/or” unless the context clearly dictates otherwise.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a system for providing follow-up responses to prior natural language inputs associated with a user, according to an implementation of the invention.

FIG. 2 illustrates a system for facilitating natural language processing, according to an implementation of the invention.

FIG. 3 illustrates a flow diagram for a method of providing follow-up responses to prior natural language inputs associated with a user, according to an implementation of the invention.

FIG. 4 illustrates another flow diagram for a method of providing follow-up responses to prior natural language inputs associated with a user, according to an implementation of the invention.

FIG. 5 illustrates a screenshot of a user interface which provides a response indicating that a follow-up response will be provided, according to an implementation of the invention.

FIG. 6 illustrates a screenshot of a user interface which provides follow-up responses to prior natural language inputs associated with a user, according to an implementation of the invention.

FIG. 7 illustrates a screenshot of a user interface which provides follow-up responses to prior natural language inputs associated with a user in between unrelated subsequent user inputs and/or responses to the subsequent user inputs, according to an implementation of the invention.

DETAILED DESCRIPTION OF THE INVENTION

In the following description, for the purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the implementations of the invention. It will be appreciated, however, by those having skill in the art that the implementations of the invention may be practiced without these specific details or with an equivalent arrangement. In other instances, well-known structures and devices are shown in block diagram form in order to avoid unnecessarily obscuring the implementations of the invention.

FIG. 1 illustrates a system 100 of providing follow-up responses to prior natural language inputs associated with a user, according to an implementation of the invention.

In an implementation, system 100 may facilitate natural language responses to natural language inputs (e.g., utterances, gestures, etc.) by enabling follow-up and/or real time updated responses for prior natural language inputs of a user (e.g., the user's prior requests) without the user having to specify a need for follow-up. In an implementation, system 100 may be enabled to obtain one or more follow-up and/or real-time updates related to a prior natural language input of a user, and provide one or more follow-up and/or real-time responses to the prior natural language input. In an implementation, the follow-up and/or real-time updates and responses to/from system 100 to prior natural language inputs of a user may be facilitated by enabling asynchronous full-duplex interaction between the user and system 100. As an example, multiple channels of communication between a personal assistant application of system 100 and the user may be provided, allowing at least one channel for the personal assistant application to receive natural language inputs (or its corresponding user request) from the user, and at least one channel for the personal assistant application to provide responses to the user. A user input may, for instance, trigger one or more system responses (e.g., including system follow-up responses) that may be provided before, during, or after receipt of subsequent user inputs and/or responses to those subsequent user inputs. In this way, a user and a personal assistant may continue a dialog with one another without having to wait for the personal assistant to provide a response (or an adequate response) to any particular natural language input of the user.

In an implementation, system 100 may continue to receive and/or respond to natural language inputs from a user before providing a response (or an adequate response) to a prior natural language input. System 100 may be enabled to subsequently provide a follow-up response when it is able to obtain sufficient information related to the prior natural language input to provide an adequate response. In this way, a user and system 100 may continue to dialog with one another without having to wait for system 100 to provide a response (or an adequate response) to any particular natural language input of the user. For example, the user may input a query: “When is Jane Doe's birthday?” System 100 may analyze the query and determine that the best place to look for an answer to this query is the Facebook profile of Jane Doe. However, in one scenario, Facebook may be inaccessible from the network available to system 100 (e.g., because of an office firewall). System 100 may then respond to the user with a response indicating that Jane Doe's Facebook profile is not currently accessible (and/or that it will get back to the user at a later time). System 100 may then continue to interact with the user rather than waiting for the Facebook profile to be accessible. Nevertheless, when system 100 eventually obtains information specifying Jane Doe's birthday (e.g., from her Facebook profile or an alternative source), it may thereafter follow-up with a subsequent response to the user's initial query that specifies Jane Doe's birthday.

In an implementation, system 100 may obtain updated information related to a natural language input for which system 100 has previously provided a response. System 100 may generate a follow-up response based on the updated information, and provide the follow-up response for the natural language input (e.g., even if an adequate response to the natural language input may have previously been provided by system 100).

For example, a user may input a query: “How long will it take to get home today?” System 100 may then query a mapping and traffic application to provide an estimated time of travel based on current traffic conditions. Since the user did not provide a time of departure, the current estimated time of travel may be considered an adequate response. The user may continue to interact with system 100, e.g., to set up a reminder for picking up groceries on the way home. The user may at a subsequent time decide to use a messaging application to send a message to his/her spouse indicating that the user is on the way home. Upon processing of the message, system 100 may determine that the user is driving (or about to drive) home. As such, based on this determination, and without further user input requesting an estimated travel time, system 100 may reassess the current traffic conditions and provide an estimated time of travel as a follow-up response to the user's prior inquiry for an estimated travel time to the user's house.

In an implementation, system 100 may obtain updated information and provide follow-up responses (based on the updated information) in real-time. For example, a user may input a query: “Who is winning the Monte Carlo Formula 1 Grand Prix?” In this instance, as the positions of drivers competing in the race change, system 100 may keep providing follow-up responses to the user, e.g., “Felipe Massa is currently in the first position”; “Lewis Hamilton has just passed Felipe Massa to take back the first position” and so forth. In an implementation, for example, these follow-up responses regarding the Formula 1 race may be provided by a personal assistant in between receipt of other natural language inputs (e.g., related to other user requests) from the user and/or responses of the personal assistant to the other natural language inputs.

In an implementation, upon generation of a follow-up response based on updated information related to a prior natural language input of a user, system 100 may provide the follow-up response to the user during an interactive session (e.g., a dialog) between system 100 and the user that is related or unrelated to the prior natural language input from the user. For example, after receiving a first response (e.g., “Felipe Massa is currently in the first position”), the user may decide to continue a dialog with system 100 with inputs relating to current standings of teams and drivers in the current Formula 1 season. If during this dialog, system 100 receives the updated information indicating that Lewis Hamilton has passed Felipe Massa, system 100 may interrupt the dialog to provide this updated information (with a response, e.g., “By the way, Lewis Hamilton has just passed Felipe Massa to take back the first position”) and subsequently provide responses to the user's queries relating to the current standings.

Alternatively, in an implementation, a determination to provide a follow-up response based on the updated information may be effectuated without regard to whether the interactive session is related to the prior natural language input. After receiving the first response, the user may decide to set up reminders for various meetings that the user is supposed to attend. If during this dialog, system 100 receives the updated information regarding a change in position in the race, system 100 may interrupt the session relating to the reminders to provide the updated race status and subsequently continue with setting reminders.

In another implementation, information regarding an interactive session (e.g., context information associated with the interactive session) may be utilized to tailor follow-up responses (to prior natural language inputs) that are provided during the interactive session. For example, a user may have input a query regarding Jane Doe's birthday at a time when the best resource (e.g., Facebook profile) for providing a correct answer to the query may be inaccessible to system 100. At a later time, if the Facebook profile of Jane Doe becomes accessible, system 100 may interrupt whatever interactive session the user may be engaged in (e.g., even if the session is regarding a different topic, such as Formula 1 racing) to provide Jane Doe's birthday. Upon recognizing that the topic of the current interactive session is unrelated to the user's prior inquiry, system 100 may apologize for the interruption and remind the user of the prior inquiry before informing the user of Jane Doe's birthday (e.g., “Sorry to interrupt, but you previously asked about Jane Doe's birthday. It's next Wednesday.”). Additionally, as an example, if system 100 determines that Jane Doe's birthday is the same as some famous Formula 1 driver (e.g., Nicki Lauda), system 100, in the context of Formula 1 racing, may provide this additional information in the follow-up response(e.g., “It so happens that Jane Doe has the same birthday as Nicki Lauda”).

Other uses of system 100 are described herein and still others will be apparent to those having skill in the art. Having described a high level overview of some of the system functions, attention will now be turned to various system components that facilitate these and other functions.

System Components

System

100 may include a computer system 104, one or more content providers 160, one or more user devices 170, one or more search engines 180, and/or other components. Computer system 104 may interface with search engine(s) 180 to query for information for providing a response to a natural language user input, and/or content provider(s) 160 to allow users to access information offered by content provider(s) 160. Search engine(s) 180 may interface with content provider(s) to provide the information to computer system 104. Computer system 104 may further interface with various interfaces of the user device(s) 170 such that users may interact with computer system 104.

To facilitate these and other functions, computer system 104 may include one or more computing devices 110. Each computing device 110 may include one or more processors 112, one or more storage devices 114, and/or other components.

Processor(s) 112 may be programmed by one or more computer program instructions, which may be stored in storage device(s) 114. The computer program instructions may include, without limitation, user input processing instructions 118, profile management instructions 120, user interface instructions 122, response instructions 140, and/or other instructions 130, each of which may themselves include different sets of instructions that each program the processor(s) 112 (and therefore computer system 104) to perform one or more operations described herein. For example, response instructions 140 may include information retrieval instructions 142, and session context determination instructions 144. Other applications may, of course, include one or more of the instructions 118-152 to perform one or more operations as described herein.

In some implementations, a given user device 170 may comprise a given computer device 110. As such, the given user device 170 may comprise processor(s) 112 that are programmed with one or more computer program instructions, such as user input processing instructions 118, profile management instructions 120, user interface instructions 122, response instructions 140, information retrieval instructions 142, session context determination instructions 144, or other instructions 130.

As used hereinafter, for convenience, the foregoing instructions will be described as performing an operation, when, in fact, the various instructions may program processor(s) 112 (and thereafter computer system 104) to perform the operation. It should be appreciated that the various instructions are described individually as discreet sets of instructions by way of illustration and not limitation, as two or more of the instructions may be combined.

User Input Processing

In an implementation, user input processing instructions 118 may process one or more user inputs of a user to determine one or more user requests that are intended by the user when the user provided the user inputs. The user inputs may comprise an auditory input (e.g., received via a microphone), a visual input (e.g., received via a camera), a tactile input (e.g., received via a touch sensor device), an olfactory input, a gustatory input, a keyboard input, a mouse input, or other user input. As described elsewhere herein, user input processing instructions 118 may comprise instructions associated with one or more speech recognition engines (e.g., speech recognition engine(s) 220 of FIG. 2), one or more natural language processing engines (e.g., natural language processing engine(s) 230 of FIG. 2), or other components for processing user inputs to determine user requests related to the user inputs.

In one use case, if the user input is a natural language utterance spoken by a user, the natural language utterance may be processed by a speech recognition engine to recognize one or more words of the natural language utterance. The recognized words may then be processed, along with context information associated with the user, by a natural language processing engine to determine a user request intended by the user when the user provided the natural language utterance. A determination of the user request may, for instance, comprise predicting one or more potential user requests intended by the user, assigning a confidence score to each of the potential user requests, and selecting the potential user request with the highest confidence score as the user request intended by the user.

FIG. 2 illustrates a system 200 for facilitating natural language processing, according to an implementation of the invention. As shown in FIG. 2, system 200 may comprise input device(s) 210, speech recognition engine(s) 220, natural language processing engine(s) 230, application(s) 240, output device(s) 250, database(s) 132, or other components.

In an implementation, one or more components of system 200 may comprise one or more computer program instructions of FIG. 1 and/or processor(s) 112 programmed with the computer program instructions of FIG. 1. As an example, speech recognition engine(s) 220 and/or natural language processing engine(s) 230 may comprise user input processing instructions 118 or other instructions 130 (e.g., mapping instructions, context determination instructions, or other instructions).

Input device(s) 210 may comprise an auditory input device (e.g., microphone), a visual input device (e.g., camera), a tactile input device (e.g., touch sensor), an olfactory input device, a gustatory input device, a keyboard, a mouse, or other input devices. Input received at input device(s) 210 may be provided to speech recognition engine(s) 220 and/or natural language processing engine(s) 230.

Speech recognition engine(s) 220 may process one or more inputs received from input device(s) 210 to recognize one or more words represented by the received inputs. As an example, with respect to auditory input, speech recognition engine(s) 220 may process an audio stream captured by an auditory input device to isolate segments of sound of the audio stream. The sound segments (or a representation of the sound segments) are then processed with one or more speech models (e.g., acoustic model, lexicon list, language model, etc.) to recognize one or more words of the received inputs. Upon recognition of the words of received inputs, the recognized words may then be provided to natural language processing engine(s) 230 for further processing. In other examples, natural language processing engine(s) 230 may process one or more other types of inputs (e.g., visual input representing sign language communication, gestures, or other forms of communication) to recognize one or more words represented by the other types of inputs.

Natural language processing engine(s) 230 may receive one or more inputs from input device(s) 210, speech recognition engine(s) 220, application(s) 240, database(s) 132, or other components. As an example, natural language processing engine(s) 230 may process inputs received from input device(s) 210, such as user inputs (e.g., voice, non-voice, etc.), location-based inputs (e.g., GPS data, cell ID, etc.), other sensor data input, or other inputs to determine context information associated with one or more user inputs. As another example, natural language processing engine(s) 230 may obtain grammar information, profile information, context information, or other information from database(s) 132. The obtained information (or context information determined based on inputs from input device(s) 210) may be processed to determine one or more user requests associated with one or more user inputs of a user. In yet another example, natural language processing engine(s) 230 may process one or more recognized words from speech recognition engine(s) 220 and other information (e.g., information from input device(s) 210, application(s) 240, and/or database(s) 132) to determine one or more user requests associated with one or more user inputs of a user.

In an implementation, natural language processing engine(s) 230 may solicit further inputs from a user by responding with a request for more information via output device(s) 250 if, for instance, a user request associated with a user input of a user cannot be determined with sufficient confidence, more information would helpful to process the user request, etc.

In an implementation, upon determination of a user request of a user, natural language processing engine(s) 230 may determine an application 240 suitable for executing the user request, and provide the user request to the application for further processing. In one scenario, the application 240 may provide one or more results of the user request to output device(s) 250 for presentation to the user.

In an implementation, the application 240 may provide the results of the user request to natural language processing engine(s) 230 for further processing. As an example, the results of the user request may comprise intermediate results that are provided as a parameter for another user request of the user that is to be executed at another application 240. As such, the natural language processing engine(s) 230 may generate the other user request based on the intermediate results, and provide the other user request to the other application 240. As another example, natural language processing engine(s) 230 may formulate a natural language response based on the results received from the application 240, and provide the natural language response to output device(s) 250 for presentation to the user.

In an implementation, a given application 240 may obtain profile information, account information, or other information from database(s) 132 to authenticate a user before executing a user request of the user. As an example, the application 240 may be part of a given service provider. As such, the application 240 may determine whether the user has access to one or more services associated with the application 240 before executing the user request on behalf of the user.

In an implementation, a given application 240 may obtain content from database(s) 132 and/or content provider(s) 160 to provide one or more results of a user request of a user. In one use case, where the user request comprises a command to play a media item (e.g., song, video clip, movie, etc.), and the application 240 comprises a media stream application, the application 240 may obtain the media item from a given content provider(s) 160 and stream the media item to output device(s) 250 for presentation to the user.

In an implementation, natural language processing engine(s) 230, application(s) 240, or other components may store information in database(s) 132 for later use by natural language processing engine(s) 230, application(s) 240, or other components. As an example, as described in further detail elsewhere herein, natural language processing engine(s) 230 may store information regarding user inputs in database(s) 132 and/or update profile information, grammar information, or other information in database(s) 132 based on the information regarding the user inputs.

Profile Management

In an implementation, stored user profile information (e.g., name, address, preferences, etc.) may be utilized to determine the adequacy of available information for responding to a user's natural language input. In one use case, for example, a user may request recommendations for nearby restaurants via a personal assistant application on a user device associated with the user. If, for instance, the user device does not currently have network access (e.g., the user device is not within its service provider's cellular network range), the personal assistant application may utilize the user device's GPS coordinates and pre-stored map data (e.g., map data obtained prior to leaving the cellular network range and stored at the user device) to obtain restaurant information and generate recommendations for the user. However, the user's profile information may indicate that the user is asking for a search of at least a 5 mile radius for restaurants (e.g., a predefined setting specifying the 5 mile radius search), and the pre-stored map data may only allow the personal assistant application to search for restaurants within a 2 mile radius of the user. As such, the personal assistant application may determine that information sufficient to provide an adequate response is not currently accessible. Although the personal assistant application may still generate a response with restaurant recommendations, it may subsequently follow-up with additional and/or updated recommendations when it is able to search for restaurants within at least a 5 mile radius of the user.

In an implementation, profile management instructions 120 may automatically set profile information such as, for example, home and/or work addresses of the user. As an example, the location of a user device of the user may be monitored to determine home and work addresses. In one use case, with user input specifying that a location corresponds to the user's home address, a particular location may be set as the user's home address based on the location being associated with a place at which the user device is located for the vast majority of time (compared to other places). Other profile information may, of course, be automatically (or manually) set, including, for example, preferred brands, preferred shopping locations, sizes, or other parameters related to products or services, entertainment interests, available devices, historical information relating to the user (e.g., school, cities lived in or visited, etc.), and/or other information relating to the user.

User Interface

In an implementation, user interface instructions 122 may generate a personal assistant interface. The personal assistant interface may provide an interface for the user to interact with system 100. For example, a user may touch/tap a prompt/button provided on the personal assistant interface to initiate interaction with the personal assistant. In an implementation, the personal assistant interface may provide a visual, audio, and/or other mode of response to a natural language query from a user. For example, the personal assistant interface may display a list of nearby preferred shopping locations, or a local map and/or directions for a requested shopping location to the user in response to a corresponding query from the user. In an implementation, the personal assistant interface may provide a visual (or other) representation of the natural language input from the user such as, for example, text transcribing the natural language input. Examples of personal assistant interfaces are illustrated with respect to FIGS. 5, 6, and 7.

In an implementation, the personal assistance interface may be associated with a personal assistant application that manages requests (e.g., queries, commands, etc.) of the user and responses to the user requests. As an example, upon determining a user request intended by a user, the personal assistant application may determine an application that is to process the user request, and provide the user request to the application. In one use case, for instance, if the user request comprises a command to play a song, the personal assistant application may provide the command to a music application. In another use case, if the user request comprises a query for weather information, the personal assistant application may provide the query to a weather application. As another example, the personal assistant application may receive results (related to a user request) from an application selected by personal assistant application to process the user request, and then generate a response to the user request for presentation to the user based on the results. In an implementation, the personal assistant application may comprise one or more of user input processing instructions 118, profile management instructions 120, user interface instructions 122, response instructions 140, or other instructions for managing user requests and/or responses to the user requests.

In an implementation, the personal assistant application may comprise an application that is independently operable of other applications that it selects to process user requests and/or from which it receives corresponding results. In one use case, for example, the personal assistant application may be an autonomous application that acts as an interface between users and subsystems (e.g., third party applications) that provide services to the users.

In an implementation, user interface instructions 122 may generate a profile information management interface. The profile information management interface may allow the user to set profile information related to the personal assistant. For example, the profile information management interface enables the user to set default locations, and/or parameters related to products or services preferred by the user. In an implementation, the profile information management interface enables the user to set preferences with respect to operations of the personal assistant, such as, for example, times when active notification is to be suspended (e.g., when in a meeting, when sleeping, etc.).

Information Retrieval

In an implementation, information retrieval instructions 142 may enable computer system 104 to retrieve information for responding to a natural language input associated with a user. For example, in one scenario, a user may ask “Who holds the current world record for the fastest marathon time?” In response, information retrieval instructions 142 may enable computer system 104 (or a personal assistant application thereof) to interface with search engine(s) 180 to retrieve information relating to the query. The retrieved information may be displayed for the user by implementing user interface instructions 122.

In an implementation, information retrieval instructions 142 may enable computer system 104 to determine adequacy of the retrieved information. For example, in the context of Jane Doe's Facebook profile being inaccessible at a given time, information retrieval instructions 142 may enable computer system 104 to determine that the information retrieved while interfacing with content provider 160 (e.g., Facebook) is null (e.g., since Facebook is inaccessible).

In an implementation, information retrieval instructions 142 may enable computer system 104 to determine that user requested information corresponds to dynamic events, activities, or statuses, and, thus, subject to change. For example, in the context of a query relating to race leaders in a Formula 1 race, information retrieval instructions 142 may enable computer system 104 to determine that a Formula 1 race is currently ongoing and that the race leaders may be different at a subsequent time.

In an implementation, information retrieval instructions 142 may enable computer system 104 to determine that providing an adequate response to a natural language input may require interfacing with search engine(s) 180, or content provider(s) 160 at a subsequent time. For example, in the context of estimated time of travel, information retrieval instructions 142 may enable computer system 104 to retrieve information regarding traffic at a time when the user is departing from his/her current location so as to provide updated information relating to the estimated time of travel.

Session Context Determination

In an implementation, session context determination instructions 144 may enable computer system 104 to determine context of an interactive session between computer system 104 and a user. For example, if a user inquires about a currently ongoing Formula 1 race, and follows up with additional inquiries relating to specific drivers or teams competing in Formula 1 races, session context determination instructions 144 may determine that the interactive session between computer system 104 and the user may be specific to Formula 1 racing. As another example, a user may provide a natural language input relating to his wife's anniversary (e.g., “Remind me to purchase a gift for my wife on January 1”), and proceeds to provide several natural language inputs relating to specific shopping locations (e.g., “How far is Tiffany's from here?”; “How long will it take to reach the nearest Macy's?”; “Which is the best bakery in my 10 mile radius?”; etc.) session context determination instructions 144 may enable computer system 104 to determine that the user is attempting to prepare for his anniversary.

In an implementation, session context determination instructions 144 may enable information retrieval instructions 142 to provide follow-up responses to prior natural language input from a user. For example, a user may provide a natural language input related to a query for Jane Doe's birthday. If the only known source of Jane Doe's birthday is Jane Doe's Facebook profile, and the Facebook profile is currently inaccessible, computer system 104 may not be able to provide a response that adequately addresses the user's query at that particular time. At a subsequent time, however, while the user is in a dialog with computer system 104 regarding Formula 1 racing, and Jane Doe's Facebook profile becomes accessible, session context determination instructions 144 may prompt information retrieval instructions 142 to search for a Formula 1 driver having the same birthday as Jane Doe. Information retrieval instructions 142 may then enable computer system 104 to interface with search engine(s) 180 or content provider(s) 160 to retrieve such information and provide it to the user in the context of the dialog relating to Formula 1 racing.

Examples of System Architectures and Configurations

Different system architectures may be used. For example, all or a portion of response instructions 140 (or other instructions described herein) may be executed on a user device. In other words, computing device 110 as illustrated may include a user device operated by the user. In implementations where all or a portion of response instructions 140 are executed on the user device, the user device may interface with search engine(s) 180, and/or content provider(s) 160 to retrieve information, determine a context of an interactive session with a user, and/or perform other functions/operations of response instructions 140.

As another example, all or a portion of response instructions 140 (or other instructions described herein) may be executed on a server device. In other words, computing device 110 as illustrated may include a server device that obtains a user input from a user device operated by the user. In implementations where all or a portion of response instructions 140 are executed on the server device, the server may interface with search engine(s) 180, and/or content provider(s) 160 to retrieve information, determine a context of an interactive session with a user, and/or perform other functions/operations of response instructions 140.

Although illustrated in FIG. 1 as a single component, computer system 104 may include a plurality of individual components (e.g., computer devices) each programmed with at least some of the functions described herein. In this manner, some components of computer system 104 may perform some functions while other components may perform other functions, as would be appreciated. Processor(s) 112 may each include one or more physical processors that are programmed by computer program instructions. The various instructions described herein are exemplary only. Other configurations and numbers of instructions may be used, so long as processor(s) 112 are programmed to perform the functions described herein.

It should be appreciated that although the various instructions are illustrated in FIG. 1 as being co-located within a single computing device 110, one or more instructions may be executed remotely from the other instructions. For example, some computing devices 110 of computer system 104 may be programmed by some instructions while other computing devices 110 may be programmed by other instructions, as would be appreciated. Furthermore, the various instructions described herein are exemplary only. Other configurations and numbers of instructions may be used, so long as processor(s) 112 are programmed to perform the functions described herein.

The description of the functionality provided by the different instructions described herein is for illustrative purposes, and is not intended to be limiting, as any of instructions may provide more or less functionality than is described. For example, one or more of the instructions may be eliminated, and some or all of its functionality may be provided by other ones of the instructions. As another example, processor(s) 112 may be programmed by one or more additional instructions that may perform some or all of the functionality attributed herein to one of the instructions.

The various instructions described herein may be stored in a storage device 114, which may comprise random access memory (RAM), read only memory (ROM), and/or other memory. The storage device may store the computer program instructions (e.g., the aforementioned instructions) to be executed by processor(s) 112 as well as data that may be manipulated by processor(s) 112. The storage device may comprise floppy disks, hard disks, optical disks, tapes, or other storage media for storing computer-executable instructions and/or data.

The various components illustrated in FIG. 1 may be coupled to at least one other component via a network 102, which may include any one or more of, for instance, the Internet, an intranet, a PAN (Personal Area Network), a LAN (Local Area Network), a WAN (Wide Area Network), a SAN (Storage Area Network), a MAN (Metropolitan Area Network), a wireless network, a cellular communications network, a Public Switched Telephone Network, and/or other network. In FIG. 1 and other drawing Figures, different numbers of entities than depicted may be used. Furthermore, according to various implementations, the components described herein may be implemented in hardware and/or software that configure hardware.

User device(s) may include a device that can interact with computer system 104 through network 102. Such user device(s) may include, without limitation, a tablet computing device, a smartphone, a laptop computing device, a desktop computing device, a network-enabled appliance such as a “Smart” television, a vehicle computing device, and/or other device that may interact with computer system 104.

The various databases 132 described herein may be, include, or interface to, for example, an Oracle™ relational database sold commercially by Oracle Corporation. Other databases, such as Informix™, DB2 (Database 2) or other data storage, including file-based (e.g., comma or tab separated files), or query formats, platforms, or resources such as OLAP (On Line Analytical Processing), SQL (Structured Query Language), a SAN (storage area network), Microsoft Access™, MySQL, PostgreSQL, HSpace, Apache Cassandra, MongoDB, Apache CouchDB™, or others may also be used, incorporated, or accessed. The database may comprise one or more such databases that reside in one or more physical devices and in one or more physical locations. The database may store a plurality of types of data and/or files and associated data or file descriptions, administrative information, or any other data. The database(s) 132 may be stored in storage device 114 and/or other storage that is accessible to computer system 104.

Example Flow Diagrams

The following flow diagrams describe operations that may be accomplished using some or all of the system components described in detail above and, in some implementations, various operations may be performed in different sequences and various operations may be omitted. Additional operations may be performed along with some or all of the operations shown in the depicted flow diagrams. One or more operations may be performed simultaneously. Accordingly, the operations as illustrated (and described in greater detail below) are exemplary by nature and, as such, should not be viewed as limiting.

FIG. 3 illustrates a flow diagram for a method of providing follow-up responses to prior natural language inputs associated with a user, according to an implementation of the invention. The various processing data flows depicted in FIG. 3 (and in the other drawing figures) are described in greater detail herein. The described operations may be accomplished using some or all of the system components described in detail above and, in some implementations, various operations may be performed in different sequences and various operations may be omitted. Additional operations may be performed along with some or all of the operations shown in the depicted flow diagrams. One or more operations may be performed simultaneously. Accordingly, the operations as illustrated (and described in greater detail below) are exemplary by nature and, as such, should not be viewed as limiting.

In an operation 301, a natural language input associated with a user may be received at a user device. In an operation 302, the natural language input may be processed to determine a user request related to the natural language utterance. As an example, a user input comprising a natural language utterance may be received and processed by a speech recognition engine to recognize one or more words of the natural language utterance. The recognized words may then be processed, along with context information associated with the user, by a natural language processing engine to determine the user request related to the natural language utterance.

In an operation 303, a determination of whether information sufficient for providing an adequate response to the user request (or the natural language input) is currently accessible may be effectuated.

In an operation 311, upon determination that information sufficient for providing an adequate response to the user request of the user is currently not accessible (or not available), a first response may be provided indicating that a follow-up response will be provided. The first response may be, for example, a visual or audio output indicating that requested information is currently not accessible and that the requested information will be provided when it becomes available. In one implementation, whatever information is available may be provided to the user as part of the first response along with a message indicating that more information is forthcoming at a subsequent time as it becomes available. For example, a user may provide a query such as: “Who won today's Formula 1 race?” If the race is still ongoing (i.e., insufficient information to respond to the query), a response such as: “Today's Formula 1 race is not yet finished, results will be provided when the race is over” may be provided.

In an operation 312, more information for providing an adequate response to the user request may be requested at a subsequent time such as, for example, when a resource (e.g., a Facebook profile) becomes available or accessible. The system request for the information (for providing an adequate response) may, for example, be performed without a subsequent user request for such information after it is initially determined that such information is currently inaccessible.

In an operation 313, first information sufficient for providing adequate response to the user is received, e.g., from the newly available or accessible resource (e.g., a Facebook profile).

In an operation 314, a second response to the user request may be provided to the user based on the first information.

In an operation 321, upon determination that information sufficient for providing an adequate response is currently accessible (or available), such information may be obtained, and a response to the user request may be generated (and provided to the user) based on the obtained information. For example, when a user inquires about Jane Doe's birthday, and a database storing information regarding Jane Doe's birthday is accessible, the information may be obtained and provided to the user.

In an operation 322, additional information relating to the user request may be requested at a subsequent time. The request for additional information may, for example, be performed without a subsequent user request for the additional information.

In an operation 323, additional information relating to the user request may be received (e.g., from a search engine or a content provider). For example, in the context of Jane Doe's birthday coinciding with Nicki Lauda's birthday, this additional information may be received, for example, from a Wikipedia entry about Nicki Lauda.

In an operation 324, a second response based on the additional information relating to the user request may be provided at a subsequent time (e.g., when the user is in a dialog with the personal assistant about Formula 1 racing).

FIG. 4 illustrates another flow diagram for a method of providing follow-up responses to prior natural language inputs associated with a user, according to an implementation of the invention. In an operation 401, a natural language input of a user may be received. In an operation 402, the natural language input may be processed to determine a user request related to the natural language input. As an example, a user input comprising a natural language utterance may be received and processed by a speech recognition engine to recognize one or more words of the natural language utterance. The recognized words may then be processed, along with context information associated with the user, by a natural language processing engine to determine the user request related to the natural language utterance.

In an operation 403, information sufficient for providing an adequate response to the user request may be requested.

In an operation 404, one or more subsequent natural language inputs may be received from the user. In an implementation, the subsequent natural language inputs may be received and processed irrespective of whether information sufficient for providing an adequate response to the user request has been received (responsive to the system request for information sufficient for providing an adequate response to the user request) and/or whether a response to the user request has been provided to the user. As an example, asynchronous full-duplex communication between the user and a personal assistant application (e.g., hosted on computing system 104) may be enabled to facilitate receipt and processing of natural language inputs, and system responses to user inputs.

In an operation 405, first information sufficient for providing an adequate response to the user request may be received (e.g., after receiving the subsequent natural language inputs, after providing responses to the subsequent natural language inputs, etc.).

In an operation 406, a response to the user request may be provided based on the first information. In an implementation, the response based on the first information is provided subsequent to providing a response to the subsequent natural language inputs.

Example Screenshots

FIG. 5 illustrates a screenshot 500 of a user interface which provides a response indicating that a follow-up response will be provided, according to an implementation of the invention. The described screenshots may be accomplished using some or all of the system components described in detail above. With respect to screenshot 500, a user may provide a natural language input (e.g., natural language utterance or other input) indicating a user request for Jane Doe's birthday. In response to the natural language input, an application associated with the depicted user interface (e.g., a personal assistant application) may provide a first response to the natural language input from the user. As an example, the first response may be indicative of insufficient information for providing an adequate response to the natural language input. If, for example, the best resource (e.g., a Facebook profile) for obtaining information for providing a response to the natural language input is inaccessible (e.g., due to resource downtime), a response indicating that sufficient information related to the user's request is not currently available, and that an adequate response will be provided when information becomes available. As depicted in screenshot 500, the response to the user request may include a reason for insufficient information and an indication that a follow-up response will be provided.

FIG. 6 illustrates a screenshot 600 of a user interface which provides follow-up responses to prior natural language inputs associated with a user, according to an implementation of the invention. The described screenshots may be accomplished using some or all of the system components described in detail above. With respect to screenshot 600, continuing the Jane Doe birthday example in FIG. 5, a follow-up response may be provided at a later time (e.g., when an available source storing information regarding Jane Doe's birthday is found). As indicated in screenshot 600, the follow-up response (e.g., “Jane Doe's birthday is October 2. Would you like me to set a reminder?”) may be provided after receiving one or more subsequent natural language inputs from the user and/or providing responses to the subsequent user inputs.

FIG. 7 illustrates a screenshot 700 of a user interface which provides follow-up responses to prior natural language inputs associated with a user in between unrelated subsequent user inputs and/or responses to the subsequent user inputs. The described screenshots may be accomplished using some or all of the system components described in detail above. With respect to screenshot 700, a user may interact with the system (or a personal assistant application thereof) by providing a number of natural language inputs, which results in immediate system responses that adequately addresses the inputs. In some cases, updated information relating to the user inputs may be available at a subsequent time during this interaction between the user and the system. In such cases, the updated information (e.g., driver positions in a race) may be provided (e.g., in the form of follow-up responses) in real-time to the user as it becomes available, without a prompt or subsequent related input from the user. In other cases, the updated information (e.g., current estimated time to the user's home) may be provided based on a subsequent natural language input (e.g., “I′m leaving for home”) from the user.

Other implementations, uses and advantages of the invention will be apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. The specification should be considered exemplary only, and the scope of the invention is accordingly intended to be limited only by the following claims.

Claims

What is claimed is:

1. A method for providing follow-up responses to prior natural language inputs of a user, the method being in implemented on a computer system having one or more physical processors programmed with computer program instructions which, when executed, programs the computer system to perform the method, the method comprising:

receiving, at the computer system, a first natural language input associated with a user;

determining, at the computer system, a first context of the first natural language input;

determining, at the computer system, that first information for providing a first response to the first natural language input is not available:

receiving, at the computer system, a second natural language input associated with the user after the receipt of the first natural language input;

determining, at the computer system, a second context, different from the first context, of the second natural language input;

providing, by the computer system, a second response to the second natural language input;

receiving, at the computer system, the first information for providing the first response after the second response is provided;

generating, at the computer system, the first response based on the second context relating to the second natural language input; and

providing, by the computer system, the first response, different than the second response based on the first information.

2. The method of claim 1, further comprising:

determining, by the computer system, whether information sufficient for providing an adequate response to the natural language input is currently accessible to the computer system; and

providing, by the computer system, based on a determination that information sufficient for providing an adequate response to the natural language input is not currently accessible, a first response to the natural language input that indicates that a follow-up response will be provided.

3. The method of claim 1, further comprising:

determining, by the computer system, context information associated with an interactive session between the computer system and the user, wherein providing the response comprises providing the response based on the first information and the context information associated with the interactive session.

4. The method of claim 1, wherein the response is provided to the user in real-time when the first information is received.

5. A system for providing follow-up responses to prior natural language inputs of a user, the system comprising: one or more physical processors programmed with computer program instructions which, when executed, cause the one or more physical processors to:

receive a first natural language input associated with a user;

determine a first context of the first natural language input;

determine that first information for providing a first response to the first natural language input is not available;

receive a second natural language input associated with the user after the receipt of the first natural language input;

determine a second context, different from the first context, of the second natural language input:

provide a second response to the second natural language input;

receive the first information for providing the first response after the second response is provided;

generate the first response based on the second context relating to the second natural language input; and

provide the first response, different than the second response based on the first information.

6. The system of claim 5, wherein the one or more physical processors are further caused to:

determine whether information sufficient for providing an adequate response to the natural language input is currently accessible to the computer system; and

provide, based on a determination that information sufficient for providing an adequate response to the natural language input is not currently accessible, a first response to the natural language input that indicates that a follow-up response will be provided.

7. The system of claim 5, wherein the one or more physical processors are further caused to:

determine context information associated with an interactive session between the computer system and the user, wherein to provide the response the one or more physical processors are caused to provide the response based on the first information and the context information associated with the interactive session.

8. The system of claim 5, wherein the response is provided to the user in real-time when the first information is received.