US20060009973A1 - Multi-slot dialog systems and methods - Google Patents

Multi-slot dialog systems and methods Download PDF

Info

Publication number
US20060009973A1
US20060009973A1 US10/885,934 US88593404A US2006009973A1 US 20060009973 A1 US20060009973 A1 US 20060009973A1 US 88593404 A US88593404 A US 88593404A US 2006009973 A1 US2006009973 A1 US 2006009973A1
Authority
US
United States
Prior art keywords
slot
user
slots
correction
segment
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
US10/885,934
Other versions
US7228278B2 (en
Inventor
Patrick Nguyen
Jesus Amaro
Amit Desai
Adeeb Shana'a
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
24 7 AI Inc
Original Assignee
Voxify Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Family has litigation
First worldwide family litigation filed litigation Critical https://patents.darts-ip.com/?family=35542461&utm_source=google_patent&utm_medium=platform_link&utm_campaign=public_patent_search&patent=US20060009973(A1) "Global patent litigation dataset” by Darts-ip is licensed under a Creative Commons Attribution 4.0 International License.
Application filed by Voxify Inc filed Critical Voxify Inc
Priority to US10/885,934 priority Critical patent/US7228278B2/en
Assigned to VOXIFY, INC. reassignment VOXIFY, INC. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: DESAI, AMIT V., LOPEZ-AMARO, JESUS, NGUYEN, PATRICK T.M., SHANA'A, ADEEB W.M.
Priority to ES10190283.1T priority patent/ES2621808T3/en
Priority to ES05768313.8T priority patent/ES2526698T3/en
Priority to JP2007520443A priority patent/JP2008506156A/en
Priority to PCT/US2005/023844 priority patent/WO2006014432A2/en
Priority to EP10190283.1A priority patent/EP2282308B1/en
Priority to EP05768313.8A priority patent/EP1779376B1/en
Publication of US20060009973A1 publication Critical patent/US20060009973A1/en
Priority to US11/787,763 priority patent/US7747438B2/en
Publication of US7228278B2 publication Critical patent/US7228278B2/en
Application granted granted Critical
Assigned to 24/7 CUSTOMER, INC. reassignment 24/7 CUSTOMER, INC. PATENT ASSIGNMENT Assignors: VOXIFY, LLC
Assigned to [24]7.ai, Inc. reassignment [24]7.ai, Inc. CHANGE OF NAME (SEE DOCUMENT FOR DETAILS). Assignors: 24/7 CUSTOMER, INC.
Active legal-status Critical Current
Adjusted expiration legal-status Critical

Links

Images

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04MTELEPHONIC COMMUNICATION
    • H04M3/00Automatic or semi-automatic exchanges
    • H04M3/42Systems providing special services or facilities to subscribers
    • H04M3/487Arrangements for providing information services, e.g. recorded voice services or time announcements
    • H04M3/493Interactive information services, e.g. directory enquiries ; Arrangements therefor, e.g. interactive voice response [IVR] systems or voice portals
    • H04M3/4936Speech interaction details
    • GPHYSICS
    • G10MUSICAL INSTRUMENTS; ACOUSTICS
    • G10LSPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00Speech recognition
    • G10L15/22Procedures used during a speech recognition process, e.g. man-machine dialogue

Definitions

  • the present invention relates generally to speech recognition systems. More specifically, systems and methods for constructing a series of interactions with a user to collect multiple pieces of related information for the purpose of accomplishing a specific goal or topic (a multi-slot dialog) using a component-based approach are disclosed.
  • Speech recognition systems are a promising method for automating service functions without requiring extensive changes in user behavior. Many companies have sought to expand or improve their customer service functions by using speech recognition technology to automate tasks that have traditionally been handled by human agents. To achieve this, speech recognition systems should allow a user to ask for and provide information using natural, conversational spoken input. Recent advances in certain areas of speech recognition technology have helped alleviate some of the traditional obstacles to usable speech recognition systems. For example, technology advances have enabled unrehearsed spoken input to be decoded under a wider range of realistic operating conditions, such as background noise and imperfect telephone line quality. Additionally, recent advances have allowed voice applications to recognize voice inputs from a broader population of users with different accents and speaking styles.
  • APIs application program interfaces
  • voice applications present a new user interaction model that is sufficiently distinct from the (well understood) graphical user interface to require specialized design and implementation expertise.
  • Speech interface concepts such as dialog management, grammar optimization, and multi-slot interfaces, are manually implemented in every custom-built voice system. Given the relative newness of the speech paradigm, this further burdens the developers.
  • the demands on applications to handle presentation, business logic, and data access functions resulted in piecemeal architectures combining static and dynamically generated documents, backend servlets, grammars, and other disjoint components.
  • a number of products are available to simplify the development of enterprise voice applications.
  • a central element of many of these products is a library of predefined and customizable voice components whose use reduces the amount of code that needs to be developed by a programmer. These components usually encapsulate the voice language code, grammars, internal call flows, prompts and error recovery routines required to obtain one piece of information from the caller, such as a date, a time, a dollar amount, a sequence of digits, or an item from a set or list of allowable items (such as a set of airports).
  • a major limitation of these component frameworks is that the components are not combinable to allow the user to provide multiple pieces of information in each utterance.
  • a flight reservation application could use four components: a departure airport, a destination airport, a departure date and a departure time.
  • the existing frameworks would allow a user to provide the four pieces of information in four separate utterances.
  • the application were to allow the user to say the departure airport, destination airport and departure date in one utterance (e.g. “I'm flying from Boston to San Francisco on Monday”), the departure airport, destination airport, and departure date components could not be simply combined. Instead, a new component would need to be developed with new grammars, call flows, prompts, etc. to recognize the two airports and the date.
  • the application were to allow the caller to retain some pieces of information while changing others pieces of information (e.g. “No, I'm actually flying to Oakland on Tuesday”), an even more complex component would have to be developed.
  • voice applications that rely on existing component frameworks implement highly directed dialogs in which the call flow is largely predetermined and each step accepts only a single item of information, such as in an interchange illustrated in FIG. 1 a .
  • Such voice systems are rigid and often penalize a caller who provides too much information, such as in an interchange illustrated in FIG. 1 b .
  • these systems are neither intuitive nor efficient since they cannot capture information rapidly or adapt to the user's preferences for providing information.
  • the method generally includes outputting a primary header prompt to elicit values for slots in a segment from the user, receiving a primary user response containing a value for each slot in at least a subset of the slots in the segment, processing the primary user response to determine at least one possible recognition value for each slot contained in the primary user response, filling each slot contained in the primary user response with a matched value selected from the corresponding possible recognition values, and repeating the outputting, receiving, processing and filling for any unfilled slots in the segment until all slots in the segment of slots are filled.
  • the method may include performing turns to confirm and/or clarify the matched slot values such as by silently accepting a best match, passively confirming the best match, actively confirming the best match, disambiguating among the best matches, and notifying the user of non-recognition.
  • the method for confirmation and/or clarification may be selected based on, for example, the number of possible recognition values for the slots in the primary user response and a corresponding confidence level for each of the possible recognition values.
  • a user confirmation response is recognized as a confirmation, a cancellation, or a cancellation and correction.
  • the correction is processed by determining at least one possible recognition value for each slot contained in the correction.
  • a passive confirmation a passive confirmation prompt is output with a next primary header prompt.
  • the method may also include enabling any unfilled slots in the segment of slots, in which the primary header prompt elicits values for enabled slots in the segment.
  • the method may be text- or speech-based.
  • a system for constructing and processing a multi-slot dialog with a user may generally include slot objects representing slots in a segment, each slot capable of being assigned a value based on the multi-slot dialog, at least one slot segment object each containing a corresponding set of slot objects, at least one set of slot group objects for each slot segment object, each slot group object defining parameters associated with a particular combination of slots in the slot segment object, and dialog objects that define a flow of the multi-slot dialog.
  • the system may be implemented in an object-oriented programming language.
  • a method for constructing a multi-slot dialog with a user to obtain multiple items of information over a number of turns may generally include providing at least one slot segment object, each slot segment object containing a corresponding set of slot objects, each representing a slot in a segment, each slot representing an item of information to be provided by the user, providing at least one set of slot group objects for each slot segment object, each slot group object defining parameters associated with a particular combination of slots in the slot segment object, and executing a multi-slot dialog flow defined by dialog objects.
  • FIG. 1 a and FIG. 1 b illustrate examples of highly directed dialogs using conventional voice applications.
  • FIGS. 2 a - 2 c illustrates various examples of multi-slot dialogs.
  • FIG. 3 is a block diagram of an illustrative multi-slot voice application system.
  • FIG. 4 illustrates one embodiment of a framework for managing a multi-slot speech recognition-based conversation.
  • FIG. 5 illustrates examples of segments for a multi-slot flight reconfirmation dialog.
  • FIG. 6 illustrates the contents of one of the exemplary segments, namely, the flight itinerary, in the multi-slot flight reconfirmation dialog of FIG. 5 .
  • FIG. 7 illustrates examples of exchanges involved in a flight itinerary dialog.
  • FIG. 8 is a flowchart illustrating various steps of an exemplary multi-slot dialog.
  • FIG. 9 is a flowchart of an exemplary exchange using active confirmation.
  • FIG. 10 is a flowchart of an exemplary exchange using passive confirmation.
  • FIG. 11 is a flowchart illustrating an example of a conversation utilizing a go back functionality of a multi-slot dialog system.
  • FIG. 12 is a flowchart illustrating an example of a conversation utilizing a change functionality of the multi-slot dialog system.
  • FIG. 13 is a flowchart illustrating an example of a conversation utilizing a review functionality of the multi-slot dialog system.
  • FIGS. 14 and 15 illustrate some of the possible slot group objects for the pre-recognition and post-recognition slot group classes for a flight itinerary segment, respectively.
  • FIG. 16 is a block diagram illustrating an exemplary dialog flow structure.
  • FIGS. 17 and 18 are flowcharts illustrating exemplary processing of the user's input in normal and passive confirm modes, respectively.
  • FIG. 19 illustrates a possible grammar for an exchange with three slots.
  • FIG. 20 illustrates a possible grammar for a confirmation exchange that includes accept and cancel grammars as well as grammars for the three slots being confirmed.
  • the multi-slot dialog systems and methods obtain information from a user by conducting a speech-recognition based series of interactions.
  • the systems and methods include determining the prompts output to the user as well as the grammars and semantic rules utilized to recognize user inputs such as utterances at each point in the conversation or interchange.
  • FIGS. 2 a - 2 c illustrate various examples of multi-slot dialogs.
  • a multi-slot dialog has the objective of collecting multiple pieces of related information (“slots”) for the purpose of accomplishing a specific goal or topic, such as locating an airline reservation.
  • a multi-slot dialog application preferably handles certain behavior and interactions typical of human interactions in a spoken medium, including: (i) a caller may provide the slots in an arbitrary order, (ii) a caller may provide multiple slots in a single input such as a spoken utterance, (iii) a caller may provide only a subset of slots requested by the application in a single utterance, (iv) a caller may clarify or correct the application's interpretation of slots the caller has provided, (v) a caller may modify earlier slots in subsequent utterances.
  • dialog application may perform a lengthy and sophisticated call path of considerable complexity.
  • conventional voice applications are ill-suited for implementing multi-slot dialogs.
  • dynamic order and combinations in which information may be provided cannot be easily handled by existing component frameworks and development methodologies of conventional voice applications that specify rigid, predetermined call flows.
  • the multi-slot dialog systems and methods as described herein may be utilized for constructing multi-slot dialog applications using a component-based approach.
  • component-based approach automates the multi-slotting process with components for the behavioral elements of a multi-slotting exchange between human and machine, including sequencing, grouping, prompting, confirmation, and/or modification strategies.
  • FIG. 3 is a block diagram of an illustrative voice application system 300 that generally includes a telephone 301 in communication with a voice browser 303 via a telephone network 302 which is in turn in communication with a voice application 309 via a data network 308 .
  • the voice browser 303 includes the hardware and software for conducting bi-directional audio communications with a caller via the telephone network 302 and the telephone 301 .
  • the voice browser 303 may execute a program expressed in a voice language transmitted in the form of documents over the data network 308 (such as the Internet or an intranet) from the voice application 309 .
  • the voice browser 303 and voice application 309 may reside on any of various suitable computer systems.
  • the voice language may be a markup language such as VoiceXML or Speech Application Language Tags (SALT).
  • the voice browser 303 may include various components such as a speech recognition engine 304 , a text-to-speech synthesizer 305 , an audio playback player 306 for audibly rendering files recorded using generally available audio formats, and a component for handling calls over the telephone network 307 .
  • a speech recognition engine 304 a text-to-speech synthesizer 305
  • an audio playback player 306 for audibly rendering files recorded using generally available audio formats
  • a component for handling calls over the telephone network 307 a component for handling calls over the telephone network 307 .
  • Commercially available and/or proprietary components for building voice applications may be employed in implementing any or all of the various components of the voice browser 303 .
  • the voice browser 303 may be responsible for detecting an incoming call, answering the incoming call, requesting an initial voice document from the voice application 309 , and interpreting the voice document and executing instructions contained in the voice document according to the rules of the applicable voice language.
  • the instructions may include the outputting of audible prompts to the user and the processing of voice responses from the user using specified grammars.
  • the voice browser 303 may utilize the audio playback player 306 to audibly render prerecorded messages or may utilize the text-to-speech synthesizer 305 to audibly render text messages.
  • the voice browser 303 can then generate actions in response to user events (such as spoken input or call disconnection) or system events (such as subsystem exceptions), and/or actions that may be defined in the current document or in another document to be fetched from the voice application 309 .
  • user events such as spoken input or call disconnection
  • system events such as subsystem exceptions
  • FIG. 4 illustrates one embodiment of a multi-slot framework 401 for managing a multi-slot speech recognition-based conversation.
  • the multi-slot framework 401 implements a multi-slot mode for obtaining information from a user.
  • the framework 401 may include a multi-slot application 402 that manages the business logic and data access responsibilities for a set of slots 403 .
  • a slot is referred to herein as a data item whose value can be obtained from a user input such as a text input or a spoken utterance.
  • a flight reservation application may manage a departure airport slot whose valid values are found in a flight schedule database, and whose value, once uttered or otherwise entered by the user, should be stored in a reservation record.
  • a multi-slot platform 404 constructs one or more voice interactions to obtain the desired slots from the user.
  • FIG. 5 illustrates examples of segments for a multi-slot flight reconfirmation dialog
  • FIG. 6 illustrates the contents of one of the exemplary segments, namely, the flight itinerary, in the multi-slot flight reconfirmation dialog.
  • a multi-slot dialog can be decomposed into sub-units at various levels. At the smallest level, a turn is an uninterrupted stream of input (e.g., speech) from one participant, i.e., the system or the user. Thus a system turn is one in which the system prompts the user with a message and a user turn is one in which the user makes an utterance that the system attempts to interpret.
  • An exchange is a series of turns that captures the values for one or more slots from the user.
  • an exchange may include one or more confirmation, correction, or disambiguation turns until a final, single set of slot values is accepted by the system.
  • a segment is a series of exchanges that captures a set of related slots. The segment is the largest unit of a dialog that allows all its associated slots to be provided in one user turn.
  • the five slots namely, a departure airport, a destination airport, a date, a time, and an AM/PM indicator, form the flight itinerary segment such that up to all five slots of the flight itinerary segment may be entered in one user turn such as by the user uttering “from San Francisco to Boston tomorrow at 9AM.”
  • Multi-slot dialogs can vary in complexity from single segment dialogs to complex, multi-segment dialogs in which each segment includes one or more slots.
  • An example of a single-segment dialog is a phone attendant application that requests an employee name from the caller.
  • an example of a multi-segment dialog is a flight reconfirmation dialog such as that shown in FIG. 5 .
  • the flight reconfirmation dialog may request a flight itinerary from the caller (which may include slots for the departure and destination airports as well as the date, time and AM/PM indicator of the departure as shown in FIG. 6 ), and may verify the caller's identity by requesting a record locator and by requesting personal details, such as a surname.
  • FIG. 7 illustrates examples of exchanges involved in a flight itinerary dialog.
  • the flight itinerary segment 701 contains 5 slots that represent a flight itinerary.
  • the system may prompt the user with “What is your flight itinerary?” to allow the user to enter information or values for up to all 5 slots.
  • the user may respond with “I'm flying from Boston” and after any applicable confirmation/clarification turns, the application accepts Boston as the value for the departure airport slot.
  • the application determines, for example, that only the destination airport should be prompted for in the next exchange 703 , and outputs a prompt “And what is your destination?”
  • the value San Francisco is stored in the destination airport slot.
  • the application may then prompt for the remaining three unfilled slots in the next exchange 704 such as by outputting a prompt “And when are you flying?”
  • the user's response of “next Friday at nine thirty AM” fills all three remaining slots and the dialog is complete.
  • FIG. 8 is a flowchart illustrating an exemplary multi-slot conversation 800 .
  • the voice application system outputs a primary header prompt to elicit values for a set of slots from the user.
  • the system prompt is typically worded in such a way that the user may potentially provide values for a number of desired slots in one turn.
  • the user responds with a primary user turn in response to the output prompt. The user turn may provide one or more slot values.
  • a number of confirmation/clarification turns at block 806 may follow in which the system may optionally silently accept the best match, passively confirm the best match, demand active confirmation of the best match, disambiguate among the best matches, or notify the user of a non-recognition, for example, as will be described in more detail below.
  • the slot values from the turn are confirmed either explicitly or implicitly, the slots are filled with their respective values at block 808 .
  • Blocks 802 - 808 represent one exchange.
  • the system determines if there are any remaining unfilled slots in the current segment at decision block 810 . If all slots in the current segment are filled, then the system determines if there are any additional segment(s) of slots at decision block 812 . If all slots in all segments are filled, then the multi-slot dialog 800 is complete. Alternatively, if there are remaining unfilled slots in the current segment and/or if there are additional segment(s) of slots, the dialog 800 returns to block 802 to initiated the next exchange. For example, the next exchange may either follow up on the previous exchange by prompting for any remaining slots that were not filled (e.g., “ . . . and around what time?”) or continue onto the next segment of slots by prompting for a new set of slots (e.g., “What type of car do you want to rent?”).
  • confirmation and/or clarification turns may follow in which the system may optionally demand active confirmation of the best match, passively confirm the best match, disambiguate among the best matches, silently accept the best match, or notify the user of a non-recognition, for example. Details of the confirmation/clarification turns will now be described in more detail with reference to FIGS. 9 and 10 .
  • FIG. 9 is a flowchart of an exemplary exchange using active confirmation.
  • a confirmation prompt is output to the user (e.g., “I think you said Austin to San Francisco. Is that correct?”).
  • the confirmation prompt may escalate during an exchange if there are several confirmation cycles.
  • the system may require that the user explicitly accept the value(s) for the associated slots prior to filling the slots.
  • the system may optionally accept an utterance by the user that includes a cancel phrase followed by a correction such as “No, I said Boston to San Francisco.” If the user cancels the previous recognized values, the system may clear all slot values recognized in the user turn and play a message such as “Let's try that again. What is your flight itinerary?” as shown in FIG. 9 and begin the exchange again. Alternatively, the system may treat the user response uttered in the user turn as a correction and repeat the confirmation/clarification turn, e.g., by prompting “OK. Boston to San Diego. Is that correct?” If the correction omits some of the slots that are being confirmed, the system may retain the previously uttered values of such slots.
  • FIG. 10 is a flowchart of an exemplary exchange using passive confirmation.
  • the multi-slot dialog system outputs a prompt that is a combination of a passive confirmation prompt and a header prompt for the next exchange.
  • the combined output prompt may be, for example, “Ok. Austin to San Francisco. And when are you flying?”
  • the passive confirmation technique facilitates in speeding up the dialog and thus is suitable for confirming recognition results where the system has a high level of confidence.
  • the slots are filled with their respective values and those filled values may be removed or rolled back when the user utters a cancel phrase such as “No” or “No, that's wrong,” or a cancel phrase followed by a correction such as “That's wrong. I said Boston to San Francisco.”
  • a cancel phrase such as “No” or “No, that's wrong,” or a cancel phrase followed by a correction such as “That's wrong. I said Boston to San Francisco.”
  • the system may clear the slot values accepted in the previous exchange, issue a prompt such as “Sorry, let's try that again,” and repeat the previous exchange.
  • the system may treat the user response uttered in the user turn as a correction and repeat the confirmation/clarification turn, e.g., by prompting “OK. Boston to San Francisco.
  • the header prompt for the new exchange is repeated.
  • an active confirmation may be utilized after the user issues a cancel phrase in response to a passive confirmation.
  • other confirmation types may be similarly utilized after the user issues a cancel phrase in response to a passive confirmation. If the next exchange contains a prompt for a “Yes/No” response, the functionality to rollback a previous passive confirm of the passive confirmation may be disabled.
  • the system outputs a prompt containing a list of the top matches and requests the user to select one of the top matches, e.g., by its corresponding number.
  • Each match may include a single slot value or a set of slot values and may be presented by a prompt similar to a confirmation prompt that contains all the recognized slot values.
  • the corresponding value or values are treated as if they had been uttered in the primary user turn and the system repeats the confirmation/clarification process.
  • the system may also notify the user of a non-recognition.
  • the system may issue an exception prompt such as “I didn't get that” and repeat the previous system turn or an amended version thereof.
  • the system may be configured with various features and functionalities to further facilitate the exchange as will be described below in more detail with reference to FIGS. 11-13 .
  • the system may be configured to maintain a skip list in a given exchange in which a skip list entry corresponding to the set of slot values presented in a confirmation is added each time a user cancels the confirmation during that exchange.
  • the skip list helps to ensure that, within the same exchange, the system does not utilize and thus will not present again a set of values that matches an entry in skip list. Instead, the system may utilize the next best match when available.
  • FIG. 11 is a flowchart illustrating an example of a conversation that includes yet another optional functionality of the system, namely, a go back functionality.
  • the user may utter a go back command, e.g., “go back,” at any time so as to return to the start of the previous turn, the start of the current exchange, or the start of the current segment, depending on the application. If the system goes back over a step that has filled some slots, these slots may be cleared.
  • the multi-slot dialog system may be configured to adaptively present a more directed prompt after a threshold number of exceptions, corrections, or gobacks has occurred during an exchange. For example, the system may present a new, more directed header prompt “What date will you be picking up the car? For example, say tomorrow, next Tuesday, or July 24 th .” If the exceptions, corrections or gobacks continue to occur, the system may transfer the user to a live operator for further assistance.
  • FIG. 12 is a flowchart illustrating an example of a conversation that includes yet another optional functionality of the system, namely, a change functionality.
  • the change functionality allows the user to change a previously filled slot value by uttering, for example, “I'd like to change the airport.” If the user's utterance requesting a change does not fill all the slots required for a change request, the system initiates a follow-up exchange such as “Departure or destination airport?”
  • the change command may optionally be confirmed using any of the confirmation mechanisms described above. For example, the system may actively confirm the change command by prompting “I heard you say you wanted to change the arrival airport. Is that correct?”
  • the change command cancels the exchange the user is currently in and clears at least some of the previously filled slot values.
  • a new exchange then begins that prompts the user for the cleared slots which can be filled in one or more exchanges. Once the cleared slots are filled, the system continues processing and will bypass some previous exchanges if the slots associated with those exchanges are still filled.
  • FIG. 13 is a flowchart illustrating an example of a conversation that includes yet another optional functionality of the system, namely, a review functionality.
  • the user may request to review a previously filled slot value by uttering a request phrase such as “Can I check my date of departure?”
  • the system plays the filled slot value such as “You're flying on Friday, Aug. 9, 2002” and returns to the current exchange. If the user does not provide the values for all the slots required for a review request, the system initiates a follow-up exchange such as “Departure or return date?”
  • the platform for implementing multi-slot dialogs is described herein as being implemented utilizing Java.
  • the system may be implemented using any suitable programming language, preferably an object-oriented programming language such as Java or C++.
  • the system generally includes slot objects, slot group objects, segment objects, and dialog objects. Each of these objects is described below.
  • a slot object represents a slot which, as noted above, is an item of information to be obtained from the user.
  • a slot object contains the grammar that recognizes the possible values that can be provided by the user for the corresponding slot, the rules that map grammar recognition results to semantic values for the slot, and the variables indicating the enabled and filled state (among other state variables) for the slot.
  • the slot objects can be based on a Java interface that provides basic default functionality and/or functionality common to all slot objects.
  • the grammar that recognizes the possible values that can be provided by the user for the corresponding slot is a formal specification of the utterances the system accepts for expressing the possible values for the slot.
  • the grammar may include the vocabulary of words that may be used and valid structures for sequencing those words.
  • the grammar for a date slot should allow various date formats to be recognized, ranging from absolute specifications such as “January the twelfth two thousand and four” to relative forms such as “this coming Friday” and familiar terms such as “today” and “yesterday.”
  • the grammar may also include fillers that may precede and/or follow the slot value in a user's utterance but that do not specify or distinguish one slot value from another.
  • an airport slot may have the preceding filler phrase “I'm flying from.”
  • Some grammars may be highly application specific such as the grammar for the allowable airports in a flight reservation application.
  • Other grammars may be reused across applications, such as the grammar for a date, a time, or a dollar amount.
  • the common portion of these grammars may be predefined in a grammar object and customized for a particular slot.
  • each slot object also contains rules that map the grammar recognition results to semantic values for the corresponding slot that are meaningful to the specific application. For example, a destination airport slot object may map the utterances “San Francisco,” “San Francisco Airport,” “San Francisco International Airport,” and “SFO” to a single airport identifier such as “SFO.” As another example, a date slot object may map the utterance “tomorrow” to a date value that is computed as the next date following the current date.
  • Each slot object also maintains a number of state variables or flags used during the execution of a multi-slot dialog to indicate the enabled and filled state (among other state variables) for the corresponding slot.
  • flags include enabled, optional, filled, and pending flags.
  • an enabled flag is set to true to indicate that a slot can be accepted in the upcoming user turn.
  • An optional flag is set to true if an enabled slot does not need to be explicitly provided by the user.
  • a filled flag is set to true when a slot value has been accepted after any applicable confirmation/clarification.
  • a pending flag is set to true if a value for the slot has been recognized from the user but has not yet been accepted, e.g., pending confirmation.
  • the system maintains slot group objects for each slot segment, examples of which are shown in FIGS. 14 and 15 for a flight itinerary segment.
  • Each slot group object defines parameters or properties associated with a particular group or combination of slots in the slot segment.
  • Examples of slot group properties include prompts such as header or main prompts, help prompts, error prompts, confirmation prompts, and/or disambiguation prompts, as well as recognition properties, i.e., variables that affect recognition behavior such as timeouts, recognition thresholds, recognition parameters, caching policies and so on.
  • Different slot group classes e.g., pre-recognition and post-recognition slot group classes as shown in FIGS. 14 and 15 , respectively, may be defined to maintain separate sets of properties.
  • FIG. 14 illustrates some of the possible slot group objects for the pre-recognition slot group class for the flight itinerary segment.
  • the pre-recognition slot group class may contain the prompts and recognition properties used before slot values are recognized, such as the header or main prompts, the help prompts, no match prompts, no input prompts, timeouts, confidence thresholds and so on.
  • the slot combination would typically be compared to the currently enabled set of slots.
  • FIG. 15 illustrates some of the possible slot group objects for the post-recognition slot group class for the flight itinerary segment.
  • the post-recognition slot group class may contain the prompts and recognition properties used after slot values are recognized, such as the active confirmation prompts, disambiguation prompts, passive confirmation prompts, and so on.
  • the slot combination would typically be compared to the currently pending set of slots, i.e., the slots recognized from the user but not yet confirmed.
  • the system When a parameter such as a prompt or a recognition property is required at a point in a multi-slot dialog, the system identifies a best match slot group object from a slot group class that contains the parameter and looks up the value of the desired parameter in the identified best match slot group object. For example, where a header prompt is required at a point in a multi-slot dialog in order for the system to elicit from the user the values for the slots in the currently enabled slot combination, e.g., date, time, and AM-PM, the system identifies a best match slot group object from the pre-recognition slot group class in FIG. 14 . The system selects the slot group object whose slot combination is the closest to the currently enabled slot combination.
  • a header prompt is required at a point in a multi-slot dialog in order for the system to elicit from the user the values for the slots in the currently enabled slot combination, e.g., date, time, and AM-PM
  • the system identifies a best match slot group
  • the closest slot combination may be determined utilizing various suitable methods such as the most number of overlapping slots, the fewest number of non-overlapping slots in the slot group object, or the fewest number of non-overlapping slots in the enabled group.
  • the system identifies and utilizes the slot group object in the pre-recognition slot group class shown in FIG. 14 having an exact match slot combination as the currently enabled slot combination, i.e., date, time, and AM-PM.
  • another group object deemed as the closest may be identified and utilized, e.g., a group object with a two-slot combination such as time and AM-PM slots.
  • the system may define a separate slot group object for each slot of the slot segment to ensure that a slot group can always be found for any enabled slot combination.
  • the system also includes segment objects.
  • a segment object maintains a set of slots in a slot segment that determines a series of one or more successive multi-slot exchanges.
  • the values for the slots in a given slot segment may potentially be obtained from the user in a single exchange. Alternatively, if the user does not provide the values for all the slots in that segment in one exchange, follow-up exchanges are initiated until all required, i.e., non-optional, slots are filled. When a segment is filled completely, the next segment, if any, is then invoked.
  • FIG. 16 is a block diagram illustrating an exemplary dialog flow structure. After the system initiates a dialog at block 1601 , the system obtains the first segment in the dialog at block 1602 . The system determines the slots in this segment that should be enabled at block 1603 such as by including the slots that are not yet filled. The closest pre-recognition slot group is selected to retrieve the header prompt and other pre-recognition parameters at block 1604 (such as the help prompt and any exception prompts). The header prompt is usually worded in such a way that the user may potentially provide a number of desired slots in one utterance.
  • the prompt could be “When will you be picking up the car?”
  • the user's response is then received and processed by the system at block 1605 such as by obtaining the best hypotheses of the user's intended word sequence from the speech recognition engine and performing any desired confirmation, clarification or disambiguation based on the custom settings of the application until a single set of slot values is accepted by the system.
  • the system determines whether the slot segment contains any more slots to be filled at decision block 1606 which usually includes slots whose values have not yet been filled. It is noted that the system, in determining whether the slot segment contains any more slots to be filled at decision block 1606 , may apply application-specific logic specifying that some slot values may or should be automatically filled from other slot values, that some slots are optional, or that certain additional slots need to be filled as a result of the values of other accepted slots. If some slots do remain to be filled as determined in decision block 1606 , the system returns to block 1603 to enable the next set of slots. Otherwise, the system continues to decision block 1607 to determine if there are any more slot segments. If there is at least one more slot segment remaining, the system obtains the next segment at block 1608 and returns to block 1603 . Alternatively, if no other slot segments remain, the dialog ends at block 1610 .
  • the system When processing the user's input, the system may be in a normal mode or passive confirm mode. Passive confirm mode is activated when the system determines that the user's response should be passively confirmed based on some predefined criteria. One such set of criteria could be that the confidence level returned by the speech engine is below the threshold for silent acceptance but above the threshold requiring active confirmation. If passive confirm mode is not activated, the system may be in normal mode by default.
  • FIG. 17 is a flowchart illustrating an exemplary processing of the user's input in normal mode while FIG. 18 is a flowchart illustrating an exemplary processing the user's input in passive confirm mode.
  • the determination as to normal versus passive mode may be based on, for example, a weighted or average confidence level for the set of slots to which the user's response corresponds.
  • the determination may be made with a single determination for all slots in the user response or a separate determination for each slot in the user response and the separate results averaged or weighted, for example.
  • the speech engine recognizes the user's input and returns one or more hypotheses at block 1701 . If several possible matches are returned by the speech engine as determined at decision block 1702 , the possible matches may be disambiguated at block 1708 . For example, the user may be presented with a list of top matches and asked to select one. Each match may be presented by a disambiguation prompt similar to a confirmation prompt. When a match selected by the user is received and processed at block 1709 , the system proceeds to the next exchange at block 1710 .
  • the system determines whether the confidence level for the one hypothesis is at or above a predefined high confidence threshold at decision block 1703 . If the high confidence level is at or above the high confidence threshold, the system accepts the slot values and enters passive confirmation mode at block 1707 . Alternatively, if the high confidence level is below the high confidence threshold, the system actively confirms the best match by outputting a confirmation prompt at block 1704 . For example, the system may construct the confirmation prompt by identifying the slot group in the post-recognition slot group class that is closest to the group of slots in the best match and retrieving the corresponding active confirmation prompt.
  • the user's response to the confirmation prompt is received and processed at block 1705 . If the user cancels the confirmation, e.g., by uttering a cancel phrase such as “no” or “that's wrong,” and provides a correction, the corrected values may return to block 1704 for confirmation. If the user cancels the confirmation with no correction, the current exchange is repeated at block 1706 . In other words, any pending slot values are discarded or cleared and the system repeats the header prompt that was originally played. If the user accepts the confirmation, e.g., by uttering a phrase such as “yes” or “that's right,” the system accepts the slots values and proceeds to the next exchange at block 1710 .
  • the confirmation prompt from one exchange is combined with the header prompt for the next exchange.
  • the combined prompt may be “Ok. Boston. And what date will you be flying?”
  • the user's response is then recognized by the speech engine at block 1801 . If the user's response does not begin with an accept or cancel phrase, i.e., a yes or no lead, as determined at decision block 1802 , the user's response is processed as in the case of the normal mode described above. Alternatively, if the user response begins with an accept or cancel phrase as determined at decision block 1802 , the previous slot values may be affected.
  • the header prompt for the new exchange may simply be repeated, for example “And what date will you be flying?” at block 1803 .
  • the system may clear the slot values accepted in the previous exchange, play a message such as “Sorry, let's try that again,” and repeat the previous exchange at block 1804 .
  • the corrected values are explicitly confirmed at block 1805 until the previous exchange's slots are explicitly accepted or cancelled.
  • FIG. 19 illustrates a possible grammar for an exchange with three slots. Each branch of the diagram represents a grammar rule that could match the user's utterance.
  • the grammars for slots 1 , 2 , and 3 are represented by reference numbers 1901 , 1902 , 1903 , respectively.
  • the postfix operator “?” applied to each of the grammars 1902 , 1903 corresponding to slots 2 and 3 in the first branch indicates that slots 2 and 3 are optional such that in this first branch the user's utterance contains a value for slot 1 , optionally a value for slot 2 , and, if there is a value for slot 2 , optionally a value for slot 3 .
  • an application may restrict the allowed combination of slots (such as a slot having to come before another) depending on the norms of the language used and the context of a particular exchange.
  • the grammar for a confirmation turn should thus be constructed to include the accept and cancel grammars as well as grammars for the slots being confirmed, an example of which is shown in FIG. 20 .
  • the accept grammar 2001 contains a set of phrases that express an affirmative response, such as “yes,” “that's right,” and “correct.”
  • the cancel grammar 2002 contains a set of phrases that express a negative response, such as “no,” “that's wrong” and “incorrect.” If the cancel grammar is present, an optional correction grammar 2003 is included which recognizes new values for the slots being cancelled.

Abstract

Systems and methods for constructing a series of interactions with a user to collect multiple pieces of related information for the purpose of accomplishing a specific goal or topic (a multi-slot dialog) using a component-based approach are disclosed. The method generally includes outputting a primary header prompt to elicit values for slots in a segment from the user, receiving a primary user response containing a value for each slot in at least a subset of the slots in the segment, processing the primary user response to determine at least one possible recognition value for each slot contained in the primary user response, filling each slot contained in the primary user response with a matched value selected from the corresponding possible recognition values, and repeating the outputting, receiving, processing and filling for any unfilled slots in the segment until all slots in the segment of slots are filled.

Description

    BACKGROUND OF THE INVENTION
  • 1. Field of the Invention
  • The present invention relates generally to speech recognition systems. More specifically, systems and methods for constructing a series of interactions with a user to collect multiple pieces of related information for the purpose of accomplishing a specific goal or topic (a multi-slot dialog) using a component-based approach are disclosed.
  • 2. Description of Related Art
  • Speech recognition systems are a promising method for automating service functions without requiring extensive changes in user behavior. Many companies have sought to expand or improve their customer service functions by using speech recognition technology to automate tasks that have traditionally been handled by human agents. To achieve this, speech recognition systems should allow a user to ask for and provide information using natural, conversational spoken input. Recent advances in certain areas of speech recognition technology have helped alleviate some of the traditional obstacles to usable speech recognition systems. For example, technology advances have enabled unrehearsed spoken input to be decoded under a wider range of realistic operating conditions, such as background noise and imperfect telephone line quality. Additionally, recent advances have allowed voice applications to recognize voice inputs from a broader population of users with different accents and speaking styles.
  • Well-engineered voice systems achieve high customer acceptance. Unfortunately, building effective voice systems using past approaches has been difficult.
  • The earliest approaches required programming in the application program interfaces (APIs) of the speech recognition engine. These approaches burdened developers with low-level, recognition engine specific details such as exception handling and resource management. Moreover, since these APIs were specific to a particular recognition engine, the resulting applications could not be easily ported to other platforms.
  • The advent of intermediate voice languages such as VoiceXML as open standards somewhat simplified the development process. These intermediate voice languages accompanied a distribution of responsibilities in a voice system between a browser—which interprets the voice language and handles the telephony, voice recognition, and text-to-speech infrastructure—and a client application—which provides the user interaction code (expressed in the voice language). As a result, application developers no longer needed to worry about low-level APIs, but instead were responsible for generating documents that would be executed by the voice browser.
  • Even with these advances, however, developing voice applications remained complex for a number of reasons. For example, voice applications present a new user interaction model that is sufficiently distinct from the (well understood) graphical user interface to require specialized design and implementation expertise. Speech interface concepts, such as dialog management, grammar optimization, and multi-slot interfaces, are manually implemented in every custom-built voice system. Given the relative newness of the speech paradigm, this further burdens the developers. In addition, the demands on applications to handle presentation, business logic, and data access functions resulted in piecemeal architectures combining static and dynamically generated documents, backend servlets, grammars, and other disjoint components.
  • A number of products are available to simplify the development of enterprise voice applications. A central element of many of these products is a library of predefined and customizable voice components whose use reduces the amount of code that needs to be developed by a programmer. These components usually encapsulate the voice language code, grammars, internal call flows, prompts and error recovery routines required to obtain one piece of information from the caller, such as a date, a time, a dollar amount, a sequence of digits, or an item from a set or list of allowable items (such as a set of airports).
  • A major limitation of these component frameworks is that the components are not combinable to allow the user to provide multiple pieces of information in each utterance. For example, a flight reservation application could use four components: a departure airport, a destination airport, a departure date and a departure time. The existing frameworks would allow a user to provide the four pieces of information in four separate utterances. However, if the application were to allow the user to say the departure airport, destination airport and departure date in one utterance (e.g. “I'm flying from Boston to San Francisco on Monday”), the departure airport, destination airport, and departure date components could not be simply combined. Instead, a new component would need to be developed with new grammars, call flows, prompts, etc. to recognize the two airports and the date. To carry the example further, if the application were to allow the caller to retain some pieces of information while changing others pieces of information (e.g. “No, I'm actually flying to Oakland on Tuesday”), an even more complex component would have to be developed.
  • Because of these limitations, voice applications that rely on existing component frameworks implement highly directed dialogs in which the call flow is largely predetermined and each step accepts only a single item of information, such as in an interchange illustrated in FIG. 1 a. Such voice systems are rigid and often penalize a caller who provides too much information, such as in an interchange illustrated in FIG. 1 b. As a result, these systems are neither intuitive nor efficient since they cannot capture information rapidly or adapt to the user's preferences for providing information.
  • What is needed is a voice application that utilizes a more intuitive, rapid and natural approach for obtaining information from a user such as a caller.
  • SUMMARY OF THE INVENTION
  • Systems and methods for constructing a series of interactions with a user to collect multiple pieces of related information for the purpose of accomplishing a specific goal or topic (a multi-slot dialog) using a component-based approach are disclosed. It should be appreciated that the present invention can be implemented in numerous ways, including as a process, an apparatus, a system, a device, a method, or a computer readable medium such as a computer readable storage medium or a computer network wherein program instructions are sent over optical or electronic communication lines. Several inventive embodiments of the present invention are described below.
  • The method generally includes outputting a primary header prompt to elicit values for slots in a segment from the user, receiving a primary user response containing a value for each slot in at least a subset of the slots in the segment, processing the primary user response to determine at least one possible recognition value for each slot contained in the primary user response, filling each slot contained in the primary user response with a matched value selected from the corresponding possible recognition values, and repeating the outputting, receiving, processing and filling for any unfilled slots in the segment until all slots in the segment of slots are filled.
  • The method may include performing turns to confirm and/or clarify the matched slot values such as by silently accepting a best match, passively confirming the best match, actively confirming the best match, disambiguating among the best matches, and notifying the user of non-recognition. The method for confirmation and/or clarification may be selected based on, for example, the number of possible recognition values for the slots in the primary user response and a corresponding confidence level for each of the possible recognition values. With an active confirmation, a user confirmation response is recognized as a confirmation, a cancellation, or a cancellation and correction. With a cancellation and correction, the correction is processed by determining at least one possible recognition value for each slot contained in the correction. With a passive confirmation, a passive confirmation prompt is output with a next primary header prompt. The method may also include enabling any unfilled slots in the segment of slots, in which the primary header prompt elicits values for enabled slots in the segment. The method may be text- or speech-based.
  • In another embodiment, a system for constructing and processing a multi-slot dialog with a user may generally include slot objects representing slots in a segment, each slot capable of being assigned a value based on the multi-slot dialog, at least one slot segment object each containing a corresponding set of slot objects, at least one set of slot group objects for each slot segment object, each slot group object defining parameters associated with a particular combination of slots in the slot segment object, and dialog objects that define a flow of the multi-slot dialog. The system may be implemented in an object-oriented programming language.
  • According to another embodiment, a method for constructing a multi-slot dialog with a user to obtain multiple items of information over a number of turns may generally include providing at least one slot segment object, each slot segment object containing a corresponding set of slot objects, each representing a slot in a segment, each slot representing an item of information to be provided by the user, providing at least one set of slot group objects for each slot segment object, each slot group object defining parameters associated with a particular combination of slots in the slot segment object, and executing a multi-slot dialog flow defined by dialog objects.
  • These and other features and advantages of the present invention will be presented in more detail in the following detailed description and the accompanying figures which illustrate by way of example principles of the invention.
  • BRIEF DESCRIPTION OF THE DRAWINGS
  • The present invention will be readily understood by the following detailed description in conjunction with the accompanying drawings, wherein like reference numerals designate like structural elements.
  • FIG. 1 a and FIG. 1 b illustrate examples of highly directed dialogs using conventional voice applications.
  • FIGS. 2 a-2 c illustrates various examples of multi-slot dialogs.
  • FIG. 3 is a block diagram of an illustrative multi-slot voice application system.
  • FIG. 4 illustrates one embodiment of a framework for managing a multi-slot speech recognition-based conversation.
  • FIG. 5 illustrates examples of segments for a multi-slot flight reconfirmation dialog.
  • FIG. 6 illustrates the contents of one of the exemplary segments, namely, the flight itinerary, in the multi-slot flight reconfirmation dialog of FIG. 5.
  • FIG. 7 illustrates examples of exchanges involved in a flight itinerary dialog.
  • FIG. 8 is a flowchart illustrating various steps of an exemplary multi-slot dialog.
  • FIG. 9 is a flowchart of an exemplary exchange using active confirmation.
  • FIG. 10 is a flowchart of an exemplary exchange using passive confirmation.
  • FIG. 11 is a flowchart illustrating an example of a conversation utilizing a go back functionality of a multi-slot dialog system.
  • FIG. 12 is a flowchart illustrating an example of a conversation utilizing a change functionality of the multi-slot dialog system.
  • FIG. 13 is a flowchart illustrating an example of a conversation utilizing a review functionality of the multi-slot dialog system.
  • FIGS. 14 and 15 illustrate some of the possible slot group objects for the pre-recognition and post-recognition slot group classes for a flight itinerary segment, respectively.
  • FIG. 16 is a block diagram illustrating an exemplary dialog flow structure.
  • FIGS. 17 and 18 are flowcharts illustrating exemplary processing of the user's input in normal and passive confirm modes, respectively.
  • FIG. 19 illustrates a possible grammar for an exchange with three slots.
  • FIG. 20 illustrates a possible grammar for a confirmation exchange that includes accept and cancel grammars as well as grammars for the three slots being confirmed.
  • DESCRIPTION OF SPECIFIC EMBODIMENTS
  • Systems and methods for constructing a series of interactions with a user to collect multiple pieces of related information for the purpose of accomplishing a specific goal or topic (a multi-slot dialog) using a component-based approach are disclosed. The multi-slot dialog systems and methods obtain information from a user by conducting a speech-recognition based series of interactions. The systems and methods include determining the prompts output to the user as well as the grammars and semantic rules utilized to recognize user inputs such as utterances at each point in the conversation or interchange. The following description is presented to enable any person skilled in the art to make and use the invention. Descriptions of specific embodiments and applications are provided only as examples and various modifications will be readily apparent to those skilled in the art. The general principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of the invention. Thus, the present invention is to be accorded the widest scope encompassing numerous alternatives, modifications and equivalents consistent with the principles and features disclosed herein. For purpose of clarity, details relating to technical material that is known in the technical fields related to the invention have not been described in detail so as not to unnecessarily obscure the present invention.
  • FIGS. 2 a-2 c illustrate various examples of multi-slot dialogs. A multi-slot dialog has the objective of collecting multiple pieces of related information (“slots”) for the purpose of accomplishing a specific goal or topic, such as locating an airline reservation. In order to achieve a successful user interaction with the convenience and ease expected by humans, a multi-slot dialog application preferably handles certain behavior and interactions typical of human interactions in a spoken medium, including: (i) a caller may provide the slots in an arbitrary order, (ii) a caller may provide multiple slots in a single input such as a spoken utterance, (iii) a caller may provide only a subset of slots requested by the application in a single utterance, (iv) a caller may clarify or correct the application's interpretation of slots the caller has provided, (v) a caller may modify earlier slots in subsequent utterances.
  • To satisfy these human interaction requirements, a dialog application may perform a lengthy and sophisticated call path of considerable complexity. However, conventional voice applications are ill-suited for implementing multi-slot dialogs. In particular, the dynamic order and combinations in which information may be provided cannot be easily handled by existing component frameworks and development methodologies of conventional voice applications that specify rigid, predetermined call flows.
  • Rather than the manual approach, the multi-slot dialog systems and methods as described herein may be utilized for constructing multi-slot dialog applications using a component-based approach. Such component-based approach automates the multi-slotting process with components for the behavioral elements of a multi-slotting exchange between human and machine, including sequencing, grouping, prompting, confirmation, and/or modification strategies.
  • FIG. 3 is a block diagram of an illustrative voice application system 300 that generally includes a telephone 301 in communication with a voice browser 303 via a telephone network 302 which is in turn in communication with a voice application 309 via a data network 308. The voice browser 303 includes the hardware and software for conducting bi-directional audio communications with a caller via the telephone network 302 and the telephone 301. The voice browser 303 may execute a program expressed in a voice language transmitted in the form of documents over the data network 308 (such as the Internet or an intranet) from the voice application 309. The voice browser 303 and voice application 309 may reside on any of various suitable computer systems.
  • The voice language may be a markup language such as VoiceXML or Speech Application Language Tags (SALT). The voice browser 303 may include various components such as a speech recognition engine 304, a text-to-speech synthesizer 305, an audio playback player 306 for audibly rendering files recorded using generally available audio formats, and a component for handling calls over the telephone network 307. Commercially available and/or proprietary components for building voice applications may be employed in implementing any or all of the various components of the voice browser 303.
  • The voice browser 303 may be responsible for detecting an incoming call, answering the incoming call, requesting an initial voice document from the voice application 309, and interpreting the voice document and executing instructions contained in the voice document according to the rules of the applicable voice language. The instructions may include the outputting of audible prompts to the user and the processing of voice responses from the user using specified grammars. In outputting the prompts to the user, the voice browser 303 may utilize the audio playback player 306 to audibly render prerecorded messages or may utilize the text-to-speech synthesizer 305 to audibly render text messages. The voice browser 303 can then generate actions in response to user events (such as spoken input or call disconnection) or system events (such as subsystem exceptions), and/or actions that may be defined in the current document or in another document to be fetched from the voice application 309.
  • The overall architecture of the voice application system 300 having been described, a multi-slot framework implemented by the voice application system 300, such as by a combination of the voice browser 303 and voice application 309, will now be presented. A system implementing the multi-slot mode of conversation generally prompts the user with more open-ended questions while a system implementing a conventional directed dialog mode of conversation prompts the user with more specific prompts and accepts a more limited set of responses. FIG. 4 illustrates one embodiment of a multi-slot framework 401 for managing a multi-slot speech recognition-based conversation. In particular, the multi-slot framework 401 implements a multi-slot mode for obtaining information from a user. The framework 401 may include a multi-slot application 402 that manages the business logic and data access responsibilities for a set of slots 403. A slot is referred to herein as a data item whose value can be obtained from a user input such as a text input or a spoken utterance. For example, a flight reservation application may manage a departure airport slot whose valid values are found in a flight schedule database, and whose value, once uttered or otherwise entered by the user, should be stored in a reservation record. A multi-slot platform 404 constructs one or more voice interactions to obtain the desired slots from the user.
  • FIG. 5 illustrates examples of segments for a multi-slot flight reconfirmation dialog and FIG. 6 illustrates the contents of one of the exemplary segments, namely, the flight itinerary, in the multi-slot flight reconfirmation dialog. In particular, a multi-slot dialog can be decomposed into sub-units at various levels. At the smallest level, a turn is an uninterrupted stream of input (e.g., speech) from one participant, i.e., the system or the user. Thus a system turn is one in which the system prompts the user with a message and a user turn is one in which the user makes an utterance that the system attempts to interpret. An exchange is a series of turns that captures the values for one or more slots from the user. Thus an exchange may include one or more confirmation, correction, or disambiguation turns until a final, single set of slot values is accepted by the system. A segment is a series of exchanges that captures a set of related slots. The segment is the largest unit of a dialog that allows all its associated slots to be provided in one user turn. In the example shown in FIG. 6, the five slots, namely, a departure airport, a destination airport, a date, a time, and an AM/PM indicator, form the flight itinerary segment such that up to all five slots of the flight itinerary segment may be entered in one user turn such as by the user uttering “from San Francisco to Boston tomorrow at 9AM.”
  • Multi-slot dialogs can vary in complexity from single segment dialogs to complex, multi-segment dialogs in which each segment includes one or more slots. An example of a single-segment dialog is a phone attendant application that requests an employee name from the caller. In contrast, an example of a multi-segment dialog is a flight reconfirmation dialog such as that shown in FIG. 5. For example, the flight reconfirmation dialog may request a flight itinerary from the caller (which may include slots for the departure and destination airports as well as the date, time and AM/PM indicator of the departure as shown in FIG. 6), and may verify the caller's identity by requesting a record locator and by requesting personal details, such as a surname.
  • FIG. 7 illustrates examples of exchanges involved in a flight itinerary dialog. The flight itinerary segment 701 contains 5 slots that represent a flight itinerary. In the first exchange 702, the system may prompt the user with “What is your flight itinerary?” to allow the user to enter information or values for up to all 5 slots. The user may respond with “I'm flying from Boston” and after any applicable confirmation/clarification turns, the application accepts Boston as the value for the departure airport slot. According to predefined logic, the application determines, for example, that only the destination airport should be prompted for in the next exchange 703, and outputs a prompt “And what is your destination?” After the user response is processed, the value San Francisco is stored in the destination airport slot. The application may then prompt for the remaining three unfilled slots in the next exchange 704 such as by outputting a prompt “And when are you flying?” The user's response of “next Friday at nine thirty AM” fills all three remaining slots and the dialog is complete.
  • FIG. 8 is a flowchart illustrating an exemplary multi-slot conversation 800. At the start of an exchange at block 802, the voice application system outputs a primary header prompt to elicit values for a set of slots from the user. The system prompt is typically worded in such a way that the user may potentially provide values for a number of desired slots in one turn. At block 804, the user responds with a primary user turn in response to the output prompt. The user turn may provide one or more slot values. If the user turn is recognized by the system, a number of confirmation/clarification turns at block 806 may follow in which the system may optionally silently accept the best match, passively confirm the best match, demand active confirmation of the best match, disambiguate among the best matches, or notify the user of a non-recognition, for example, as will be described in more detail below. When the slot values from the turn are confirmed either explicitly or implicitly, the slots are filled with their respective values at block 808.
  • Blocks 802-808 represent one exchange. When the current exchange is complete, the system determines if there are any remaining unfilled slots in the current segment at decision block 810. If all slots in the current segment are filled, then the system determines if there are any additional segment(s) of slots at decision block 812. If all slots in all segments are filled, then the multi-slot dialog 800 is complete. Alternatively, if there are remaining unfilled slots in the current segment and/or if there are additional segment(s) of slots, the dialog 800 returns to block 802 to initiated the next exchange. For example, the next exchange may either follow up on the previous exchange by prompting for any remaining slots that were not filled (e.g., “ . . . and around what time?”) or continue onto the next segment of slots by prompting for a new set of slots (e.g., “What type of car do you want to rent?”).
  • As noted above, if the user turn is recognized, a number of confirmation and/or clarification turns may follow in which the system may optionally demand active confirmation of the best match, passively confirm the best match, disambiguate among the best matches, silently accept the best match, or notify the user of a non-recognition, for example. Details of the confirmation/clarification turns will now be described in more detail with reference to FIGS. 9 and 10.
  • In particular, FIG. 9 is a flowchart of an exemplary exchange using active confirmation. In an active confirmation, a confirmation prompt is output to the user (e.g., “I think you said Austin to San Francisco. Is that correct?”). The confirmation prompt may escalate during an exchange if there are several confirmation cycles. The system may require that the user explicitly accept the value(s) for the associated slots prior to filling the slots. The user can cancel the previous recognized values by saying a cancel phrase such as “No” or “No, that's wrong.” In addition, to facilitate a more efficient interaction, the system may optionally accept an utterance by the user that includes a cancel phrase followed by a correction such as “No, I said Boston to San Francisco.” If the user cancels the previous recognized values, the system may clear all slot values recognized in the user turn and play a message such as “Let's try that again. What is your flight itinerary?” as shown in FIG. 9 and begin the exchange again. Alternatively, the system may treat the user response uttered in the user turn as a correction and repeat the confirmation/clarification turn, e.g., by prompting “OK. Boston to San Diego. Is that correct?” If the correction omits some of the slots that are being confirmed, the system may retain the previously uttered values of such slots.
  • FIG. 10 is a flowchart of an exemplary exchange using passive confirmation. In a passive confirmation, the multi-slot dialog system outputs a prompt that is a combination of a passive confirmation prompt and a header prompt for the next exchange. As an example, the combined output prompt may be, for example, “Ok. Austin to San Francisco. And when are you flying?” As the user does not need to explicitly utter a confirmation if the values are correct, the passive confirmation technique facilitates in speeding up the dialog and thus is suitable for confirming recognition results where the system has a high level of confidence. When passive confirmation is utilized, the slots are filled with their respective values and those filled values may be removed or rolled back when the user utters a cancel phrase such as “No” or “No, that's wrong,” or a cancel phrase followed by a correction such as “That's wrong. I said Boston to San Francisco.” If the user issues a cancel via a cancel phrase, the system may clear the slot values accepted in the previous exchange, issue a prompt such as “Sorry, let's try that again,” and repeat the previous exchange. Alternatively, the system may treat the user response uttered in the user turn as a correction and repeat the confirmation/clarification turn, e.g., by prompting “OK. Boston to San Francisco. Is that correct?” If the user then utters an accept phrase such as “Yes” in response, the header prompt for the new exchange is repeated. As is evident, an active confirmation may be utilized after the user issues a cancel phrase in response to a passive confirmation. However, other confirmation types may be similarly utilized after the user issues a cancel phrase in response to a passive confirmation. If the next exchange contains a prompt for a “Yes/No” response, the functionality to rollback a previous passive confirm of the passive confirmation may be disabled.
  • In a disambiguation, the system outputs a prompt containing a list of the top matches and requests the user to select one of the top matches, e.g., by its corresponding number. Each match may include a single slot value or a set of slot values and may be presented by a prompt similar to a confirmation prompt that contains all the recognized slot values. When a match is selected, the corresponding value or values are treated as if they had been uttered in the primary user turn and the system repeats the confirmation/clarification process.
  • The system may also notify the user of a non-recognition. In particular, when a user turn is not recognized by the system, the system may issue an exception prompt such as “I didn't get that” and repeat the previous system turn or an amended version thereof.
  • The system may be configured with various features and functionalities to further facilitate the exchange as will be described below in more detail with reference to FIGS. 11-13. For example, the system may be configured to maintain a skip list in a given exchange in which a skip list entry corresponding to the set of slot values presented in a confirmation is added each time a user cancels the confirmation during that exchange. The skip list helps to ensure that, within the same exchange, the system does not utilize and thus will not present again a set of values that matches an entry in skip list. Instead, the system may utilize the next best match when available.
  • FIG. 11 is a flowchart illustrating an example of a conversation that includes yet another optional functionality of the system, namely, a go back functionality. Specifically, the user may utter a go back command, e.g., “go back,” at any time so as to return to the start of the previous turn, the start of the current exchange, or the start of the current segment, depending on the application. If the system goes back over a step that has filled some slots, these slots may be cleared.
  • The multi-slot dialog system may be configured to adaptively present a more directed prompt after a threshold number of exceptions, corrections, or gobacks has occurred during an exchange. For example, the system may present a new, more directed header prompt “What date will you be picking up the car? For example, say tomorrow, next Tuesday, or July 24th.” If the exceptions, corrections or gobacks continue to occur, the system may transfer the user to a live operator for further assistance.
  • FIG. 12 is a flowchart illustrating an example of a conversation that includes yet another optional functionality of the system, namely, a change functionality. The change functionality allows the user to change a previously filled slot value by uttering, for example, “I'd like to change the airport.” If the user's utterance requesting a change does not fill all the slots required for a change request, the system initiates a follow-up exchange such as “Departure or destination airport?” The change command may optionally be confirmed using any of the confirmation mechanisms described above. For example, the system may actively confirm the change command by prompting “I heard you say you wanted to change the arrival airport. Is that correct?”
  • The change command cancels the exchange the user is currently in and clears at least some of the previously filled slot values. A new exchange then begins that prompts the user for the cleared slots which can be filled in one or more exchanges. Once the cleared slots are filled, the system continues processing and will bypass some previous exchanges if the slots associated with those exchanges are still filled.
  • FIG. 13 is a flowchart illustrating an example of a conversation that includes yet another optional functionality of the system, namely, a review functionality. In particular, the user may request to review a previously filled slot value by uttering a request phrase such as “Can I check my date of departure?” In response to the review command, the system plays the filled slot value such as “You're flying on Friday, Aug. 9, 2002” and returns to the current exchange. If the user does not provide the values for all the slots required for a review request, the system initiates a follow-up exchange such as “Departure or return date?”
  • Implementation of Multi-Slot Dialog
  • An exemplary system or platform for implementing multi-slot dialogs will now be described. Merely by way of example, the platform for implementing multi-slot dialogs is described herein as being implemented utilizing Java. However, it is to be understood that the system may be implemented using any suitable programming language, preferably an object-oriented programming language such as Java or C++. The system generally includes slot objects, slot group objects, segment objects, and dialog objects. Each of these objects is described below.
  • A slot object represents a slot which, as noted above, is an item of information to be obtained from the user. A slot object contains the grammar that recognizes the possible values that can be provided by the user for the corresponding slot, the rules that map grammar recognition results to semantic values for the slot, and the variables indicating the enabled and filled state (among other state variables) for the slot. The slot objects can be based on a Java interface that provides basic default functionality and/or functionality common to all slot objects.
  • The grammar that recognizes the possible values that can be provided by the user for the corresponding slot is a formal specification of the utterances the system accepts for expressing the possible values for the slot. The grammar may include the vocabulary of words that may be used and valid structures for sequencing those words. For example, the grammar for a date slot should allow various date formats to be recognized, ranging from absolute specifications such as “January the twelfth two thousand and four” to relative forms such as “this coming Friday” and familiar terms such as “today” and “yesterday.” The grammar may also include fillers that may precede and/or follow the slot value in a user's utterance but that do not specify or distinguish one slot value from another. For example, an airport slot may have the preceding filler phrase “I'm flying from.” Some grammars may be highly application specific such as the grammar for the allowable airports in a flight reservation application. Other grammars may be reused across applications, such as the grammar for a date, a time, or a dollar amount. The common portion of these grammars may be predefined in a grammar object and customized for a particular slot.
  • In addition to the rules of the grammar, each slot object also contains rules that map the grammar recognition results to semantic values for the corresponding slot that are meaningful to the specific application. For example, a destination airport slot object may map the utterances “San Francisco,” “San Francisco Airport,” “San Francisco International Airport,” and “SFO” to a single airport identifier such as “SFO.” As another example, a date slot object may map the utterance “tomorrow” to a date value that is computed as the next date following the current date.
  • Each slot object also maintains a number of state variables or flags used during the execution of a multi-slot dialog to indicate the enabled and filled state (among other state variables) for the corresponding slot. Examples of flags include enabled, optional, filled, and pending flags. In particular, an enabled flag is set to true to indicate that a slot can be accepted in the upcoming user turn. An optional flag is set to true if an enabled slot does not need to be explicitly provided by the user. A filled flag is set to true when a slot value has been accepted after any applicable confirmation/clarification. A pending flag is set to true if a value for the slot has been recognized from the user but has not yet been accepted, e.g., pending confirmation.
  • The system maintains slot group objects for each slot segment, examples of which are shown in FIGS. 14 and 15 for a flight itinerary segment. Each slot group object defines parameters or properties associated with a particular group or combination of slots in the slot segment. Examples of slot group properties include prompts such as header or main prompts, help prompts, error prompts, confirmation prompts, and/or disambiguation prompts, as well as recognition properties, i.e., variables that affect recognition behavior such as timeouts, recognition thresholds, recognition parameters, caching policies and so on. Different slot group classes, e.g., pre-recognition and post-recognition slot group classes as shown in FIGS. 14 and 15, respectively, may be defined to maintain separate sets of properties.
  • Specifically, FIG. 14 illustrates some of the possible slot group objects for the pre-recognition slot group class for the flight itinerary segment. The pre-recognition slot group class may contain the prompts and recognition properties used before slot values are recognized, such as the header or main prompts, the help prompts, no match prompts, no input prompts, timeouts, confidence thresholds and so on. When the pre-recognition slot group class is used, the slot combination would typically be compared to the currently enabled set of slots.
  • FIG. 15 illustrates some of the possible slot group objects for the post-recognition slot group class for the flight itinerary segment. The post-recognition slot group class may contain the prompts and recognition properties used after slot values are recognized, such as the active confirmation prompts, disambiguation prompts, passive confirmation prompts, and so on. When the post-recognition slot group class is used, the slot combination would typically be compared to the currently pending set of slots, i.e., the slots recognized from the user but not yet confirmed.
  • When a parameter such as a prompt or a recognition property is required at a point in a multi-slot dialog, the system identifies a best match slot group object from a slot group class that contains the parameter and looks up the value of the desired parameter in the identified best match slot group object. For example, where a header prompt is required at a point in a multi-slot dialog in order for the system to elicit from the user the values for the slots in the currently enabled slot combination, e.g., date, time, and AM-PM, the system identifies a best match slot group object from the pre-recognition slot group class in FIG. 14. The system selects the slot group object whose slot combination is the closest to the currently enabled slot combination. The closest slot combination may be determined utilizing various suitable methods such as the most number of overlapping slots, the fewest number of non-overlapping slots in the slot group object, or the fewest number of non-overlapping slots in the enabled group. In the current example, the system identifies and utilizes the slot group object in the pre-recognition slot group class shown in FIG. 14 having an exact match slot combination as the currently enabled slot combination, i.e., date, time, and AM-PM. However, if an exact match slot group object is not found, another group object deemed as the closest may be identified and utilized, e.g., a group object with a two-slot combination such as time and AM-PM slots. The system may define a separate slot group object for each slot of the slot segment to ensure that a slot group can always be found for any enabled slot combination.
  • The system also includes segment objects. A segment object maintains a set of slots in a slot segment that determines a series of one or more successive multi-slot exchanges. The values for the slots in a given slot segment may potentially be obtained from the user in a single exchange. Alternatively, if the user does not provide the values for all the slots in that segment in one exchange, follow-up exchanges are initiated until all required, i.e., non-optional, slots are filled. When a segment is filled completely, the next segment, if any, is then invoked.
  • The system further includes dialog objects that define the dialog flow. While each multi-slot dialog can perform a different function, the dialog flow for each dialog generally has a common structure. FIG. 16 is a block diagram illustrating an exemplary dialog flow structure. After the system initiates a dialog at block 1601, the system obtains the first segment in the dialog at block 1602. The system determines the slots in this segment that should be enabled at block 1603 such as by including the slots that are not yet filled. The closest pre-recognition slot group is selected to retrieve the header prompt and other pre-recognition parameters at block 1604 (such as the help prompt and any exception prompts). The header prompt is usually worded in such a way that the user may potentially provide a number of desired slots in one utterance. For example, if the slots include a date slot, a time slot, and a meridian (AM/PM) slot, the prompt could be “When will you be picking up the car?” The user's response is then received and processed by the system at block 1605 such as by obtaining the best hypotheses of the user's intended word sequence from the speech recognition engine and performing any desired confirmation, clarification or disambiguation based on the custom settings of the application until a single set of slot values is accepted by the system.
  • The system then determines whether the slot segment contains any more slots to be filled at decision block 1606 which usually includes slots whose values have not yet been filled. It is noted that the system, in determining whether the slot segment contains any more slots to be filled at decision block 1606, may apply application-specific logic specifying that some slot values may or should be automatically filled from other slot values, that some slots are optional, or that certain additional slots need to be filled as a result of the values of other accepted slots. If some slots do remain to be filled as determined in decision block 1606, the system returns to block 1603 to enable the next set of slots. Otherwise, the system continues to decision block 1607 to determine if there are any more slot segments. If there is at least one more slot segment remaining, the system obtains the next segment at block 1608 and returns to block 1603. Alternatively, if no other slot segments remain, the dialog ends at block 1610.
  • When processing the user's input, the system may be in a normal mode or passive confirm mode. Passive confirm mode is activated when the system determines that the user's response should be passively confirmed based on some predefined criteria. One such set of criteria could be that the confidence level returned by the speech engine is below the threshold for silent acceptance but above the threshold requiring active confirmation. If passive confirm mode is not activated, the system may be in normal mode by default. FIG. 17 is a flowchart illustrating an exemplary processing of the user's input in normal mode while FIG. 18 is a flowchart illustrating an exemplary processing the user's input in passive confirm mode. The determination as to normal versus passive mode may be based on, for example, a weighted or average confidence level for the set of slots to which the user's response corresponds. When dealing with a user response with values for multiple slots, the determination may be made with a single determination for all slots in the user response or a separate determination for each slot in the user response and the separate results averaged or weighted, for example.
  • In normal mode as illustrated in FIG. 17, the speech engine recognizes the user's input and returns one or more hypotheses at block 1701. If several possible matches are returned by the speech engine as determined at decision block 1702, the possible matches may be disambiguated at block 1708. For example, the user may be presented with a list of top matches and asked to select one. Each match may be presented by a disambiguation prompt similar to a confirmation prompt. When a match selected by the user is received and processed at block 1709, the system proceeds to the next exchange at block 1710.
  • Alternatively, if only one hypothesis, i.e., the best match, is returned by the speech engine as determined at decision block 1702, the system determines whether the confidence level for the one hypothesis is at or above a predefined high confidence threshold at decision block 1703. If the high confidence level is at or above the high confidence threshold, the system accepts the slot values and enters passive confirmation mode at block 1707. Alternatively, if the high confidence level is below the high confidence threshold, the system actively confirms the best match by outputting a confirmation prompt at block 1704. For example, the system may construct the confirmation prompt by identifying the slot group in the post-recognition slot group class that is closest to the group of slots in the best match and retrieving the corresponding active confirmation prompt. The user's response to the confirmation prompt is received and processed at block 1705. If the user cancels the confirmation, e.g., by uttering a cancel phrase such as “no” or “that's wrong,” and provides a correction, the corrected values may return to block 1704 for confirmation. If the user cancels the confirmation with no correction, the current exchange is repeated at block 1706. In other words, any pending slot values are discarded or cleared and the system repeats the header prompt that was originally played. If the user accepts the confirmation, e.g., by uttering a phrase such as “yes” or “that's right,” the system accepts the slots values and proceeds to the next exchange at block 1710.
  • In passive confirm mode as illustrated in FIG. 18, the confirmation prompt from one exchange is combined with the header prompt for the next exchange. For example, the combined prompt may be “Ok. Boston. And what date will you be flying?” The user's response is then recognized by the speech engine at block 1801. If the user's response does not begin with an accept or cancel phrase, i.e., a yes or no lead, as determined at decision block 1802, the user's response is processed as in the case of the normal mode described above. Alternatively, if the user response begins with an accept or cancel phrase as determined at decision block 1802, the previous slot values may be affected. If the user response is an affirmative user response, the header prompt for the new exchange may simply be repeated, for example “And what date will you be flying?” at block 1803. If the user response is a negative user response without a correction, the system may clear the slot values accepted in the previous exchange, play a message such as “Sorry, let's try that again,” and repeat the previous exchange at block 1804. For a negative user response with a correction, the corrected values are explicitly confirmed at block 1805 until the previous exchange's slots are explicitly accepted or cancelled.
  • During a primary user turn, the system allows the user to provide values for one or more enabled slots. The grammar for the turn should thus be flexibly configured to recognize various flexible combinations of slots. For example, FIG. 19 illustrates a possible grammar for an exchange with three slots. Each branch of the diagram represents a grammar rule that could match the user's utterance. The grammars for slots 1, 2, and 3 are represented by reference numbers 1901, 1902, 1903, respectively. The postfix operator “?” applied to each of the grammars 1902, 1903 corresponding to slots 2 and 3 in the first branch indicates that slots 2 and 3 are optional such that in this first branch the user's utterance contains a value for slot 1, optionally a value for slot 2, and, if there is a value for slot 2, optionally a value for slot 3. Note that an application may restrict the allowed combination of slots (such as a slot having to come before another) depending on the norms of the language used and the context of a particular exchange.
  • During a confirmation turn, the system allows the user not only to accept or cancel the confirmation, but also to provide corrected values. The grammar for a confirmation turn should thus be constructed to include the accept and cancel grammars as well as grammars for the slots being confirmed, an example of which is shown in FIG. 20. The accept grammar 2001 contains a set of phrases that express an affirmative response, such as “yes,” “that's right,” and “correct.” The cancel grammar 2002 contains a set of phrases that express a negative response, such as “no,” “that's wrong” and “incorrect.” If the cancel grammar is present, an optional correction grammar 2003 is included which recognizes new values for the slots being cancelled.
  • While the exemplary embodiments of the present invention are described and illustrated herein, it will be appreciated that they are merely illustrative and that modifications can be made to these embodiments without departing from the spirit and scope of the invention. For example, although the multi-slot systems and methods described herein are well suited for voice interactions using speech recognition systems, the multi-slot systems and methods may also be adapted for use with text-based multi-slot interactions such as an interactive Internet-based multi-slot dialog. Thus, the scope of the invention is intended to be defined only in terms of the following claims as may be amended, with each claim being expressly incorporated into this Description of Specific Embodiments as an embodiment of the invention.

Claims (38)

1. A method for constructing and processing a multi-slot dialog with a user, comprising:
outputting a primary header prompt to elicit values for slots in a segment from the user;
receiving a primary user response, the primary user response containing a value for each slot in at least a subset of the slots in the segment;
processing the primary user response to determine at least one possible recognition value for each slot contained in the primary user response;
filling each slot contained in the primary user response with a matched value selected from the corresponding at least one possible recognition value; and
repeating the outputting, receiving, processing and filling for any unfilled slots in the segment until all slots in the segment of slots are filled.
2. The method of claim 1, further comprising:
performing turns to at least one of confirm and clarify the matched slot values for slots contained in the primary user response.
3. The method of claim 2, wherein the at least one of confirm and clarify is selected from the group consisting of silently accept a best match, passively confirm the best match, actively confirm the best match, disambiguate among the best matches, and notify the user of a non-recognition.
4. The method of claim 3, wherein the at least one of confirm and clarify is selected based on the number of possible recognition values for the slots in the primary user response and a corresponding confidence level for each of the possible recognition values.
5. The method of claim 3, wherein when the at least one of confirm and clarify is an active confirmation, the performing turns includes recognizing a user confirmation response as one of a confirmation, a cancellation, and a cancellation and correction, and wherein when the user confirmation response is a cancellation and correction, the correction is processed by processing the correction to determine at least one possible recognition value for each slot contained in the correction.
6. The method of claim 3, wherein when the at least one of confirm and clarify is a passive confirmation, the performing turns includes recognizing a user passive confirmation response as one of a response to a next primary header prompt, a confirmation, a cancellation, and a cancellation and correction, and wherein when the user passive confirmation response is a cancellation and correction, the correction is processed by processing the correction to determine at least one possible recognition value for each slot contained in the correction.
7. The method of claim 1, further comprising:
enabling any unfilled slots in the segment of slots, wherein the primary header prompt elicits values for enabled slots in the segment.
8. The method of claim 1, wherein the processing of the primary user response includes applying grammar rules to facilitate recognition of possible values for a corresponding slot.
9. The method of claim 1, wherein the outputting the primary header prompt depends on the set of unfilled segments.
10. The method of claim 1, wherein the outputting and the receiving is one of text-based and speech-based.
11. A system for constructing and processing a multi-slot dialog with a user, comprising:
a plurality of slot objects each representing a slot in a segment, each slot capable of being assigned a value based on the multi-slot dialog;
at least one slot segment object, each slot segment object containing a corresponding set of slot objects;
at least one set of slot group objects for each slot segment object, each slot group object defining parameters associated with a particular combination of slots in the slot segment object; and
dialog objects that define a flow of the multi-slot dialog.
12. The system of claim 11, wherein the system is implemented in an object-oriented programming language.
13. The system of claim 11, wherein each parameter for at least one set of slot group objects is classified into one of at least two slot group classes defined to facilitate maintenance of separate sets of the parameters.
14. The system of claim 13, wherein the slot group classes include a pre-recognition slot group class and a post-recognition slot group class.
15. The system of claim 11, wherein each slot object contains grammar to facilitate recognizing a possible value provided by the user for the corresponding slot, rules that facilitate mapping of grammar recognition results to semantic values for the corresponding slot, and variables indicating a state for the corresponding slot.
16. The system of claim 11, wherein the parameters defined by each slot group object is selected from the group consisting of header prompts, help prompts, error prompts, confirmation prompts, disambiguation prompts, and recognition properties.
17. The system of claim 11, wherein the system is one of text-based and speech-based.
18. A method for constructing a multi-slot dialog with a user to obtain multiple items of information over a number of turns, comprising:
providing at least one slot segment object, each slot segment object containing a corresponding set of slot objects, each representing a slot in a segment, each slot representing an item of information to be provided by the user;
providing at least one set of slot group objects for each slot segment object, each slot group object defining parameters associated with a particular combination of slots in the slot segment object; and
executing a multi-slot dialog flow defined by dialog objects.
19. The method of claim 18, wherein each slot object contains grammar to facilitate recognizing a possible value provided by the user for the corresponding slot, rules that map grammar recognition results to semantic values for the corresponding slot, and variables indicating a state for the corresponding slot.
20. The method of claim 18, wherein each parameter for at least one set of slot group objects is classified into one of at least two slot group classes defined to facilitate maintenance of separate sets of the parameters.
21. The method of claim 20, wherein the slot group classes include a pre-recognition slot group class and a post-recognition slot group class.
22. The method of claim 18, wherein the parameters defined by each slot group object is selected from the group consisting of header prompts, help prompts, error prompts, confirmation prompts, disambiguation prompts, and recognition properties.
23. The method of claim 18, wherein the system is one of text-based and speech-based.
24. The method of claim 18, wherein executing includes performing an action in response to a user input, the action being selected from the group consisting of silently accepting a best match, passively confirming the best match, actively confirming the best match, disambiguating among the best matches, and notifying the user of a non-recognition.
25. The method of claim 24, wherein when the action is an active confirmation, the executing further includes recognizing a user confirmation response as one of a confirmation, a cancellation, and a cancellation and correction, and when the user confirmation response is a cancellation and correction, the correction is processed by processing the correction to determine at least one possible recognition value for each slot contained in the correction.
26. The method of claim 24, wherein when the action is a passive confirmation, the executing further includes recognizing a user passive confirmation response as one of a response to a next primary header prompt, a confirmation, a cancellation, and a cancellation and correction, and when the user passive confirmation response is a cancellation and correction, the correction is processed by processing the correction to determine at least one possible recognition value for each slot contained in the correction.
27. The method of claim 24, wherein the action is selected based on a number of possible values for the slots and a corresponding confidence level for each possible value.
28. The method of claim 18, wherein the dialog objects define turns to at least one of confirm and clarify slot values.
29. A computer program product embodied on a computer readable medium, the computer program product including instructions that, when executed by a processor, cause the processor to:
output a primary header prompt to elicit values for slots in a segment from the user;
receive a primary user response, the primary user response containing a value for each slot in at least a subset of the slots in the segment;
process the primary user response to determine at least one possible recognition value for each slot contained in the primary user response;
fill each slot contained in the primary user response with a matched value selected from the corresponding at least one possible recognition value; and
repeat the outputting, receiving, processing and filling for any unfilled slots in the segment until all slots in the segment of slots are filled.
30. The computer program product of claim 29, further including instructions that, when executed by the processor, cause the processor to:
perform turns to at least one of confirm and clarify the matched slot values for slots contained in the primary user response.
31. The computer program product of claim 30, wherein the at least one of confirm and clarify is selected from the group consisting of silently accept a best match, passively confirm the best match, actively confirm the best match, disambiguate among the best matches, and notify the user of a non-recognition.
32. The computer program product of claim 31, wherein the at least one of confirm and clarify is selected based on the number of possible recognition values for the slots in the primary user response and a corresponding confidence level for each of the possible recognition values.
33. The computer program product of claim 31, wherein when the at least one of confirm and clarify is an active confirmation, the performing turns includes recognizing a user confirmation response as one of a confirmation, a cancellation, and a cancellation and correction, and wherein when the user confirmation response is a cancellation and correction, the correction is processed by processing the correction to determine at least one possible recognition value for each slot contained in the correction.
34. The computer program product of claim 31, wherein when the at least one of confirm and clarify is a passive confirmation, the performing turns includes recognizing a user passive confirmation response as one of a response to a next primary header prompt, a confirmation, a cancellation, and a cancellation and correction, and wherein when the user passive confirmation response is a cancellation and correction, the correction is processed by processing the correction to determine at least one possible recognition value for each slot contained in the correction.
35. The computer program product of claim 29, further including instructions that, when executed by the processor, cause the processor to:
enable any unfilled slots in the segment of slots, wherein the primary header prompt elicits values for enabled slots in the segment.
36. The computer program product of claim 29, wherein the processing of the primary user response includes applying grammar rules to facilitate recognition of possible values for a corresponding slot.
37. The computer program product of claim 29, wherein the outputting the primary header prompt depends on the set of unfilled segments.
38. The computer program product of claim 29, wherein the outputting and the receiving is one of text-based and speech-based.
US10/885,934 2004-07-06 2004-07-06 Multi-slot dialog systems and methods Active 2024-10-28 US7228278B2 (en)

Priority Applications (8)

Application Number Priority Date Filing Date Title
US10/885,934 US7228278B2 (en) 2004-07-06 2004-07-06 Multi-slot dialog systems and methods
ES05768313.8T ES2526698T3 (en) 2004-07-06 2005-07-06 Procedure, system and software to collect multiple pieces of information during a user dialogue
EP10190283.1A EP2282308B1 (en) 2004-07-06 2005-07-06 Multi-slot dialog system and method
EP05768313.8A EP1779376B1 (en) 2004-07-06 2005-07-06 Method, system and computer program for collecting multiple pieces of information during a user dialog
JP2007520443A JP2008506156A (en) 2004-07-06 2005-07-06 Multi-slot interaction system and method
PCT/US2005/023844 WO2006014432A2 (en) 2004-07-06 2005-07-06 Multi-slot dialog systems and methods
ES10190283.1T ES2621808T3 (en) 2004-07-06 2005-07-06 Multi-interval dialogue system and method
US11/787,763 US7747438B2 (en) 2004-07-06 2007-04-17 Multi-slot dialog systems and methods

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
US10/885,934 US7228278B2 (en) 2004-07-06 2004-07-06 Multi-slot dialog systems and methods

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US11/787,763 Division US7747438B2 (en) 2004-07-06 2007-04-17 Multi-slot dialog systems and methods

Publications (2)

Publication Number Publication Date
US20060009973A1 true US20060009973A1 (en) 2006-01-12
US7228278B2 US7228278B2 (en) 2007-06-05

Family

ID=35542461

Family Applications (2)

Application Number Title Priority Date Filing Date
US10/885,934 Active 2024-10-28 US7228278B2 (en) 2004-07-06 2004-07-06 Multi-slot dialog systems and methods
US11/787,763 Active 2025-12-30 US7747438B2 (en) 2004-07-06 2007-04-17 Multi-slot dialog systems and methods

Family Applications After (1)

Application Number Title Priority Date Filing Date
US11/787,763 Active 2025-12-30 US7747438B2 (en) 2004-07-06 2007-04-17 Multi-slot dialog systems and methods

Country Status (5)

Country Link
US (2) US7228278B2 (en)
EP (2) EP1779376B1 (en)
JP (1) JP2008506156A (en)
ES (2) ES2621808T3 (en)
WO (1) WO2006014432A2 (en)

Cited By (146)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20060149553A1 (en) * 2005-01-05 2006-07-06 At&T Corp. System and method for using a library to interactively design natural language spoken dialog systems
US20060217978A1 (en) * 2005-03-28 2006-09-28 David Mitby System and method for handling information in a voice recognition automated conversation
US20060215824A1 (en) * 2005-03-28 2006-09-28 David Mitby System and method for handling a voice prompted conversation
US20060247931A1 (en) * 2005-04-29 2006-11-02 International Business Machines Corporation Method and apparatus for multiple value confirmation and correction in spoken dialog systems
US20070099602A1 (en) * 2005-10-28 2007-05-03 Microsoft Corporation Multi-modal device capable of automated actions
US20070219974A1 (en) * 2006-03-17 2007-09-20 Microsoft Corporation Using generic predictive models for slot values in language modeling
US20070239454A1 (en) * 2006-04-06 2007-10-11 Microsoft Corporation Personalizing a context-free grammar using a dictation language model
US20070239637A1 (en) * 2006-03-17 2007-10-11 Microsoft Corporation Using predictive user models for language modeling on a personal device
US20080091426A1 (en) * 2006-10-12 2008-04-17 Rod Rempel Adaptive context for automatic speech recognition systems
US20080114594A1 (en) * 2006-11-14 2008-05-15 Microsoft Corporation Conversation data accuracy confirmation
US20090187410A1 (en) * 2008-01-22 2009-07-23 At&T Labs, Inc. System and method of providing speech processing in user interface
US20100057463A1 (en) * 2008-08-27 2010-03-04 Robert Bosch Gmbh System and Method for Generating Natural Language Phrases From User Utterances in Dialog Systems
US20100070360A1 (en) * 2008-09-13 2010-03-18 At&T Intellectual Property I, L.P. System and method for creating a speech search platform for coupons
US20100070277A1 (en) * 2007-02-28 2010-03-18 Nec Corporation Voice recognition device, voice recognition method, and voice recognition program
US20100217603A1 (en) * 2009-02-26 2010-08-26 Hammond Daniel D Method, System, and Apparatus for Enabling Adaptive Natural Language Processing
US20120089392A1 (en) * 2010-10-07 2012-04-12 Microsoft Corporation Speech recognition user interface
US20130110518A1 (en) * 2010-01-18 2013-05-02 Apple Inc. Active Input Elicitation by Intelligent Automated Assistant
US8694324B2 (en) 2005-01-05 2014-04-08 At&T Intellectual Property Ii, L.P. System and method of providing an automated data-collection in spoken dialog systems
US20150019228A1 (en) * 2013-07-15 2015-01-15 International Business Machines Corporation Automated confirmation and disambiguation modules in voice applications
US8977255B2 (en) 2007-04-03 2015-03-10 Apple Inc. Method and system for operating a multi-function portable electronic device using voice-activation
US9190062B2 (en) 2010-02-25 2015-11-17 Apple Inc. User profiling for voice input processing
US9240197B2 (en) 2005-01-05 2016-01-19 At&T Intellectual Property Ii, L.P. Library of existing spoken dialog data for use in generating new natural language spoken dialog systems
US9262612B2 (en) 2011-03-21 2016-02-16 Apple Inc. Device access using voice authentication
US9300784B2 (en) 2013-06-13 2016-03-29 Apple Inc. System and method for emergency calls initiated by voice command
US9330720B2 (en) 2008-01-03 2016-05-03 Apple Inc. Methods and apparatus for altering audio output signals
US9338493B2 (en) 2014-06-30 2016-05-10 Apple Inc. Intelligent automated assistant for TV user interactions
US9368114B2 (en) 2013-03-14 2016-06-14 Apple Inc. Context-sensitive handling of interruptions
US9430463B2 (en) 2014-05-30 2016-08-30 Apple Inc. Exemplar-based natural language processing
US9431006B2 (en) 2009-07-02 2016-08-30 Apple Inc. Methods and apparatuses for automatic speech recognition
US9483461B2 (en) 2012-03-06 2016-11-01 Apple Inc. Handling speech synthesis of content for multiple languages
US9495129B2 (en) 2012-06-29 2016-11-15 Apple Inc. Device, method, and user interface for voice-activated navigation and browsing of a document
US9502031B2 (en) 2014-05-27 2016-11-22 Apple Inc. Method for supporting dynamic grammars in WFST-based ASR
US9535906B2 (en) 2008-07-31 2017-01-03 Apple Inc. Mobile device having human language translation capability with positional feedback
US9576574B2 (en) 2012-09-10 2017-02-21 Apple Inc. Context-sensitive handling of interruptions by intelligent digital assistant
US9582608B2 (en) 2013-06-07 2017-02-28 Apple Inc. Unified ranking with entropy-weighted information for phrase-based semantic auto-completion
US9620105B2 (en) 2014-05-15 2017-04-11 Apple Inc. Analyzing audio input for efficient speech and music recognition
US9620104B2 (en) 2013-06-07 2017-04-11 Apple Inc. System and method for user-specified pronunciation of words for speech synthesis and recognition
US9626955B2 (en) 2008-04-05 2017-04-18 Apple Inc. Intelligent text-to-speech conversion
US9633004B2 (en) 2014-05-30 2017-04-25 Apple Inc. Better resolution when referencing to concepts
US9633674B2 (en) 2013-06-07 2017-04-25 Apple Inc. System and method for detecting errors in interactions with a voice-based digital assistant
US9646614B2 (en) 2000-03-16 2017-05-09 Apple Inc. Fast, language-independent method for user authentication by voice
US9646609B2 (en) 2014-09-30 2017-05-09 Apple Inc. Caching apparatus for serving phonetic pronunciations
US9668121B2 (en) 2014-09-30 2017-05-30 Apple Inc. Social reminders
US9697820B2 (en) 2015-09-24 2017-07-04 Apple Inc. Unit-selection text-to-speech synthesis using concatenation-sensitive neural networks
US9697822B1 (en) 2013-03-15 2017-07-04 Apple Inc. System and method for updating an adaptive speech recognition model
US9711141B2 (en) 2014-12-09 2017-07-18 Apple Inc. Disambiguating heteronyms in speech synthesis
US9715875B2 (en) 2014-05-30 2017-07-25 Apple Inc. Reducing the need for manual start/end-pointing and trigger phrases
US9721566B2 (en) 2015-03-08 2017-08-01 Apple Inc. Competing devices responding to voice triggers
US9734193B2 (en) 2014-05-30 2017-08-15 Apple Inc. Determining domain salience ranking from ambiguous words in natural speech
US9760559B2 (en) 2014-05-30 2017-09-12 Apple Inc. Predictive text input
US9785630B2 (en) 2014-05-30 2017-10-10 Apple Inc. Text prediction using combined word N-gram and unigram language models
US9798393B2 (en) 2011-08-29 2017-10-24 Apple Inc. Text correction processing
US9818400B2 (en) 2014-09-11 2017-11-14 Apple Inc. Method and apparatus for discovering trending terms in speech requests
US9842105B2 (en) 2015-04-16 2017-12-12 Apple Inc. Parsimonious continuous-space phrase representations for natural language processing
US9842101B2 (en) 2014-05-30 2017-12-12 Apple Inc. Predictive conversion of language input
US9858925B2 (en) 2009-06-05 2018-01-02 Apple Inc. Using context information to facilitate processing of commands in a virtual assistant
US9865280B2 (en) 2015-03-06 2018-01-09 Apple Inc. Structured dictation using intelligent automated assistants
US9886953B2 (en) 2015-03-08 2018-02-06 Apple Inc. Virtual assistant activation
US9886432B2 (en) 2014-09-30 2018-02-06 Apple Inc. Parsimonious handling of word inflection via categorical stem + suffix N-gram language models
US9899019B2 (en) 2015-03-18 2018-02-20 Apple Inc. Systems and methods for structured stem and suffix language models
US9922642B2 (en) 2013-03-15 2018-03-20 Apple Inc. Training an at least partial voice command system
US9934775B2 (en) 2016-05-26 2018-04-03 Apple Inc. Unit-selection text-to-speech synthesis based on predicted concatenation parameters
US9953088B2 (en) 2012-05-14 2018-04-24 Apple Inc. Crowd sourcing information to fulfill user requests
US9959870B2 (en) 2008-12-11 2018-05-01 Apple Inc. Speech recognition involving a mobile device
US9966065B2 (en) 2014-05-30 2018-05-08 Apple Inc. Multi-command single utterance input method
US9966068B2 (en) 2013-06-08 2018-05-08 Apple Inc. Interpreting and acting upon commands that involve sharing information with remote devices
US9971774B2 (en) 2012-09-19 2018-05-15 Apple Inc. Voice-based media searching
US9972304B2 (en) 2016-06-03 2018-05-15 Apple Inc. Privacy preserving distributed evaluation framework for embedded personalized systems
US10043516B2 (en) 2016-09-23 2018-08-07 Apple Inc. Intelligent automated assistant
US10049668B2 (en) 2015-12-02 2018-08-14 Apple Inc. Applying neural network language models to weighted finite state transducers for automatic speech recognition
US10049663B2 (en) 2016-06-08 2018-08-14 Apple, Inc. Intelligent automated assistant for media exploration
US10057736B2 (en) 2011-06-03 2018-08-21 Apple Inc. Active transport based notifications
US10067938B2 (en) 2016-06-10 2018-09-04 Apple Inc. Multilingual word prediction
US10074360B2 (en) 2014-09-30 2018-09-11 Apple Inc. Providing an indication of the suitability of speech recognition
US10079014B2 (en) 2012-06-08 2018-09-18 Apple Inc. Name recognition system
US10078631B2 (en) 2014-05-30 2018-09-18 Apple Inc. Entropy-guided text prediction using combined word and character n-gram language models
US10083688B2 (en) 2015-05-27 2018-09-25 Apple Inc. Device voice control for selecting a displayed affordance
US10089072B2 (en) 2016-06-11 2018-10-02 Apple Inc. Intelligent device arbitration and control
US10101822B2 (en) 2015-06-05 2018-10-16 Apple Inc. Language input correction
US10120862B2 (en) * 2017-04-06 2018-11-06 International Business Machines Corporation Dynamic management of relative time references in documents
US10127220B2 (en) 2015-06-04 2018-11-13 Apple Inc. Language identification from short strings
US10127911B2 (en) 2014-09-30 2018-11-13 Apple Inc. Speaker identification and unsupervised speaker adaptation techniques
US10134385B2 (en) 2012-03-02 2018-11-20 Apple Inc. Systems and methods for name pronunciation
US10162814B2 (en) * 2014-10-29 2018-12-25 Baidu Online Network Technology (Beijing) Co., Ltd. Conversation processing method, conversation management system and computer device
US10170123B2 (en) 2014-05-30 2019-01-01 Apple Inc. Intelligent assistant for home automation
US10176167B2 (en) 2013-06-09 2019-01-08 Apple Inc. System and method for inferring user intent from speech inputs
US10185542B2 (en) 2013-06-09 2019-01-22 Apple Inc. Device, method, and graphical user interface for enabling conversation persistence across two or more instances of a digital assistant
US10186254B2 (en) 2015-06-07 2019-01-22 Apple Inc. Context-based endpoint detection
US10192552B2 (en) 2016-06-10 2019-01-29 Apple Inc. Digital assistant providing whispered speech
US10199051B2 (en) 2013-02-07 2019-02-05 Apple Inc. Voice trigger for a digital assistant
US10223066B2 (en) 2015-12-23 2019-03-05 Apple Inc. Proactive assistance based on dialog communication between devices
US10241644B2 (en) 2011-06-03 2019-03-26 Apple Inc. Actionable reminder entries
US10241752B2 (en) 2011-09-30 2019-03-26 Apple Inc. Interface for a virtual digital assistant
US10249300B2 (en) 2016-06-06 2019-04-02 Apple Inc. Intelligent list reading
US10255907B2 (en) 2015-06-07 2019-04-09 Apple Inc. Automatic accent detection using acoustic models
WO2019070684A1 (en) * 2017-10-03 2019-04-11 Google Llc User-programmable automated assistant
US10269345B2 (en) 2016-06-11 2019-04-23 Apple Inc. Intelligent task discovery
US10276170B2 (en) 2010-01-18 2019-04-30 Apple Inc. Intelligent automated assistant
US10289433B2 (en) 2014-05-30 2019-05-14 Apple Inc. Domain specific language for encoding assistant dialog
US10297253B2 (en) 2016-06-11 2019-05-21 Apple Inc. Application integration with a digital assistant
US10318871B2 (en) 2005-09-08 2019-06-11 Apple Inc. Method and apparatus for building an intelligent automated assistant
US10354011B2 (en) 2016-06-09 2019-07-16 Apple Inc. Intelligent automated assistant in a home environment
US10356243B2 (en) 2015-06-05 2019-07-16 Apple Inc. Virtual assistant aided communication with 3rd party service in a communication session
US10366158B2 (en) 2015-09-29 2019-07-30 Apple Inc. Efficient word encoding for recurrent neural network language models
US20190258710A1 (en) * 2018-02-22 2019-08-22 Oath Inc. System and method for identifying and replacing slots with variable slots
US20190259380A1 (en) * 2018-02-22 2019-08-22 Oath Inc. System and method for rule based modifications to variable slots based on context
US10410637B2 (en) 2017-05-12 2019-09-10 Apple Inc. User-specific acoustic models
US10431202B2 (en) * 2016-10-21 2019-10-01 Microsoft Technology Licensing, Llc Simultaneous dialogue state management using frame tracking
US10446143B2 (en) 2016-03-14 2019-10-15 Apple Inc. Identification of voice inputs providing credentials
US10446141B2 (en) 2014-08-28 2019-10-15 Apple Inc. Automatic speech recognition based on user feedback
US10482884B1 (en) * 2013-12-17 2019-11-19 Amazon Technologies, Inc. Outcome-oriented dialogs on a speech recognition platform
US10482874B2 (en) 2017-05-15 2019-11-19 Apple Inc. Hierarchical belief states for digital assistants
US10490187B2 (en) 2016-06-10 2019-11-26 Apple Inc. Digital assistant providing automated status report
US10496753B2 (en) 2010-01-18 2019-12-03 Apple Inc. Automatically adapting user interfaces for hands-free interaction
US10509862B2 (en) 2016-06-10 2019-12-17 Apple Inc. Dynamic phrase expansion of language input
US10521466B2 (en) 2016-06-11 2019-12-31 Apple Inc. Data driven natural language event detection and classification
US10553209B2 (en) 2010-01-18 2020-02-04 Apple Inc. Systems and methods for hands-free notification summaries
US10552013B2 (en) 2014-12-02 2020-02-04 Apple Inc. Data detection
US10567477B2 (en) 2015-03-08 2020-02-18 Apple Inc. Virtual assistant continuity
US10593346B2 (en) 2016-12-22 2020-03-17 Apple Inc. Rank-reduced token representation for automatic speech recognition
US10592095B2 (en) 2014-05-23 2020-03-17 Apple Inc. Instantaneous speaking of content on touch devices
US10652394B2 (en) 2013-03-14 2020-05-12 Apple Inc. System and method for processing voicemail
US10659851B2 (en) 2014-06-30 2020-05-19 Apple Inc. Real-time digital assistant knowledge updates
US10671428B2 (en) 2015-09-08 2020-06-02 Apple Inc. Distributed personal assistant
US10672399B2 (en) 2011-06-03 2020-06-02 Apple Inc. Switching between text data and audio data based on a mapping
US10679605B2 (en) 2010-01-18 2020-06-09 Apple Inc. Hands-free list-reading by intelligent automated assistant
US10691473B2 (en) 2015-11-06 2020-06-23 Apple Inc. Intelligent automated assistant in a messaging environment
US10706373B2 (en) 2011-06-03 2020-07-07 Apple Inc. Performing actions associated with task items that represent tasks to perform
US10705794B2 (en) 2010-01-18 2020-07-07 Apple Inc. Automatically adapting user interfaces for hands-free interaction
US10733993B2 (en) 2016-06-10 2020-08-04 Apple Inc. Intelligent digital assistant in a multi-tasking environment
US10747498B2 (en) 2015-09-08 2020-08-18 Apple Inc. Zero latency digital assistant
US10755703B2 (en) 2017-05-11 2020-08-25 Apple Inc. Offline personal assistant
US10762293B2 (en) 2010-12-22 2020-09-01 Apple Inc. Using parts-of-speech tagging and named entity recognition for spelling correction
US10789041B2 (en) 2014-09-12 2020-09-29 Apple Inc. Dynamic thresholds for always listening speech trigger
US10791176B2 (en) 2017-05-12 2020-09-29 Apple Inc. Synchronization and task delegation of a digital assistant
US10791216B2 (en) 2013-08-06 2020-09-29 Apple Inc. Auto-activating smart responses based on activities from remote devices
US10810274B2 (en) 2017-05-15 2020-10-20 Apple Inc. Optimizing dialogue policy decisions for digital assistants using implicit feedback
US10991369B1 (en) * 2018-01-31 2021-04-27 Progress Software Corporation Cognitive flow
US11010550B2 (en) 2015-09-29 2021-05-18 Apple Inc. Unified language modeling framework for word prediction, auto-completion and auto-correction
US11025565B2 (en) 2015-06-07 2021-06-01 Apple Inc. Personalized prediction of responses for instant messaging
CN113342956A (en) * 2021-06-29 2021-09-03 神思电子技术股份有限公司 Method for filling same slot value in multi-turn conversation
US11217255B2 (en) 2017-05-16 2022-01-04 Apple Inc. Far-field extension for digital assistant services
US20220360668A1 (en) * 2021-05-10 2022-11-10 International Business Machines Corporation Contextualized speech to text conversion
US11587559B2 (en) 2015-09-30 2023-02-21 Apple Inc. Intelligent device identification
US11636852B2 (en) 2018-10-10 2023-04-25 Huawei Technologies Co., Ltd. Human-computer interaction method and electronic device
EP4250286A4 (en) * 2020-12-26 2023-12-27 Huawei Technologies Co., Ltd. Speech comprehension method and device

Families Citing this family (51)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20060015335A1 (en) * 2004-07-13 2006-01-19 Ravigopal Vennelakanti Framework to enable multimodal access to applications
GB0426347D0 (en) * 2004-12-01 2005-01-05 Ibm Methods, apparatus and computer programs for automatic speech recognition
US20060140357A1 (en) * 2004-12-27 2006-06-29 International Business Machines Corporation Graphical tool for creating a call routing application
US7706780B2 (en) * 2004-12-27 2010-04-27 Nokia Corporation Mobile communications terminal and method therefore
TWI269268B (en) * 2005-01-24 2006-12-21 Delta Electronics Inc Speech recognizing method and system
US7720684B2 (en) * 2005-04-29 2010-05-18 Nuance Communications, Inc. Method, apparatus, and computer program product for one-step correction of voice interaction
JP4680691B2 (en) * 2005-06-15 2011-05-11 富士通株式会社 Dialog system
JP2008065789A (en) * 2006-09-11 2008-03-21 Canon Inc Input operation support device and control method
JP4322907B2 (en) * 2006-09-29 2009-09-02 株式会社東芝 Dialogue device, dialogue method and computer program
US20090055163A1 (en) * 2007-08-20 2009-02-26 Sandeep Jindal Dynamic Mixed-Initiative Dialog Generation in Speech Recognition
US8812323B2 (en) * 2007-10-11 2014-08-19 Agency For Science, Technology And Research Dialogue system and a method for executing a fully mixed initiative dialogue (FMID) interaction between a human and a machine
US8296144B2 (en) * 2008-06-04 2012-10-23 Robert Bosch Gmbh System and method for automated testing of complicated dialog systems
US8914289B2 (en) * 2009-12-16 2014-12-16 Symbol Technologies, Inc. Analyzing and processing a verbal expression containing multiple goals
WO2011089450A2 (en) 2010-01-25 2011-07-28 Andrew Peter Nelson Jerram Apparatuses, methods and systems for a digital conversation management platform
KR101522837B1 (en) * 2010-12-16 2015-05-26 한국전자통신연구원 Communication method and system for the same
US10642934B2 (en) 2011-03-31 2020-05-05 Microsoft Technology Licensing, Llc Augmented conversational understanding architecture
US9244984B2 (en) 2011-03-31 2016-01-26 Microsoft Technology Licensing, Llc Location based conversational understanding
US9298287B2 (en) 2011-03-31 2016-03-29 Microsoft Technology Licensing, Llc Combined activation for natural user interface systems
US9760566B2 (en) 2011-03-31 2017-09-12 Microsoft Technology Licensing, Llc Augmented conversational understanding agent to identify conversation context between two humans and taking an agent action thereof
US9842168B2 (en) 2011-03-31 2017-12-12 Microsoft Technology Licensing, Llc Task driven user intents
US9858343B2 (en) 2011-03-31 2018-01-02 Microsoft Technology Licensing Llc Personalization of queries, conversations, and searches
US9454962B2 (en) 2011-05-12 2016-09-27 Microsoft Technology Licensing, Llc Sentence simplification for spoken language understanding
US9064006B2 (en) 2012-08-23 2015-06-23 Microsoft Technology Licensing, Llc Translating natural language utterances to keyword search queries
US9424840B1 (en) 2012-08-31 2016-08-23 Amazon Technologies, Inc. Speech recognition platforms
US10235358B2 (en) * 2013-02-21 2019-03-19 Microsoft Technology Licensing, Llc Exploiting structured content for unsupervised natural language semantic parsing
US20140379334A1 (en) * 2013-06-20 2014-12-25 Qnx Software Systems Limited Natural language understanding automatic speech recognition post processing
US10073840B2 (en) 2013-12-20 2018-09-11 Microsoft Technology Licensing, Llc Unsupervised relation detection model training
US9870356B2 (en) 2014-02-13 2018-01-16 Microsoft Technology Licensing, Llc Techniques for inferring the unknown intents of linguistic items
US10726831B2 (en) * 2014-05-20 2020-07-28 Amazon Technologies, Inc. Context interpretation in natural language processing using previous dialog acts
JP2018063272A (en) * 2015-02-27 2018-04-19 パナソニックIpマネジメント株式会社 Voice dialogue apparatus, voice dialogue system, and control method of voice dialogue apparatus
JP2018063271A (en) * 2015-02-27 2018-04-19 パナソニックIpマネジメント株式会社 Voice dialogue apparatus, voice dialogue system, and control method of voice dialogue apparatus
US20170147286A1 (en) * 2015-11-20 2017-05-25 GM Global Technology Operations LLC Methods and systems for interfacing a speech dialog with new applications
US10417346B2 (en) 2016-01-23 2019-09-17 Microsoft Technology Licensing, Llc Tool for facilitating the development of new language understanding scenarios
US11222633B2 (en) * 2016-05-20 2022-01-11 Nippon Telegraph And Telephone Corporation Dialogue method, dialogue system, dialogue apparatus and program
JP6719740B2 (en) * 2016-05-20 2020-07-08 日本電信電話株式会社 Interactive method, interactive system, interactive device, and program
EP3503091A4 (en) * 2016-08-17 2019-08-07 Sony Corporation Dialogue control device and method
US11281993B2 (en) 2016-12-05 2022-03-22 Apple Inc. Model and ensemble compression for metric learning
US10373515B2 (en) 2017-01-04 2019-08-06 International Business Machines Corporation System and method for cognitive intervention on human interactions
US10235990B2 (en) 2017-01-04 2019-03-19 International Business Machines Corporation System and method for cognitive intervention on human interactions
US10318639B2 (en) 2017-02-03 2019-06-11 International Business Machines Corporation Intelligent action recommendation
DK201770383A1 (en) 2017-05-09 2018-12-14 Apple Inc. User interface for correcting recognition errors
DK201770428A1 (en) 2017-05-12 2019-02-18 Apple Inc. Low-latency intelligent automated assistant
US10490185B2 (en) 2017-09-27 2019-11-26 Wipro Limited Method and system for providing dynamic conversation between application and user
JP7000096B2 (en) * 2017-09-28 2022-01-19 株式会社東芝 Dialogue processing device and dialogue processing system
CN108922531B (en) * 2018-07-26 2020-10-27 腾讯科技(北京)有限公司 Slot position identification method and device, electronic equipment and storage medium
CN112017642A (en) * 2019-05-31 2020-12-01 华为技术有限公司 Method, device and equipment for speech recognition and computer readable storage medium
US11275902B2 (en) 2019-10-21 2022-03-15 International Business Machines Corporation Intelligent dialog re-elicitation of information
US11379446B1 (en) 2021-07-23 2022-07-05 Fmr Llc Session-based data storage for chat-based communication sessions
US11463387B1 (en) 2021-07-23 2022-10-04 Fmr Llc Capturing and utilizing context data in virtual assistant communication sessions
US11734089B2 (en) 2021-10-11 2023-08-22 Fmr Llc Dynamic option reselection in virtual assistant communication sessions
US11763097B1 (en) 2022-08-02 2023-09-19 Fmr Llc Intelligent dialogue recovery for virtual assistant communication sessions

Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4914590A (en) * 1988-05-18 1990-04-03 Emhart Industries, Inc. Natural language understanding system
US6173266B1 (en) * 1997-05-06 2001-01-09 Speechworks International, Inc. System and method for developing interactive speech applications
US6219643B1 (en) * 1998-06-26 2001-04-17 Nuance Communications, Inc. Method of analyzing dialogs in a natural language speech recognition system
US6314402B1 (en) * 1999-04-23 2001-11-06 Nuance Communications Method and apparatus for creating modifiable and combinable speech objects for acquiring information from a speaker in an interactive voice response system
US6385584B1 (en) * 1999-04-30 2002-05-07 Verizon Services Corp. Providing automated voice responses with variable user prompting
US20020128821A1 (en) * 1999-05-28 2002-09-12 Farzad Ehsani Phrase-based dialogue modeling with particular application to creating recognition grammars for voice-controlled user interfaces
US20030130849A1 (en) * 2000-07-20 2003-07-10 Durston Peter J Interactive dialogues
US6598018B1 (en) * 1999-12-15 2003-07-22 Matsushita Electric Industrial Co., Ltd. Method for natural dialog interface to car devices
US20050033582A1 (en) * 2001-02-28 2005-02-10 Michael Gadd Spoken language interface
US20060025997A1 (en) * 2002-07-24 2006-02-02 Law Eng B System and process for developing a voice application

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2002023783A (en) * 2000-07-13 2002-01-25 Fujitsu Ltd Conversation processing system
US20040085162A1 (en) * 2000-11-29 2004-05-06 Rajeev Agarwal Method and apparatus for providing a mixed-initiative dialog between a user and a machine
EP1255190A1 (en) * 2001-05-04 2002-11-06 Microsoft Corporation Interface control
US6887690B2 (en) 2001-06-22 2005-05-03 Pe Corporation Dye-labeled ribonucleotide triphosphates
US7246062B2 (en) * 2002-04-08 2007-07-17 Sbc Technology Resources, Inc. Method and system for voice recognition menu navigation with error prevention and recovery

Patent Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4914590A (en) * 1988-05-18 1990-04-03 Emhart Industries, Inc. Natural language understanding system
US6173266B1 (en) * 1997-05-06 2001-01-09 Speechworks International, Inc. System and method for developing interactive speech applications
US6219643B1 (en) * 1998-06-26 2001-04-17 Nuance Communications, Inc. Method of analyzing dialogs in a natural language speech recognition system
US6314402B1 (en) * 1999-04-23 2001-11-06 Nuance Communications Method and apparatus for creating modifiable and combinable speech objects for acquiring information from a speaker in an interactive voice response system
US6385584B1 (en) * 1999-04-30 2002-05-07 Verizon Services Corp. Providing automated voice responses with variable user prompting
US20020128821A1 (en) * 1999-05-28 2002-09-12 Farzad Ehsani Phrase-based dialogue modeling with particular application to creating recognition grammars for voice-controlled user interfaces
US6598018B1 (en) * 1999-12-15 2003-07-22 Matsushita Electric Industrial Co., Ltd. Method for natural dialog interface to car devices
US20030130849A1 (en) * 2000-07-20 2003-07-10 Durston Peter J Interactive dialogues
US20050033582A1 (en) * 2001-02-28 2005-02-10 Michael Gadd Spoken language interface
US20060025997A1 (en) * 2002-07-24 2006-02-02 Law Eng B System and process for developing a voice application

Cited By (219)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9646614B2 (en) 2000-03-16 2017-05-09 Apple Inc. Fast, language-independent method for user authentication by voice
US10199039B2 (en) 2005-01-05 2019-02-05 Nuance Communications, Inc. Library of existing spoken dialog data for use in generating new natural language spoken dialog systems
US8694324B2 (en) 2005-01-05 2014-04-08 At&T Intellectual Property Ii, L.P. System and method of providing an automated data-collection in spoken dialog systems
US20060149553A1 (en) * 2005-01-05 2006-07-06 At&T Corp. System and method for using a library to interactively design natural language spoken dialog systems
US9240197B2 (en) 2005-01-05 2016-01-19 At&T Intellectual Property Ii, L.P. Library of existing spoken dialog data for use in generating new natural language spoken dialog systems
US8914294B2 (en) 2005-01-05 2014-12-16 At&T Intellectual Property Ii, L.P. System and method of providing an automated data-collection in spoken dialog systems
US20060217978A1 (en) * 2005-03-28 2006-09-28 David Mitby System and method for handling information in a voice recognition automated conversation
US20060215824A1 (en) * 2005-03-28 2006-09-28 David Mitby System and method for handling a voice prompted conversation
US7684990B2 (en) * 2005-04-29 2010-03-23 Nuance Communications, Inc. Method and apparatus for multiple value confirmation and correction in spoken dialog systems
US20060247931A1 (en) * 2005-04-29 2006-11-02 International Business Machines Corporation Method and apparatus for multiple value confirmation and correction in spoken dialog systems
US10318871B2 (en) 2005-09-08 2019-06-11 Apple Inc. Method and apparatus for building an intelligent automated assistant
US7778632B2 (en) * 2005-10-28 2010-08-17 Microsoft Corporation Multi-modal device capable of automated actions
US20070099602A1 (en) * 2005-10-28 2007-05-03 Microsoft Corporation Multi-modal device capable of automated actions
US20070239637A1 (en) * 2006-03-17 2007-10-11 Microsoft Corporation Using predictive user models for language modeling on a personal device
US7752152B2 (en) 2006-03-17 2010-07-06 Microsoft Corporation Using predictive user models for language modeling on a personal device with user behavior models based on statistical modeling
US8032375B2 (en) 2006-03-17 2011-10-04 Microsoft Corporation Using generic predictive models for slot values in language modeling
US20070219974A1 (en) * 2006-03-17 2007-09-20 Microsoft Corporation Using generic predictive models for slot values in language modeling
US7689420B2 (en) * 2006-04-06 2010-03-30 Microsoft Corporation Personalizing a context-free grammar using a dictation language model
US20070239454A1 (en) * 2006-04-06 2007-10-11 Microsoft Corporation Personalizing a context-free grammar using a dictation language model
US9117447B2 (en) 2006-09-08 2015-08-25 Apple Inc. Using event alert text as input to an automated assistant
US8930191B2 (en) 2006-09-08 2015-01-06 Apple Inc. Paraphrasing of user requests and results by automated digital assistant
US8942986B2 (en) 2006-09-08 2015-01-27 Apple Inc. Determining user intent based on ontologies of domains
US20080091426A1 (en) * 2006-10-12 2008-04-17 Rod Rempel Adaptive context for automatic speech recognition systems
US8767718B2 (en) * 2006-11-14 2014-07-01 Microsoft Corporation Conversation data accuracy confirmation
US20080114594A1 (en) * 2006-11-14 2008-05-15 Microsoft Corporation Conversation data accuracy confirmation
US8612225B2 (en) * 2007-02-28 2013-12-17 Nec Corporation Voice recognition device, voice recognition method, and voice recognition program
US20100070277A1 (en) * 2007-02-28 2010-03-18 Nec Corporation Voice recognition device, voice recognition method, and voice recognition program
US8977255B2 (en) 2007-04-03 2015-03-10 Apple Inc. Method and system for operating a multi-function portable electronic device using voice-activation
US10568032B2 (en) 2007-04-03 2020-02-18 Apple Inc. Method and system for operating a multi-function portable electronic device using voice-activation
US10381016B2 (en) 2008-01-03 2019-08-13 Apple Inc. Methods and apparatus for altering audio output signals
US9330720B2 (en) 2008-01-03 2016-05-03 Apple Inc. Methods and apparatus for altering audio output signals
US9530415B2 (en) 2008-01-22 2016-12-27 At&T Intellectual Property I, L.P. System and method of providing speech processing in user interface
US20090187410A1 (en) * 2008-01-22 2009-07-23 At&T Labs, Inc. System and method of providing speech processing in user interface
US9177551B2 (en) * 2008-01-22 2015-11-03 At&T Intellectual Property I, L.P. System and method of providing speech processing in user interface
US9626955B2 (en) 2008-04-05 2017-04-18 Apple Inc. Intelligent text-to-speech conversion
US9865248B2 (en) 2008-04-05 2018-01-09 Apple Inc. Intelligent text-to-speech conversion
US9535906B2 (en) 2008-07-31 2017-01-03 Apple Inc. Mobile device having human language translation capability with positional feedback
US10108612B2 (en) 2008-07-31 2018-10-23 Apple Inc. Mobile device having human language translation capability with positional feedback
US8874443B2 (en) * 2008-08-27 2014-10-28 Robert Bosch Gmbh System and method for generating natural language phrases from user utterances in dialog systems
US20100057463A1 (en) * 2008-08-27 2010-03-04 Robert Bosch Gmbh System and Method for Generating Natural Language Phrases From User Utterances in Dialog Systems
US20100070360A1 (en) * 2008-09-13 2010-03-18 At&T Intellectual Property I, L.P. System and method for creating a speech search platform for coupons
US9959870B2 (en) 2008-12-11 2018-05-01 Apple Inc. Speech recognition involving a mobile device
US20100217603A1 (en) * 2009-02-26 2010-08-26 Hammond Daniel D Method, System, and Apparatus for Enabling Adaptive Natural Language Processing
US11080012B2 (en) 2009-06-05 2021-08-03 Apple Inc. Interface for a virtual digital assistant
US9858925B2 (en) 2009-06-05 2018-01-02 Apple Inc. Using context information to facilitate processing of commands in a virtual assistant
US10475446B2 (en) 2009-06-05 2019-11-12 Apple Inc. Using context information to facilitate processing of commands in a virtual assistant
US10795541B2 (en) 2009-06-05 2020-10-06 Apple Inc. Intelligent organization of tasks items
US10283110B2 (en) 2009-07-02 2019-05-07 Apple Inc. Methods and apparatuses for automatic speech recognition
US9431006B2 (en) 2009-07-02 2016-08-30 Apple Inc. Methods and apparatuses for automatic speech recognition
US10276170B2 (en) 2010-01-18 2019-04-30 Apple Inc. Intelligent automated assistant
US11423886B2 (en) 2010-01-18 2022-08-23 Apple Inc. Task flow identification based on user intent
US10741185B2 (en) 2010-01-18 2020-08-11 Apple Inc. Intelligent automated assistant
US8903716B2 (en) * 2010-01-18 2014-12-02 Apple Inc. Personalized vocabulary for digital assistant
US9548050B2 (en) 2010-01-18 2017-01-17 Apple Inc. Intelligent automated assistant
US9318108B2 (en) 2010-01-18 2016-04-19 Apple Inc. Intelligent automated assistant
US10705794B2 (en) 2010-01-18 2020-07-07 Apple Inc. Automatically adapting user interfaces for hands-free interaction
US10496753B2 (en) 2010-01-18 2019-12-03 Apple Inc. Automatically adapting user interfaces for hands-free interaction
US10679605B2 (en) 2010-01-18 2020-06-09 Apple Inc. Hands-free list-reading by intelligent automated assistant
US8892446B2 (en) 2010-01-18 2014-11-18 Apple Inc. Service orchestration for intelligent automated assistant
US10706841B2 (en) 2010-01-18 2020-07-07 Apple Inc. Task flow identification based on user intent
US10553209B2 (en) 2010-01-18 2020-02-04 Apple Inc. Systems and methods for hands-free notification summaries
US20130117022A1 (en) * 2010-01-18 2013-05-09 Apple Inc. Personalized Vocabulary for Digital Assistant
US8670979B2 (en) * 2010-01-18 2014-03-11 Apple Inc. Active input elicitation by intelligent automated assistant
US20130110518A1 (en) * 2010-01-18 2013-05-02 Apple Inc. Active Input Elicitation by Intelligent Automated Assistant
US9633660B2 (en) 2010-02-25 2017-04-25 Apple Inc. User profiling for voice input processing
US9190062B2 (en) 2010-02-25 2015-11-17 Apple Inc. User profiling for voice input processing
US10049675B2 (en) 2010-02-25 2018-08-14 Apple Inc. User profiling for voice input processing
US20120089392A1 (en) * 2010-10-07 2012-04-12 Microsoft Corporation Speech recognition user interface
US10762293B2 (en) 2010-12-22 2020-09-01 Apple Inc. Using parts-of-speech tagging and named entity recognition for spelling correction
US9262612B2 (en) 2011-03-21 2016-02-16 Apple Inc. Device access using voice authentication
US10102359B2 (en) 2011-03-21 2018-10-16 Apple Inc. Device access using voice authentication
US11120372B2 (en) 2011-06-03 2021-09-14 Apple Inc. Performing actions associated with task items that represent tasks to perform
US10706373B2 (en) 2011-06-03 2020-07-07 Apple Inc. Performing actions associated with task items that represent tasks to perform
US10672399B2 (en) 2011-06-03 2020-06-02 Apple Inc. Switching between text data and audio data based on a mapping
US10057736B2 (en) 2011-06-03 2018-08-21 Apple Inc. Active transport based notifications
US10241644B2 (en) 2011-06-03 2019-03-26 Apple Inc. Actionable reminder entries
US9798393B2 (en) 2011-08-29 2017-10-24 Apple Inc. Text correction processing
US10241752B2 (en) 2011-09-30 2019-03-26 Apple Inc. Interface for a virtual digital assistant
US10134385B2 (en) 2012-03-02 2018-11-20 Apple Inc. Systems and methods for name pronunciation
US9483461B2 (en) 2012-03-06 2016-11-01 Apple Inc. Handling speech synthesis of content for multiple languages
US9953088B2 (en) 2012-05-14 2018-04-24 Apple Inc. Crowd sourcing information to fulfill user requests
US10079014B2 (en) 2012-06-08 2018-09-18 Apple Inc. Name recognition system
US9495129B2 (en) 2012-06-29 2016-11-15 Apple Inc. Device, method, and user interface for voice-activated navigation and browsing of a document
US9576574B2 (en) 2012-09-10 2017-02-21 Apple Inc. Context-sensitive handling of interruptions by intelligent digital assistant
US9971774B2 (en) 2012-09-19 2018-05-15 Apple Inc. Voice-based media searching
US10199051B2 (en) 2013-02-07 2019-02-05 Apple Inc. Voice trigger for a digital assistant
US10978090B2 (en) 2013-02-07 2021-04-13 Apple Inc. Voice trigger for a digital assistant
US11388291B2 (en) 2013-03-14 2022-07-12 Apple Inc. System and method for processing voicemail
US10652394B2 (en) 2013-03-14 2020-05-12 Apple Inc. System and method for processing voicemail
US9368114B2 (en) 2013-03-14 2016-06-14 Apple Inc. Context-sensitive handling of interruptions
US9922642B2 (en) 2013-03-15 2018-03-20 Apple Inc. Training an at least partial voice command system
US9697822B1 (en) 2013-03-15 2017-07-04 Apple Inc. System and method for updating an adaptive speech recognition model
US9620104B2 (en) 2013-06-07 2017-04-11 Apple Inc. System and method for user-specified pronunciation of words for speech synthesis and recognition
US9633674B2 (en) 2013-06-07 2017-04-25 Apple Inc. System and method for detecting errors in interactions with a voice-based digital assistant
US9582608B2 (en) 2013-06-07 2017-02-28 Apple Inc. Unified ranking with entropy-weighted information for phrase-based semantic auto-completion
US9966060B2 (en) 2013-06-07 2018-05-08 Apple Inc. System and method for user-specified pronunciation of words for speech synthesis and recognition
US10657961B2 (en) 2013-06-08 2020-05-19 Apple Inc. Interpreting and acting upon commands that involve sharing information with remote devices
US9966068B2 (en) 2013-06-08 2018-05-08 Apple Inc. Interpreting and acting upon commands that involve sharing information with remote devices
US10185542B2 (en) 2013-06-09 2019-01-22 Apple Inc. Device, method, and graphical user interface for enabling conversation persistence across two or more instances of a digital assistant
US10176167B2 (en) 2013-06-09 2019-01-08 Apple Inc. System and method for inferring user intent from speech inputs
US9300784B2 (en) 2013-06-13 2016-03-29 Apple Inc. System and method for emergency calls initiated by voice command
US20150019228A1 (en) * 2013-07-15 2015-01-15 International Business Machines Corporation Automated confirmation and disambiguation modules in voice applications
US9298811B2 (en) * 2013-07-15 2016-03-29 International Business Machines Corporation Automated confirmation and disambiguation modules in voice applications
US10791216B2 (en) 2013-08-06 2020-09-29 Apple Inc. Auto-activating smart responses based on activities from remote devices
US11037572B1 (en) 2013-12-17 2021-06-15 Amazon Technologies, Inc. Outcome-oriented dialogs on a speech recognition platform
US10482884B1 (en) * 2013-12-17 2019-11-19 Amazon Technologies, Inc. Outcome-oriented dialogs on a speech recognition platform
US11915707B1 (en) 2013-12-17 2024-02-27 Amazon Technologies, Inc. Outcome-oriented dialogs on a speech recognition platform
US9620105B2 (en) 2014-05-15 2017-04-11 Apple Inc. Analyzing audio input for efficient speech and music recognition
US10592095B2 (en) 2014-05-23 2020-03-17 Apple Inc. Instantaneous speaking of content on touch devices
US9502031B2 (en) 2014-05-27 2016-11-22 Apple Inc. Method for supporting dynamic grammars in WFST-based ASR
US10078631B2 (en) 2014-05-30 2018-09-18 Apple Inc. Entropy-guided text prediction using combined word and character n-gram language models
US9715875B2 (en) 2014-05-30 2017-07-25 Apple Inc. Reducing the need for manual start/end-pointing and trigger phrases
US10083690B2 (en) 2014-05-30 2018-09-25 Apple Inc. Better resolution when referencing to concepts
US9785630B2 (en) 2014-05-30 2017-10-10 Apple Inc. Text prediction using combined word N-gram and unigram language models
US10169329B2 (en) 2014-05-30 2019-01-01 Apple Inc. Exemplar-based natural language processing
US10170123B2 (en) 2014-05-30 2019-01-01 Apple Inc. Intelligent assistant for home automation
US9633004B2 (en) 2014-05-30 2017-04-25 Apple Inc. Better resolution when referencing to concepts
US10497365B2 (en) 2014-05-30 2019-12-03 Apple Inc. Multi-command single utterance input method
US9760559B2 (en) 2014-05-30 2017-09-12 Apple Inc. Predictive text input
US9734193B2 (en) 2014-05-30 2017-08-15 Apple Inc. Determining domain salience ranking from ambiguous words in natural speech
US11133008B2 (en) 2014-05-30 2021-09-28 Apple Inc. Reducing the need for manual start/end-pointing and trigger phrases
US10289433B2 (en) 2014-05-30 2019-05-14 Apple Inc. Domain specific language for encoding assistant dialog
US9430463B2 (en) 2014-05-30 2016-08-30 Apple Inc. Exemplar-based natural language processing
US11257504B2 (en) 2014-05-30 2022-02-22 Apple Inc. Intelligent assistant for home automation
US9966065B2 (en) 2014-05-30 2018-05-08 Apple Inc. Multi-command single utterance input method
US9842101B2 (en) 2014-05-30 2017-12-12 Apple Inc. Predictive conversion of language input
US10659851B2 (en) 2014-06-30 2020-05-19 Apple Inc. Real-time digital assistant knowledge updates
US10904611B2 (en) 2014-06-30 2021-01-26 Apple Inc. Intelligent automated assistant for TV user interactions
US9338493B2 (en) 2014-06-30 2016-05-10 Apple Inc. Intelligent automated assistant for TV user interactions
US9668024B2 (en) 2014-06-30 2017-05-30 Apple Inc. Intelligent automated assistant for TV user interactions
US10446141B2 (en) 2014-08-28 2019-10-15 Apple Inc. Automatic speech recognition based on user feedback
US10431204B2 (en) 2014-09-11 2019-10-01 Apple Inc. Method and apparatus for discovering trending terms in speech requests
US9818400B2 (en) 2014-09-11 2017-11-14 Apple Inc. Method and apparatus for discovering trending terms in speech requests
US10789041B2 (en) 2014-09-12 2020-09-29 Apple Inc. Dynamic thresholds for always listening speech trigger
US10074360B2 (en) 2014-09-30 2018-09-11 Apple Inc. Providing an indication of the suitability of speech recognition
US9646609B2 (en) 2014-09-30 2017-05-09 Apple Inc. Caching apparatus for serving phonetic pronunciations
US10127911B2 (en) 2014-09-30 2018-11-13 Apple Inc. Speaker identification and unsupervised speaker adaptation techniques
US9668121B2 (en) 2014-09-30 2017-05-30 Apple Inc. Social reminders
US9986419B2 (en) 2014-09-30 2018-05-29 Apple Inc. Social reminders
US9886432B2 (en) 2014-09-30 2018-02-06 Apple Inc. Parsimonious handling of word inflection via categorical stem + suffix N-gram language models
US10162814B2 (en) * 2014-10-29 2018-12-25 Baidu Online Network Technology (Beijing) Co., Ltd. Conversation processing method, conversation management system and computer device
US11556230B2 (en) 2014-12-02 2023-01-17 Apple Inc. Data detection
US10552013B2 (en) 2014-12-02 2020-02-04 Apple Inc. Data detection
US9711141B2 (en) 2014-12-09 2017-07-18 Apple Inc. Disambiguating heteronyms in speech synthesis
US9865280B2 (en) 2015-03-06 2018-01-09 Apple Inc. Structured dictation using intelligent automated assistants
US9721566B2 (en) 2015-03-08 2017-08-01 Apple Inc. Competing devices responding to voice triggers
US9886953B2 (en) 2015-03-08 2018-02-06 Apple Inc. Virtual assistant activation
US10311871B2 (en) 2015-03-08 2019-06-04 Apple Inc. Competing devices responding to voice triggers
US10567477B2 (en) 2015-03-08 2020-02-18 Apple Inc. Virtual assistant continuity
US11087759B2 (en) 2015-03-08 2021-08-10 Apple Inc. Virtual assistant activation
US9899019B2 (en) 2015-03-18 2018-02-20 Apple Inc. Systems and methods for structured stem and suffix language models
US9842105B2 (en) 2015-04-16 2017-12-12 Apple Inc. Parsimonious continuous-space phrase representations for natural language processing
US10083688B2 (en) 2015-05-27 2018-09-25 Apple Inc. Device voice control for selecting a displayed affordance
US10127220B2 (en) 2015-06-04 2018-11-13 Apple Inc. Language identification from short strings
US10101822B2 (en) 2015-06-05 2018-10-16 Apple Inc. Language input correction
US10356243B2 (en) 2015-06-05 2019-07-16 Apple Inc. Virtual assistant aided communication with 3rd party service in a communication session
US10186254B2 (en) 2015-06-07 2019-01-22 Apple Inc. Context-based endpoint detection
US11025565B2 (en) 2015-06-07 2021-06-01 Apple Inc. Personalized prediction of responses for instant messaging
US10255907B2 (en) 2015-06-07 2019-04-09 Apple Inc. Automatic accent detection using acoustic models
US10747498B2 (en) 2015-09-08 2020-08-18 Apple Inc. Zero latency digital assistant
US10671428B2 (en) 2015-09-08 2020-06-02 Apple Inc. Distributed personal assistant
US11500672B2 (en) 2015-09-08 2022-11-15 Apple Inc. Distributed personal assistant
US9697820B2 (en) 2015-09-24 2017-07-04 Apple Inc. Unit-selection text-to-speech synthesis using concatenation-sensitive neural networks
US10366158B2 (en) 2015-09-29 2019-07-30 Apple Inc. Efficient word encoding for recurrent neural network language models
US11010550B2 (en) 2015-09-29 2021-05-18 Apple Inc. Unified language modeling framework for word prediction, auto-completion and auto-correction
US11587559B2 (en) 2015-09-30 2023-02-21 Apple Inc. Intelligent device identification
US11526368B2 (en) 2015-11-06 2022-12-13 Apple Inc. Intelligent automated assistant in a messaging environment
US10691473B2 (en) 2015-11-06 2020-06-23 Apple Inc. Intelligent automated assistant in a messaging environment
US10049668B2 (en) 2015-12-02 2018-08-14 Apple Inc. Applying neural network language models to weighted finite state transducers for automatic speech recognition
US10223066B2 (en) 2015-12-23 2019-03-05 Apple Inc. Proactive assistance based on dialog communication between devices
US10446143B2 (en) 2016-03-14 2019-10-15 Apple Inc. Identification of voice inputs providing credentials
US9934775B2 (en) 2016-05-26 2018-04-03 Apple Inc. Unit-selection text-to-speech synthesis based on predicted concatenation parameters
US9972304B2 (en) 2016-06-03 2018-05-15 Apple Inc. Privacy preserving distributed evaluation framework for embedded personalized systems
US10249300B2 (en) 2016-06-06 2019-04-02 Apple Inc. Intelligent list reading
US10049663B2 (en) 2016-06-08 2018-08-14 Apple, Inc. Intelligent automated assistant for media exploration
US11069347B2 (en) 2016-06-08 2021-07-20 Apple Inc. Intelligent automated assistant for media exploration
US10354011B2 (en) 2016-06-09 2019-07-16 Apple Inc. Intelligent automated assistant in a home environment
US10490187B2 (en) 2016-06-10 2019-11-26 Apple Inc. Digital assistant providing automated status report
US10509862B2 (en) 2016-06-10 2019-12-17 Apple Inc. Dynamic phrase expansion of language input
US10192552B2 (en) 2016-06-10 2019-01-29 Apple Inc. Digital assistant providing whispered speech
US10733993B2 (en) 2016-06-10 2020-08-04 Apple Inc. Intelligent digital assistant in a multi-tasking environment
US10067938B2 (en) 2016-06-10 2018-09-04 Apple Inc. Multilingual word prediction
US11037565B2 (en) 2016-06-10 2021-06-15 Apple Inc. Intelligent digital assistant in a multi-tasking environment
US10297253B2 (en) 2016-06-11 2019-05-21 Apple Inc. Application integration with a digital assistant
US10089072B2 (en) 2016-06-11 2018-10-02 Apple Inc. Intelligent device arbitration and control
US11152002B2 (en) 2016-06-11 2021-10-19 Apple Inc. Application integration with a digital assistant
US10521466B2 (en) 2016-06-11 2019-12-31 Apple Inc. Data driven natural language event detection and classification
US10269345B2 (en) 2016-06-11 2019-04-23 Apple Inc. Intelligent task discovery
US10553215B2 (en) 2016-09-23 2020-02-04 Apple Inc. Intelligent automated assistant
US10043516B2 (en) 2016-09-23 2018-08-07 Apple Inc. Intelligent automated assistant
US10431202B2 (en) * 2016-10-21 2019-10-01 Microsoft Technology Licensing, Llc Simultaneous dialogue state management using frame tracking
US10593346B2 (en) 2016-12-22 2020-03-17 Apple Inc. Rank-reduced token representation for automatic speech recognition
US10120862B2 (en) * 2017-04-06 2018-11-06 International Business Machines Corporation Dynamic management of relative time references in documents
US11151330B2 (en) 2017-04-06 2021-10-19 International Business Machines Corporation Dynamic management of relative time references in documents
US10592707B2 (en) 2017-04-06 2020-03-17 International Business Machines Corporation Dynamic management of relative time references in documents
US10755703B2 (en) 2017-05-11 2020-08-25 Apple Inc. Offline personal assistant
US10791176B2 (en) 2017-05-12 2020-09-29 Apple Inc. Synchronization and task delegation of a digital assistant
US10410637B2 (en) 2017-05-12 2019-09-10 Apple Inc. User-specific acoustic models
US11405466B2 (en) 2017-05-12 2022-08-02 Apple Inc. Synchronization and task delegation of a digital assistant
US10810274B2 (en) 2017-05-15 2020-10-20 Apple Inc. Optimizing dialogue policy decisions for digital assistants using implicit feedback
US10482874B2 (en) 2017-05-15 2019-11-19 Apple Inc. Hierarchical belief states for digital assistants
US11217255B2 (en) 2017-05-16 2022-01-04 Apple Inc. Far-field extension for digital assistant services
US11276400B2 (en) * 2017-10-03 2022-03-15 Google Llc User-programmable automated assistant
US20220130387A1 (en) * 2017-10-03 2022-04-28 Google Llc User-programmable automated assistant
US11887595B2 (en) * 2017-10-03 2024-01-30 Google Llc User-programmable automated assistant
WO2019070684A1 (en) * 2017-10-03 2019-04-11 Google Llc User-programmable automated assistant
US10431219B2 (en) * 2017-10-03 2019-10-01 Google Llc User-programmable automated assistant
US10991369B1 (en) * 2018-01-31 2021-04-27 Progress Software Corporation Cognitive flow
US20190258710A1 (en) * 2018-02-22 2019-08-22 Oath Inc. System and method for identifying and replacing slots with variable slots
US20210074278A1 (en) * 2018-02-22 2021-03-11 Oath Inc. System and method for rule based modifications to variable slots based on context
US20190259380A1 (en) * 2018-02-22 2019-08-22 Oath Inc. System and method for rule based modifications to variable slots based on context
US10783879B2 (en) * 2018-02-22 2020-09-22 Oath Inc. System and method for rule based modifications to variable slots based on context
US11030400B2 (en) * 2018-02-22 2021-06-08 Verizon Media Inc. System and method for identifying and replacing slots with variable slots
US11915693B2 (en) * 2018-02-22 2024-02-27 Yahoo Assets Llc System and method for rule based modifications to variable slots based on context
US11636852B2 (en) 2018-10-10 2023-04-25 Huawei Technologies Co., Ltd. Human-computer interaction method and electronic device
EP4250286A4 (en) * 2020-12-26 2023-12-27 Huawei Technologies Co., Ltd. Speech comprehension method and device
US20220360668A1 (en) * 2021-05-10 2022-11-10 International Business Machines Corporation Contextualized speech to text conversion
US11711469B2 (en) * 2021-05-10 2023-07-25 International Business Machines Corporation Contextualized speech to text conversion
CN113342956A (en) * 2021-06-29 2021-09-03 神思电子技术股份有限公司 Method for filling same slot value in multi-turn conversation

Also Published As

Publication number Publication date
US20070255566A1 (en) 2007-11-01
EP1779376A4 (en) 2008-09-03
EP1779376A2 (en) 2007-05-02
JP2008506156A (en) 2008-02-28
ES2621808T3 (en) 2017-07-05
ES2526698T3 (en) 2015-01-14
WO2006014432A3 (en) 2007-01-04
EP1779376B1 (en) 2014-09-17
EP2282308B1 (en) 2017-02-22
EP2282308A1 (en) 2011-02-09
US7228278B2 (en) 2007-06-05
US7747438B2 (en) 2010-06-29
WO2006014432A2 (en) 2006-02-09

Similar Documents

Publication Publication Date Title
US7228278B2 (en) Multi-slot dialog systems and methods
EP1380153B1 (en) Voice response system
CA2493265C (en) System and method for augmenting spoken language understanding by correcting common errors in linguistic performance
US9088652B2 (en) System and method for speech-enabled call routing
US8229745B2 (en) Creating a mixed-initiative grammar from directed dialog grammars
US8064573B2 (en) Computer generated prompting
US8457973B2 (en) Menu hierarchy skipping dialog for directed dialog speech recognition
US7542907B2 (en) Biasing a speech recognizer based on prompt context
US8185399B2 (en) System and method of providing an automated data-collection in spoken dialog systems
US20030130849A1 (en) Interactive dialogues
US20060287868A1 (en) Dialog system
US7870000B2 (en) Partially filling mixed-initiative forms from utterances having sub-threshold confidence scores based upon word-level confidence data
JP2000194386A (en) Voice recognizing and responsing device
US20050043953A1 (en) Dynamic creation of a conversational system from dialogue objects
US7461344B2 (en) Mixed initiative interface control
CN113987149A (en) Intelligent session method, system and storage medium for task robot
US20060031853A1 (en) System and method for optimizing processing speed to run multiple dialogs between multiple users and a virtual agent
US7853451B1 (en) System and method of exploiting human-human data for spoken language understanding systems
Thymé-Gobbel et al. Conveying Reassurance with Confidence and Confirmation

Legal Events

Date Code Title Description
AS Assignment

Owner name: VOXIFY, INC., CALIFORNIA

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:NGUYEN, PATRICK T.M.;LOPEZ-AMARO, JESUS;DESAI, AMIT V.;AND OTHERS;REEL/FRAME:015558/0308

Effective date: 20040701

STCF Information on status: patent grant

Free format text: PATENTED CASE

FPAY Fee payment

Year of fee payment: 4

AS Assignment

Owner name: 24/7 CUSTOMER, INC., CALIFORNIA

Free format text: PATENT ASSIGNMENT;ASSIGNOR:VOXIFY, LLC;REEL/FRAME:029590/0508

Effective date: 20120301

FEPP Fee payment procedure

Free format text: PAT HOLDER NO LONGER CLAIMS SMALL ENTITY STATUS, ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: STOL); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

FPAY Fee payment

Year of fee payment: 8

MAFP Maintenance fee payment

Free format text: PAYMENT OF MAINTENANCE FEE, 12TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1553); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

Year of fee payment: 12

AS Assignment

Owner name: (24)7.AI, INC., CALIFORNIA

Free format text: CHANGE OF NAME;ASSIGNOR:24/7 CUSTOMER, INC.;REEL/FRAME:049688/0636

Effective date: 20171019