WO2009029802A2

WO2009029802A2 - A system and method for screening and rating agents to assess multiple skills as a predictor of agent performance

Info

Publication number: WO2009029802A2
Application number: PCT/US2008/074805
Authority: WO
Inventors: Jayant M. Naik; Cordell Coy; Ajay Harikumar Warrier; Karthik M. Narayanaswami; Mary-Ann Ruth Claridge; Matthew John Yuschik
Original assignee: Convergys Gmc Utah Inc.
Priority date: 2007-08-29
Filing date: 2008-08-29
Publication date: 2009-03-05
Also published as: WO2009029802A3

Abstract

One version of a physical system disclosed in the application relates to predicting the expected retention of an agent based on skill assessment scores. In this version, an automated system for predicting a retention rate for an individual based upon a set of performance indicators, includes a testing device and a prediction subsystem. The testing device is encoded with computer- executable instructions to test the individual with regard to a chosen set of performance indicators. The testing device formulates a score which it sends to the prediction subsystem. The prediction subsystem compares the score with retention level data stored in a repository and issues a recommendation relating to the employment status of the individual.

Description

A System and Method for Screening and Rating Agents to Assess Multiple Skills as a

Predictor of Agent Performance

Jayant M. Naik

Cordell Coy

Aj ay Harikumar Warrier

Karthik M. Narayanaswami

Mary-Ann Ruth Claridge

Matthew John Yuschik

PRIORITY

[0001] This application claims priority to U.S. Non-provisional Application Serial

Number 11/846,590 filed August 29, 2007 which is hereby incorporated in its entirety.

FIELD OF THE INVENTION

[0002] The present disclosure relates in general to physical computing systems, and in some versions to an at least partly automated testing system for individuals.

BACKGROUND

[0003] Contact centers typically experience very high agent attrition rates. These drive up the cost of recruitment and training significantly. Furthermore, current methods for evaluating a candidate for said jobs are highly subjective and do not provide a consistent, objective assessment of a candidate.

[0004] Predictive analytics are employed by companies like SAS (www.sas.com). Other companies and products (including, but not limited to, FurstPerson, KnowlAgent, Versant, Carnegie Speech, Nexidia, Nemesysco, MeritTrac, Employment Technologies Corporation, Degarmo Group, Versant, and VURV) provide tools which test typing, fluency, vocabulary, data entry, work habits, pronunciation, work attitudes, listening skills, computer skills, analytical ability, decision making, risk assessment, sentence mastery, business reasoning, and work ability.

None of these tools, however, teach a system for predicting the expected retention of an agent, based on skill assessment scores tested and evaluated prior to employment.

[0005] Call centers historically face up to 100% attrition rates so there is a long-felt need in this industry to provide a better form of analysis to hire and nurture capable agents, as well as intervene appropriately with under-performing or marginal agents. By using machine learning to correlate scores with profiles of good agents and candidates, more accurate predictions/reviews can be made.

SUMMARY

[0006] Automated speech recognition (ASR) is a technology that allows users of information systems to speak entries rather than punching numbers on a keypad. ASR is used primarily to provide information and to forward telephone calls. In recent years, ASR has become popular in the customer service departments of large corporations. Basic ASR systems recognize single-word entries such as yes- or-no responses and spoken numerals. Sophisticated ASR systems allow the user to enter direct queries or responses. (See, http://searchmobilecomputing.techtarget.eom/sDefmition/Q,, sid40_gci786138,00. html ).

[0007] Text-to- speech (TTS) is a type of speech synthesis application that is used to create a spoken sound version of the text in a computer document, such as a help file or a Web page. TTS can enable the reading of computer display information for the visually challenged person, or may simply be used to augment the reading of a text message. Current TTS applications include voice-enabled e-mail and spoken prompts in voice response systems. TTS is often used with voice recognition programs. There are numerous TTS products available, including Read Please 2000, Proverbe Speech Unit, and Next Up Technology's TextAloud. Lucent, Elan, and AT&T each have products called "Text-to-Speech." (See, http://searchmobilecomputing.techtarget.com/sDefinition/0,,sid40 gci775360,00. html) [0008] In computing, natural language refers to a human language such as English,

Russian, German, or Japanese as distinct from the typically artificial command or programming language with which one usually talks to a computer. The term usually refers to a written language but might also apply to spoken language. (See, http://whatis.techtarget.com/definition/0,,sid9 gci887541,00.htm1^*)

[0009] As a broad subfield of artificial intelligence, machine learning is concerned with the design and development of algorithms and techniques that allow computers to "learn." At a general level, there are two types of learning: inductive, and deductive. Inductive machine learning methods extract rules and patterns out of massive data sets. The major focus of machine learning research is to extract information from data automatically by computational and statistical methods, hence, machine learning is closely related to data mining and statistics but also theoretical computer science. (See, http://en.wikipedia.org/wiki/Machine learning). Embodiments of machine learning may appear in "supervised adaption" and "adaption of algorithms" to post-hire agent performance.

[0010] A user interface is everything designed into an information device with which a human being may interact ~ including display screen, keyboard, mouse, light pen, the appearance of a desktop, illuminated characters, help messages, and how an application program or a Web site invites interaction and responds to it. (See, http://searchwebservices.techtarget.com/sDefinition/0,,sid26_gci214505,00.html)

[0011] Second, as many aspects of this disclosure are described as being stored on or incorporating "computer readable medium," a "computer readable medium" should be understood to refer to any device or material which is capable of storing information which can be retrieved using a machine. The phrase "computer readable medium" should not be restricted to physically contiguous structures. Thus, distributed media, decentralized media, remote media and local media should all be understood to be non-limiting examples of computer readable media. Similarly, a "computer," as used in this disclosure, should be understood to refer to any device or group of devices which is capable of performing one or more logical and/or physical operations on data to produce a result. The phrase "computer executable instructions," as used in this disclosure refers to data which can be used to specify physical or logical operations which can be performed by a computer. Additionally, a number of aspects of this disclosure refer to, or incorporate, an engine. As used in this disclosure, an "engine" should be understood to refer to a core piece of code which provides common functionality which is utilized by other documents or pieces of code. Examples of how portions of this disclosure can be embodied are set forth below to, among other reasons, provide additional illumination as to the meaning of the terms used herein.

In an embodiment of the invention, there is disclosed an automated system for predicting agent retention rates based upon a set of performance indicators comprising a testing module and a prediction module. The testing module further comprises a speaker verification module encoded with programming instructions to set the identity of a candidate and at least one testing module encoded with computer-executable instructions to evaluate said set of performance indicators and wherein said at least one testing module calculates a score for said candidate wherein said score is calculated through the use of a machine learning algorithm selected from the group consisting of: feed-forward artificial neural networks, within-class agent related training, and discriminative training. One or more performance indicators may be chosen from the group consisting of: average handle time, customer satisfaction survey ratings, attendance, work performance ratings, script compliance, workflow efficiency, emotion events detected during a customer interaction, a quality rating for said customer interaction. The prediction module receives said score form said testing module. The score may be intermediately stored in a database or other computerized repository en route to the prediction module. The prediction module may also have to query the intermediate storage location for the score in a "pull" paradigm. The prediction module is encoded with programming instructions to determine if the candidate should be hired by querying a retention level associated with said score from said repository; comparing said retention level against a predetermined threshold; if said retention level equals or exceeds a predetermined level, issue a recommendation to hire said candidate.

[0013] Certain aspects of this disclosure can be embodied in a computer readable medium having stored thereon computer executable instructions operable to configure a computer to perform acts comprising predicting agent retention rates based upon a set of performance indicators comprising a testing module and a prediction module via identifying a candidate through a testing module through speaker verification techniques; evaluating said set of performance indicators; calculating a score for said candidate wherein said score is calculated through the use of a machine learning algorithm selected from the group consisting of: feedforward artificial neural networks, within-class agent related training, and/or discriminative training. One or more performance indicators may be chosen from the group consisting of: average handle time, customer satisfaction survey ratings, attendance, work performance ratings, script compliance, workflow efficiency, emotion events detected during a customer interaction, a quality rating for said customer interaction. Further steps may include receiving, at a prediction module, a score from said testing module; storing the score in a database or other computerized repository en route to the prediction module; alternatively, querying the intermediate storage location for the score in a "pull" paradigm. Further steps may include determining if the candidate should be hired by querying a retention level associated with said score from said repository; comparing said retention level against a predetermined threshold; and if said retention level equals or exceeds a predetermined level, issuing a recommendation to hire said candidate.

[0014] An automated system may be a computer or other system which performs tasks automatically, that is, done or produced as if by machine. "Predict" and various forms thereof refer to a foretelling on the basis of observation, experience, or scientific reason. An agent refers to a human operating in a customer service capacity including, but not limited to, call center agents, technical specialists, sales representatives, etc. [0015] Retention rates refer to the prediction that a given agent will continue in the employ of a specific employer.

[0016] A testing module and a prediction module refer to computer programs (or even modules or sections of computer executable instructions within a single program) which accomplish the functions of calculating a candidate score and determining an agent retention prediction, respectively.

[0017] A speaker verification module is a set of computer-executable instructions stored on a computer-readable medium which set the identity of a candidate. Other forms of identity verification may be substituted here (e.g., fingerprinting, other biometrics, specific personal knowledge quiz, etc.) Identity refers to the condition of being the same with something described or asserted. A candidate should be understood to refer to a person applying for a job in some embodiments although evaluation of an existing employee may also be referred to as a candidate. Retention evaluations of candidate may also be applied in other paradigms than the employment scenario (e.g., a candidate may be a student who is being evaluated to predict their likely retention rate in an academic program).

[0018] Performance indicators include, but are not limited to, data which indicates the performance of a given agent such as average handle time (refers to the time the agent spends on a call resolving one or more issues), customer satisfaction survey ratings (e.g., which might be developed at the conclusion of a call or via on-line), attendance (refers to the presence of the agent in a capacity to work either physically, virtually, or remotely), work performance ratings (which may be developed based on subjective criteria, objective criteria, by a supervisor or otherwise), script compliance (this measures how often and how far the agent deviates from the prepared script and whether such deviations are desirable or not), workflow efficiency (this refers to processing time and the agent's ability to utilize tools correctly and efficiently to resolve customer issues), emotion events detected during a customer interaction (this refers to either a manual or automated notation, regarding an emotion which may utilize variations in volume, spoken words, etc., in conjunction with natural language understanding, review of recorded interactions, etc.), and a quality rating for said customer interaction (this refers to a calculation regarding the overall quality of the interaction as opposed to the agent's specific quality - it may take into account the difficulty level of the interaction as well as the customer satisfaction with the resolution).

[0019] Score should be construed to mean a number that expresses accomplishment (as in a game or test) or excellence (as in quality) either absolutely in points gained or by comparison to a standard. Calculating should be construed to mean to solving.

BRIEF DESCRIPTION OF THE DRAWINGS

[0020] FIGURE 1 shows a sample flow for an embodiment of the system.

[0021] FIGURE 2 illustrates how an embodiment of the system architecture.

[0022] FIGURES 3-4 illustrate some example machine learning paradigms.

DETAILED DESCRIPTION

[0023] Referring to Figure 1, there is a secure automated system (100) which tests and scores a candidate (1 10) on their comprehension and comprehensibility via, at least, one or more of the following: spoken speech patterns, accuracy of textual and speech responses to spoken passages, and quality of text/voice dialog determination. A user interface is provided for testing comprehension and comprehensibility.

[0024] The system (100) may be set up to verify the identity of the candidate (110), participating in the test via any number of security and/or integrity tools that are well-known to those of skill in the art including voiceprints, fingerprints, retinal scanning, other biometrics, etc. These measures may be employed to ensure the integrity of each test by providing a signal if an impersonation takes place or if a candidate (1 10) attempts a fraudulent retest by pretending to be someone else.

[0025] The system (100) may be implemented within a portable kiosk (120). [0026] Once the test is complete, a final score may be calculated which is then compared with a threshold score associated with high-quality agents that have a met threshold of retention. In an embodiment, a final score may be calculated from a combination of test scores weighted by correlation to top agent scores (which may be based on a plurality of key performance indicators). The right agent profile may be determined through statistical analysis. Of course, a given agent's test scores may change throughout their career and this would also be factored into the overall profile.

[0027] The test may pull from technology detailed in "A System for Dialog Quality

Management in an Automated Care System," US Serial No. 11/749,983, which was filed May 17, 2007, which utilizes a dialog quality agent to assess actual dialog quality in an interaction. This application is hereby incorporated by reference.

[0028] A machine learning component may use post-screening, post-training information about an agent and a pool of agents to optimize the rating algorithm based on new, empirical information about agent performance.

[0029] Input to the algorithm may include:

Agent demographics

Agent Key Performance Index (KPI) o Average handle time o Customer satisfaction survey rating o Attendance, work performance ratings o Script compliance and workflow efficiency o Information from automated monitoring, related to :

^■ Emotion events during customer interaction

^■ Quality of interaction

[0030] Various algorithmic approaches may be used to re-train the rating algorithm from the aforementioned information types. These may include, but not be limited to, feed-forward artificial neural networks, within-class agent related training, discriminative training. [0031] Referring to Figures 3 and 4, "Feed-Forward Artificial Neural Networks" comprise multiple layers of back-propagation networks. The weights (w's) in Figure 3 may be determined from a preliminary analysis of archival data related to agent hiring and post-training agent performance. During training, forward and backward passes through the input data and the output result may be adjusted by reducing the back-propagation error using a gradient descent algorithm. Referring to Figure 4, a hidden layer may be added to the simple back- propagation model, which results in a non-linear network that is used to optimize the rating output by searching a large hypothesis space for the correct output (rating) from among a large number of possible weight values.

[0032] "Within-class" agent related training, the weights may be determined from post- screening and post-training information, such as, KPIs as the agents progress through their screening, training and successful performance in their jobs. Reinforcement learning techniques may be used to determine which set of KPIs with appropriate weights are correlated with the target agent hires - who perform well in their jobs and have high retention rates in their assignments. These may then be used to define the hidden layer in Figure 4.

[0033] "Discriminative training" may process KPI information related to agents who constitute the rejection class, i.e., poor performers will be used as negative weights to further optimize the rating algorithm, such that correlates of poor performance in initial screening data, such as, performance on screening tests, skills assessment and analytical abilities are used to optimize the rating algorithm to select target agent candidates.

[0034] A means for communicating the results of the test (e.g., screen, printer, etc.)

(130) may be included in the system to provide a diagnostic score and/or other feedback to the candidate (110).

[0035] Aspects of scoring may be utilized from "Program Performance Management

System," US Serial No. 10/624,283, which was filed July 22, 2003, which integrates a scorecard with information available from consolidated reports database to provide an assessment of current performance. This application is hereby incorporated by reference.

[0036] Spoken Speech Patterns & Accuracy of Textual and Speech Responses to Spoken and Written Passages

[0037] A candidate may participate in a simulated dialog. To provide consistency across candidate scores, the simulated dialog may be a control. The system utilizes tools to analyze the spoken speech patterns including audibility, rate of speech, intonation patterns, accents, word gaps, word stress, (intonation and word stress may be collectively referred to as prosody), recognizability (was the candidate's meaning understood), intelligibility (is the candidate's speech understandable), phonetic structure, and comprehension of spoken utterances.

[0038] The candidate may be tested on his/her reading rate/style. For instance, the agent may be provided with a control script to read to a customer. Comprehension and comprehensibility of text-based interactions may also be evaluated.

[0039] Quality of Text/Voice Dialog

[0040] Decision analytics and dialog monitoring may be employed to the result set of the candidate's responses to evaluate the overall interaction as well as rate individual responses for desirability in addition to comprehension.

[0041] Implementation

[0042] Various tools exist for assessing different aspects of speech which would be known to one of skill in the art. Such tools may include (but are not limited to Third Party Vendor Assessments, E Skills, Call Center Simulation, Customer Service Fit Index, Ordinate Tool, Applicant Tracking System, Knowlagent Continuing Training and Communications, and Knowlagent Initial Training).

[0043] Referring to Figure 2, there is a screening module (210) which interfaces with a speaker verification module (215), a large vocabulary ASR (220), a text-to-speech interface (225), and an automatic speech recognition module (230). Other module configurations using third party tools (such as those described above) may also be utilized. Candidate scores and speaker verification data are sent (201) to a repository (140). A repository may comprise any computer-readable storage media including, but not limited to, a database. The repository may be queried across all of the candidates/agents being tested, a demographic set of agents, a particular agent, the type of testing being conducted (candidate, training or operational) as well as other demarcations. In an embodiment, the geographic attribute associated with a candidate may affect the prediction of retention because agents may have one set of reasons to quit in, for example, India (e.g., job is viewed as a stepping stone to other paths) versus another location, for example, America (agent quits out of boredom). Repository data (including scores, metrics, etc.) is fed (291) into the machine learning module (300). The machine learning algorithms may be structured as cases to treat whatever subset of data is pulled from the repository.

[0044] Rating and screening algorithms, included in the skills validation module, may be utilized to evaluate a simulated dialogue with a candidate (110) using Natural Language Understanding tools, known to those of skill in the art, to develop a candidate score. The candidate score as well as speaker verification data (SV Data) are sent to a back-end repository (140) to be utilized by the machine learning algorithms (300) . The system (100) may also evaluate risk factors and utilize this data to adjust predictions regarding the tenure of a given candidiate with the company (e.g., whether or not the candidate has a history of using drugs).

[0045] The machine learning algorithms may produce a recommendation which is tailored to a specific task. That is, the recommendation may be to hire the candidate but for webchat interactions rather than voice interactions in order to ensure a greater probability of retention.

[0046] The system (100) may also be applied to an agent's training (240) as well as live operations (270). Their performance and tenure will be similarly evaluated, using the tools described above or their equivalents in the art, and used as input for improving the screening aspect (210) of the system (100). Data concerning whether a candidate remains employed, is an exemplary employee, is a below- average employee, is fired, or quits is also stored in the repository (140) so that initial scores associated with a candidate may be matched to the expected performance and retention of that employee.

[0047] Training (240) may include "Culture and Communications Training" module

(CCT) (250) and Program Specific Training (PST) (255) (i.e., if the agent is being hired to handle in-bound calls for a computer technical support company then there may be specific training associated with that job function as well). A skills validation test (210) may be performed to determine whether the prospective hire is qualified to begin live operations. Once again, speaker verification data may be sent from the repository (140) to the testing module (210) to ensure that the same person who took the test is the one who showed up for training.

[0048] Data from the CCT and PST modules may be sent along with the candidate scores to the central repository (140). The machine learning module (300) may be designed to evaluate this data in making its predictions concerning retention. In an embodiment, the data may be statistically evaluated and looking at a set of key performance indicators, from a set of known high performing agents, a "model" may be developed to compare against incoming candidate scores. Although new candidate scores may not score as high as experienced agents, a curve can be developed by continuing to score an individual over the tenure of their employment. This will facilitate the development of scores desired from a new candidate for future predictions.

[0049] It is not necessary that a candidate have participated in screening for an embodiment to assess their performance in the training module or for those scores to be utilized in the machine learning aspect of the system.

[0050] Operations

[0051] Once a candidate (110) is operating in a live contact center, an embodiment of the invention may be utilized to perform screenings, in that environment, which determine, at least, one or more of the following: spoken speech patterns, accuracy of textual and speech responses to spoken passages, and quality of text/voice dialog determination. Key Performance Indicators (KPIs) (275) (e.g., punctuality, average handle time, customer satisfaction scores (C-SAT scores), conformance to standards such as following the script, up-sells, frequency of barge-in, ability to use different types of interfaces such as voice/multi- modal/web-chat/etc. , and teamwork) as well as agent scores may be collected and sent to the repository (140). Screenings [280] at predetermined intervals (e.g., annual, monthly, weekly) may take place.

[0052] It is not necessary that a candidate have participated in screening and/or training for an embodiment to assess their performance in the operations module (270) and make recommendations regarding retention or intervention.

[0053] Machine Learning

[0054] Embodiments of the invention employ the use of machine learning techniques to process the data collected in the repository (140) and utilize patterns discovered therein to predict retention rates based on performance levels. These predictions may be used to make hiring decisions, focus training as well as perform interventions with already hired employees that are determined to have a high expectation of quitting.

[0055] In an embodiment, the machine learning may be tailored to geographic indicators.

That is, the prediction algorithm (or the weighting of characteristics) for a candidate in India might be different than for an agent in America. Therefore, the machine learning processes may be geographically specific. The machine learning algorithms could also be tailored on the basis of other demographic information (e.g., age of candidate, educational background of candidate, etc.).

[0056] The "score" developed re a candidate/trainee/agent's performance which is used as an input for the prediction algorithm could be arrived at via a number of different pathways. For instance, Agent #1 may be higher in one aspect but lower in another and vice versa for Agent #2, but they may end up with the same "score." Alternatively, a candidate may be more suitable to certain tasks (e.g., webchat versus phone interactions) than others and yet have the same overall score as another agent. Thus, a recommendation may need to be qualified by some additional attribute that would be considered by either the system or via a live interview.

What is claimed:

Claims

We claim:

(1) An automated system for predicting a retention rate for an individual based upon a set of performance indicators, comprising:

(a) a testing device comprising an evaluation module, wherein the testing device is encoded with computer-executable instructions to: i. accept a set of performance indicators for an individual, ii. tune the evaluation module to choose a subset of the set of performance indicators, iii. evaluate the individual based on the subset of performance indicators, and iv. calculate a score for the individual based on the evaluation of the subset of performance indicators; and

(b) a prediction subsystem encoded with computer-executable instructions to: i. accept the score from the testing device, ii. compare the score with a set of retention level data stored in a repository, iii. predict a retention rate based on the previous comparison of the score and the set of retention level data, and iv. issue an employment status recommendation based on the retention rate.

(2) The automated system of claim 1 wherein tuning the evaluation module comprises assigning a desired emphasis to be placed on at least one performance indicator from the subset of performance indicators.

(3) The automated system of claim 2 wherein assigning a desired emphasis comprises assigning a weight percentage.

(4) The automated system of claim 1 wherein the set of performance indicators is chosen from the group consisting of: average handle time, customer satisfaction survey ratings, attendance, work performance ratings, script compliance, workflow efficiency, emotion events detected during a customer interaction, a quality rating for the customer interaction, or any combination thereof.

(5) An automated system for predicting a retention rate of at least one individual having at least one attribute comprising:

(a) a repository for storing retention level data;

(b) a testing device encoded with computer-executable instructions to evaluate a set of performance indicators and calculate a score for the candidate; and

(c) a prediction device encoded with computer-executable instructions to receive the score from the testing module and formulate an employment status recommendation for the individual by:

(i) querying a retention level associated with the score from the repository, and

(ii) comparing the retention level against a predetermined threshold.

(6) The automated system of claim 5 wherein querying the retention level associated with the score from the repository comprises:

(a) weighting the score of the individual, the at least one attribute relating to the individual, and the at least one performance indicator to which the test was directed;

(b) comparing the individual's score, the at least one attribute, and the at least one performance indicator to the retention level data stored in the repository; and

(c) formulating a retention level based on the comparison and respective weights accorded to the individual's score, the at least one attribute, and the at least one performance indicator.

(7) The automated system of claim 6 wherein the prediction device may vary the weight respectively accorded to the individual's score, the at least one attribute relating to the individual, and the at least one performance indicator to which the test was directed.

(8) The automated system of claim 5 wherein the set of performance indicators is selected by the testing device using a machine learning algorithm.

(9) The automated system of claim 8 wherein using the machine learning algorithm to select the set of performance indicators comprises:

(a) analyzing the at least one attribute of the individual in view of the retention level data in the repository; and

(b) formulating the set of performance indicators to be tested based on the analysis.

(10) The automated system of claim 5 wherein the at least one attribute is selected from the group consisting of age, sex, educational background, race, employment status, income, place of birth, previous employment history, financial history, location of residences, demographic factors, and any combination thereof.