CA2773219A1

CA2773219A1 - Displaying relationships between electronically stored information to provide classification suggestions via nearest neighbor

Info

Publication number: CA2773219A1
Application number: CA2773219A
Authority: CA
Inventors: William C. Knight; Nicholas I. Nussbaum
Original assignee: FTI Consulting Inc
Current assignee: Nuix North America Inc
Priority date: 2009-07-28
Filing date: 2010-07-27
Publication date: 2011-02-10
Anticipated expiration: 2030-07-27
Also published as: US9542483B2; US8700627B2; EP2460099A1; WO2011017065A1; US20110029527A1; US20140236947A1; WO2011017133A4; CA2773220C; US8635223B2; WO2011017155A4; US20170046617A1; US9064008B2; US20150286957A1; US8909647B2; CA2773220A1; WO2011017155A1; CA2773194A1; US20140136539A1; US8713018B2; WO2011017134A1

Abstract

A system (11) and method (40) for providing reference documents (14b) as a suggestion for classifying uncoded documents (14a) is provided. Reference electronically stored information items (14b) and a set of uncoded electronically stored information items (14a) are designated. Each of the reference information items are previously classified. At least one uncoded electronically stored information item (14a) is compared with the reference electronically stored information items (14b). One or more of the reference electronically stored information items (14b) similar to the at least one uncoded electronically stored information items (14a) are identified. Relationships are depicted between the at least one uncoded electronically stored information item (14a) and the similar reference electronically stored information items (14b) for classifying the at least one uncoded electronically stored information item (14a).

Description

DISPLAYING RELA'I'IO SHIPS BETAk'EEN ELECTRONICALLY STORED
IN FORMATION TO PROVIDE CLASSIFICATION SUGGESTIONINS
VIA NEAREST NEIGHBOR

TECHNICAL FIELD
This application relates in general to using document: as a reference point and, in particular, to a system and method for displaying relationships bet veera electronically stored information to provide classification suggestions via nearest neighbor.

BACKGROUND ART
Historically, document. review during the discover phase of litigation and for other types of legal matters. such as due diligence and regulatory compliance, have been conducted manually. During document review, individual reviez Hers, generally licensed attorneys, are assigned sets of documents for coding, A reviewer must carefully study each document and categorize he document by assigning a code or other marker from a set of descriptive classifications, such as "privileged," "responsi.ve," and "non-responsive."
The classifications can affect the disposition of each document, including admissibility into evidence.
During discovery, document review Can potentially a.ffct the outcome of the underlying legal matter, so consistent a gad accurate results are crucial. Manual document review is tedious and tinae-consuming. Marking documents is solely at the discretion of each reviewer and iracoaasistent results maa_v occur due to misunderstanding, time pressures, f rtigue, or other factors.
A large volume of documents reviewed, often with only limited time, can create a loss of maental focus and a loss of purpose for the resultant classification, Each new reviewer also faces a steep learning curve to become familiar with the legal matter, classification categories, and review techniques.
Currently, with the increasingly widespread movement to electronically stored, information (E I), manual document review is no longer practicable, The often exponential growth of ES1. exceeds the bounds reasonable or conventional manual human document review and underscores the need for computer-assisted 1.1 review tools.
Conventional ES! review tools have proven inadequate to providing efficient, accurate, and consistent results. For example. DiscoverReady LLC, a Delaware limited liability cor-npany.
custom programs ES! review tools, which conduct semi-auto.rnated document review through multiple passes over a document set in .l S! form. During the first pass, documents are grouped by category and basic codes are assigned. ubse luent p a ~ ~ f ine and 3i .rtt~ r ss.ign c ~clf.~~ s.
fultiple pass review requires a Priori project-specific knowledge engineering, which is only useful for the sirr-le project, tlaereb losing the benefit of any.inferred knowledge or know-how for use in other review projects-Thus, there :rema.ins a need .for a system and method for increasing the efficiency of document review that bootstraps knowledge gained from other reviews while ultimately ensuring independent reviewer discretion.

DISCLOSURE OF THE INVENTION
Document review efficiency call. be increased by identifying relationships between reference EST and encoded ESI, and providing a suggestion for classification based on the relationships. The uncoded E`I for a document review project are identified and clustered. At least one of the uncoded EST is selected from the clusters and compared with the reference M
based on as sinaila_rity metric. The reference ESI most similar to the selected. uncoded EST are ide tined. Classification codes assigned to the similar reference ESI can be used to. provide suggestions for classification of the selected trncoded. ESI.. Further, a machine-generated suggestion for classification code can be provided with a confidence level.
An embodiment provides a system and method for displaying relationships between electronically stored information to provide classification suggestions via nearest neitghbor.
Reference electronically stored information items and a set of uncoded. e lectronicaally> stored information items are, designated. Each of the reference information items are previously classified. At least one encoded electronically stored .information item is compared with the reference electronically stored information items. One or more of the reference electronically stored information items similar to the at least one uncoded electronically stored information items are identified. Relationships are depicted between the at least one unceded electronically stored infiorniation item and the sianilaar reference electronically stored rnlbrtnation items for classifying the at least one trncoded electronically stored information iteraa.
further embodiment provides a system aand method for identifying reference documents for use in claassif~.inu uncoded documents. A set of reference documents is designated. Each reference document is associated with a classification code. A set of clusters each .including uncoded documents is designated. At least one uncoded document is selected and compared with each of the reference documents. One or more reference documents that satisfy a threshold of similarity with the at least one uncoded document is identified.Relaationships between the at least one uncoded document and the similar reference documents are displayed based on. the associated classification codes as suggestions for classifying the at least one uncoded document.

2 Still other embodiments of the present iaven.Ãioii will become readily apparent to those skilled in the art from the following detailed. description, wherein are described embodiments by-ww aay of illustrating the best mode contemplated for carrying out the invention. As will be realized, the invention is capable of other and different embodiments and its several details are capable of modifications i a various obvious respects, all without departing from the spirit and the scope of the present invention, Accordingly, the drawings and detailed description are to be regarded as illustrative in nature and not as restrictive.

DESCRIPTION OF THE DRAWINGS
FIGURE I is a block diagram showing a system for displaying relationships between electronically stored information to provide classification suggestions via nearest neighbor, in accordance with one embodirwaent.
FIGURE'.' is a. process.flow diagram showing. a method for displaying relationships between electronically stored information to provide classification sug4gestions via nearest neighbor, in accordance with one embodiment, FIGURE 3 is a block diagram. showing, by way of example, measures for selecting a document reference sabsct.
FIGURE 4 is a process flow diagram showing, by way of example, a method or comparing an encoded document to ref rernce documents for Use in the n. ethod of FIGURE 1 FIGURE 5 is a screenshot showing, by way of exaar:rple, a visual display of re.Ãerc rrce documents in relation to encoded docunaents.
FIGURE 6 is an alternative visual display of the similar reference documents and uncoded documents.
FIGURE 7 is a process flow diagram showing, br way of example, a method for classifying uncoded documents for use in the method of FIGURE 2, The eve-increasin volume of ES underlies the .need for automating document review for improved consist:enc ~ and throughput. l revi.ousl : coded doc:aara tints offer knowwled e Fle aped front earlier work in similar legal projects. as well as a reference point for classifying uncoded EST.
Reference documents are documents that have been previously classified by content and can be used to influence classification of uncoded, that is unclassified, ESL
Specifically.
relationships between the encoded ESI and the reference ES! can be visually depicted to provide suggestions, for instance to a human reviewer, for classifying the visually-Proximal uncoded EST,

3 C oniplete EST review requires a support enviroiimetat within which classification, call be performed. FIGURE I is a block diagram showing a system 10 for displaying relationships between electronically stored inforrnatiotn to provide el'a.5s{.Ã.aca'ion satp~est.ions via nearest neis hbor, in accordance with one embodiment. By way of illustration, the system 10 operates in a, distc.ibuted coraaput:.ine environtrie.nt Which includes a plurality and ESI sources. Henceforth, a single item of ESI will be referenced as a <.dc cc tzaetaÃ altho gh ESI
can include other.forty s of non-document data, as described E?f w, A backend server I I is coupled to a storage device 13, which stores docin ea :s 14a, such as uncoded documents, in the form of structured or unstructa.ured data, a database 30 for maintaining information about the documents, and a look-Lip database 38 for storing many-to-many mappings 39 between documents and document features, such as concepts. The storage device 13 also stores reference documents 14b. which can. provide a training set of trusted and knovn results for use in guiding EST classification. The reference documents 1.4b are each associated with an assigned classification code and considered as classified or coded. Hereinafter, the terms' classified" an d "coded" are used interchaanYpeaabl.y with the same intended meaning, unless otherwise indicated.
A. set of reference documents can be hand-selected or automaatica.lly selected through gitided.
review, which is further discussed below. Additionally. the set of reference documents can be predetermined or can be generated dynamically, as the selected encoded documents are classified and subsequently added to the. set of reference documents.
The backend server I I is coupled to an intranetwork 2I and executes a workbench suite 3I for providing a user interface framework for automated document management, processing, analysis, and classification. In a further embodiment, the baackend server IT
can be accessed via an internetwoak 22. The workbench software suite 31 includes a document mapper 32 that includes a clustering engine 33, similarity searcher 34, classifier 35, ,and display generator 36.
Other workbench suite modules a:te possible.
The clustering engine 33 performs efficient document scoring and clustering of documents, including uncoded and coded documents, such as described in commonly-assigned U.S. Patent. No. ?,60,313, the disclosure of which is incorporated by reference. Clusters of encoded documents I4a can be formed and organized along vectors, known as spines, based on a similarity of the clusters- which can be expressed in terms of distance.
During clustering, groupings of related documents are provided. The content of each document can be converted into a set of tokens. which are word-level or character- level nrgr arias, raw terms, concepts, or entities. Other tokens are possible.. An n-grant is a predetermined :ntumbe.r of items selected fromal a source. The items can include syllables, letters, or words, as well as other items. A raw term is

4 a term that has not been processed or nranipuulaated. Concepts typically include nouns and noun phrases obtained through part-of-speech tagging that have a common semantic meaning. Entities further refine nouns and noun phrases into people, places. and things, such as.tateetings, animals, relationships, and various other objects. Entities can be extracted using entity extract]( M.1 techniques known in the, field. Clustering. of the documents can be based on cluster criteria, such.
as the similarity of tokens, including n-grams: raw terms, concepts, entities, email addresses. or other metadata.
In as further embodiment. the clusters can include uncoded and coded.
documents, which are generated based on a similarity with the uncoded documents, as discussed in commonly-owned U.S. Patent Application Serial No, 12.1"833,860, entitled. "System and Iethod. tot Displaying Relationships Between Electronically Stored Information to Provide Classification Suggestions via Inclusion," filed July 9, 2010, pending, and U.S. Patent Application Serial No.
1-1/8'3'3,872, entitled "1Systern and. Method f r Displaying Relationships Between Electronically Stored Information to Provide Classification SuggesÃions via hrjection." filed July 9, 2010, pending, the disclosures of which are incorporated by reference.
The similarity searcher 34 identifies the reference documents l 4b that. are most si a ilaar to ,,elected uncoded documents 14aa.. clusters, or spines, as.further described below with reference to l IGLTRF 4. For exatmaple. the uncoded documents, reference documents, clusters, and spines can each. he represented by a score vector, which includes paired values consisting of a token, such as a term occurring in that document, cluster or spine, and the associated score for that token.
Subsequently, the score vector of the uncoded document, cluster, or spine is then compared with the score vectors of the reference documents to identif-~' similar reference docu.nients.
The classifier 35 provides a machine-generated suggestion and confidence level for classification of selected uncoded documents 14a, clusters, or spines, as further described below with reference to FIGURE 7. The display generator Y, arranges the clusters and spiries in thematic relationships in a two-dimensional visual display space, as further described below beginning with reference to FIGURE 5. Once generated, the visual display space is transmitted to a work client 12 by the backend server 1 1 via the document mapper 32 for presenting to a reviewver on a display 32. The reviewer can. include an individual. person who is assigned. to revie and claassif ,one. car aaaore. taaae.odecl doetaraae its b} designating as code:. Hereinafter, the terms "reviewer" and "custodian" are used interchangeably e: ith the same intended nleanirng.
unless otherwise indicated. Other types of reviewers are possible, including machirte.-iitmpl.emented reviewers. 5 The document mapper 32 opernates on uncoded l4a and coded documents I4b, which can be retrieved from the storage 13, as well as from a plurality of local and remote sources, The locaal source's include a local server 15, which is coupled to a storage device 16 with docr.unerats 17 and a local client 18, which is coupled to a storage device 1.9 with documents 20. The local server 15 and local client 18 are interconnected to the backend server 1. l and the work client 12 over an intranetwor=k 21. In adddition_ the document mapper 32 can identify and retrieve documents from remote sources over an internetwork 22, including= the Internet-through. a gateway 2;:1 interflrced to the intraanetwork 21. The remote sources include as rerr:r(ite server 24, which is coupled to a storage device 2.5 with documents 26 and a remote client. 27, which is coupled to a storage device 28 with documents 29. Other document sources, either local or remote, are possible.
The individual documents 17, 20, 26, 29 include all forms and tees of structured and, unstructured ESI, including electronic message stores, word processing documents, electronic mail (email.) folders. Web pages, and graphical or multimedia data. Notw Withstanding, the documents could. be in the form of structurally organized data, such as stored in ti spreadsheet. or database.
In one embodiment, the individual documents 1.4a, 141), 17, 20, ':6, 29 include electronic message folders storing email and attachments, such as maintained by the Outlook and Outlook Express products-, licensed. by Microsoft. Corporation, Redmond, WA. The database can be an SQL-based relational database, such as the Oracle database management system, Release 8, licensed by Oracle Corporation, Redwood Shores, CA.
The individual documents 17, 20, 26.29 can be designated and stored as uracoded documents or reference documents. The uncoded documents, which are unclassified, are selected for a document review project and stored as a document corpus for classification. The reference documents are initially encoded documents that can be selected from the corpus or other source of uncoded documents, and subsequently classified. The reference documents can assist in prop iding suggestions for classification of the remaining uncoded documents based on visual relationships between the uncoiled documents and reference documen s.
Ina further embodiment, the .reference documents can provide classification suggestions for a document 3ti corpus associated with a related document review project. In. yet a further embodiment, the reference documents can be used as aa. training set to forna machine--generaaied suggestions for classifying uncoded documents, as further described below with reference to I;IGUIR 7.
The document corpus for a document review project can. be divided into subset, of uacoded documents, which are each provided to a particular reviewer as an assignment, To maintain consistency, the aaane classification codes can be used across aall a.sssiEraaaraents in the document review protect. Alternatively, the classification codes can be different for each assignment. The classification codes can be detennined using taxonomy generation, during which a list of Classification codes can be provided b-,,, a reviewer or determined automatically.
For purposes of legal discovery, the list of classification codes can include "privileged,"
responsive." or " nora-respousiv e," however, other classificat on codes are possible. A
13rivile:ged" document coratairrs i.nformai:ic n that is protected by a pa-iv i1c ge, meaning that the document should not be disclosed or "produced" to an opposing party.
Disclosing a "privileged"
document can result in an unintentional waiver of the subject matter disclosed. A "respou ive"
document contains information that as related to a legal matter on which the document review project is based and a " aon-resporas.iv e" document includes information that is .not related to the.
legal matter.
The system 10 includes individual computer systems, such as the backend server 11, work server 12, server 1.5, client 18, remote server 24 and remote client 27.
The individual computer systems are general purpose, pro rarnarari cl digital cc raaputi.iag devices consisting. of a central processing unit. (CPU)_ random access memory> (RAM), non-volatile secondary storage, sracla as a hard drive or CD ROM drive, network interfiaces, and peripheral devices, including user interfacing means, such as a keyboard and display. The various implementations of the source code and object and byte codes can be held on a computer-readable storage mediuram, such as a floppy disk, hard drive, digital video disk. (DVD), random access memory (RAM.), read-only meanory. (ROM) and similar storage mediums. For example, program code, including soft-ware pro rains, and data are loaded into the RA 4 for execution and processing by the CPU and results are generated for display, output, transmittal, or storage.
Identifying relationships between the reference documents and uncoiled documents includes clustering and similarity measures. FIGURE 2 is a process flow dia,;rarn showing a method 40 for displaying relationships between electronically stored information to provide classification suggestions via nearest neighbor, in accordance with one embodiment, A set of document clusters is obtaained. (block 41), In one embodiment, the clusters can hiclude encoded documents, a.nd in a further embodiment, the clusters can include uncoded and coded docauaarments.
The clustered uncoded documents can represent a corpus of encoded documents fora document review project, or one or more assignments of uncoded documents. The document corpus can include all uricoded documents for a document review project, while, each assignment can include a subset of uncoded documents selected from. the corpus and assigned to a reviewer. The corpus can be divided into aassignrraents using assignment criteria, such as custodian or source of the encoded document, content, documeaat type, and date. Other criteria are possible. Prior to, concurrent with, or subsequent to obtaining, the cluster set, reference documents are identified (block 42). The reference documents can include all reference documents generated for a document review project., or alternatively, a subset of the reference documents. Obtaining retere nce documents is further discussed below with reference to FIGURE 3.
An uncoded document is selected from. one of the clusters 'in tile set and compared aaotaus( the reference documents (block. 43) to iclearÃ:if:y one or more reference documents that are similar to the selected uncoded document (block- 44). The similar reference docurrients are identified based on a similarity measure calculated between the selected uncoded document and each reference document, Co a p arina the selected encoded document with the reference documents is further discussed below with reference to FIGURE 4. Once identified, relationships between the selected encoded document and the similar reference documents can be identified (block 45) to provide classification hints, including a suggestion for the selected encoded document, as fÃarther discussed below with reference to FIGURE 5.
Additionally, raaa_chine- eneraated suggestions for classification can be provided (block 46) withal associated.
confidence level for use in classifying the selected uncoded document. Machine-generated suggestions are further discussed below with reference to FK R1 . Once the selected uncoded document is assigned a classification code. either by the reviewer or a atatomaaticaa.lly, the .nevvl\
classified document can be added to the set of reference documents for use in claassif =ing further uncoded documents. Subsequently, a further encoded document can be selected for classification using similar reference doÃ u Dents.
In a further eaa:alacadi.aiaea t, similar reference documents can also be identified For a selected cluster or a selected spine along which the clusters are placed.
After the clusters have been generated, one or more uncoded documents can be selected from at least one of he clusters for comparing, with. a reference document set or subset.
FIGURE 3 is a block diagram slro a ramp,, by way of example, measures 50 for selecting a document reference subset ti 1. The subset of reference documents 51 can be previously defined 54 and maaintaained.for related document review projects or can be specifically generated for each review prr ject:. A predefined reference subset 54 provides knowledge previously obtained.
during the related document review protect to increase efficiency, accuracy., and consistency.
Reference subsets newly generated for each review project can include arbitrary 52 or customized S3 reference subsets that are deternrined aautomaticaall , or by a human reviewer. An arbitrary reference subset 52 includes reference documents randoinily selected .for inclusion in the reference subset. A customized reference subset 53 includes reference documents specifically selected for inclusion in the refrence subset based on cr.iteria_ such as reviewer preference, classification caategor4y, document source, content, and review project. Other criteria are possible.
The subset of refiere.nce documents, whether predetermined or newly =raer<ate htaild be selected trom a set of reference documents that are representative of the docunierat corpus for aa, review project in which data Organization or classificatiora is desired, Guided review assists a reviewer or other user in identatvinL, reference do?ctir erits that are rE
resentati e of the corpus for use in classifying ulicoded docu.aaaenÃs. During added review, the uncoded documents that area dissimilar to all other imoded documents are identified based on a similarity threshold. In one embodiment, the dissimilarity can be determined as the cos of the score vectors for the uncoded documents. Other methods for determining dissimilarity are possible.
Identifying the dissimilar documents prods ides a group of documents that are representative of the corpus for a document review project. Each identified dissimilar document is then classified by assigning a particular classification code based on the content of the document to collectively, tgenerate the.
reference documents. Guided review can he performed by a reviewer, a machine, car a combination of the rev iec per and machine, Other methods for generating reference documents for a document review project using guided review are possible, including clustering. A set of uracoded documents to be classified. is clustered, as described. in c:omnaonly-assigned ..S. Patent No, 7,610,313. the disclosure of which 'is incorporated by reference. A plurality of the clustered uncoded documents are selected based on selection criteria, such as cluster centers car sample clusters. The cluster centers can be used, to identify taracoded documents in a cluster that are most similar or dissimilar to the cluster center, The selected uracoded documents are then assigned classification codes. fn a .f ur-ther embodiment, sample clusters can. he used to i eneraate reference documents by selecting one or more sample clusters based on cluster relation criteria, such as sire, content, sin, ii1arity, or dissimilarity. The uncoded documents in the selected sample clusters are then selected for classification by assigning classification codes. The classified documents represent reference documents for the document re-,.view prcject, The number ofrefercnce documents can be determined automatically or by :a reviewwer. Other methods for selecting documents for use as reference documents are possible.
An encoded document selected from one of the clusters can be compared. to the reference documents to identify simi.laar reference documents for use in. providing suggestions regarding classification of the selected cancoded document. FIGURE 4 is a process flow diagram showing, by way of example, a method. 6t) .for comparing an uncoded document to reference documents for use in the t:-rtethod. of FIGURE 2. The encoded docttmertt is selected front a cluster (block 61) and applied to the reference documents (block 62). 'f'lee reference documents can include all reference docurnents.for a document review project or a subset of the reference documents. Each .
of the reference documents and the selected uncoded docurnent can be represented by a score vector having paired values of tokens occurring within. that document and associated token scores. A similarity betty een the uncoded document and each reference document is determined (block 63) as the cos cr of the score vectors for the uncoded document and reference document being compared. and. is equivalent to the inner product between the score vectors. In the described embodiment, the cos (5 is calculated in accordance with tile equation:.

cos [7 -----'-------------where Cosa, , comprises a similarity between encoded document ..A: and reference document R, comprises a score vector for uricoded document A. and 5', comprises a score vector for reference document .t3. Other .forms ofcleteimini.n similarity using a distance metric are possible, as would be recognized by one skilled in the art, including using Euclidean distance.
1 One or more of the reference docurnents that are most similar to the selected encoded document, based on the similarity metric, are identified. The most similar reference documents can be identified by satisfying a predetermined threshold of similarity. Other met-ltods for determining the similar reference documents are possible, such as setting a predetermined.
absolute number of the most similar reference documents. The classification codes of the identified similar reference documents can be used as suggestions for classifying the selected encoded document, as further described below with reference to FIGURE 5. Once identified, the similar reference documents can be used to provide suggestions regarding classification of the selected uncoded document, as .further described below with reference to FIGURES 5 and '7.
The similar reference documents can. be displayed with the clusters of uncoded documents. In the display, the similar reference documents can be provided as a. list, while The clusters can be can be organi/ed along spines of thematically related clusters, as described in coin monle assigned U.S. Patent No, 7,217 1,804, the disclosure of which is incorporated by reference. The spines can be positioned in relation to other cluster spines based on a, theme shared by those cluster spines, as described in commonly-assigned US, Patent No. 7,610,_113, the disclosure of which is incorporated by reference. Other displays of the clusters and similar reference documents are possible.

Organizing the clusters into spines and groups of cluster spines provides an individual reviewer with a display that presents the documents according to a die-me while rtraximi:ing the number of relationships depicted between the documents. FIGURE 5 is a screenshot 70 showing. by way of example, a visual display 7.1 of similar reference documents 74 and. uncoded documents 74. Clusters 72 of the encoded documents 73 can be located along a spine, w Which is a vector, based on a similarity of the uncoded documents 73 in the clusters 72.
The encoded documents 73 are each represented by a smaller circle within the clusters 72.
Similar reference documents 74 identified fora selected uncode d. document 73 can be.
displayed in a list 75 by document title or other identifer. Also, classification codes 76 associated with the similar reference documents 74 can be displayed as circles having a diamond shape within the boundary of the circle. The classification codes 76 can include "privileged,"
"`responsive," and "non-responsive" codes, as well as other codes. "1'lre Jiff rent classification codes 76 can each lie represented by a color, such as blue for "privileged"
reference documents arrcl ello~ fcrr `ricrrr-resl onsi4e" reference docrtrirents. Other display representations of the uncoded documents, similar reference docunments, and classification codes are possible, including by syrribols and shapes.
The classification codes 76 of the similar reference documents 7 4 can provide suggestions for classifying the selected encoded document based oil factors.
such as a number of different classification codes for the similar reference documents and. a number of similar reference documents associated with each classification code. For example, the list of reference documents incItides four similar reference documents identified for a particular uncoded document. Three of the reference documents are classified as "privileged,"
while one is classified as "r}on-responsive. In making a decision to assign a classification code to a selected uncoded docu mient, the reviewer can consider classification factors based on the similar reference documents, such as such as a presence or absence of similar reference documents with different classification codes and. a quantity of the similar reference documents for each classification code. Other classification factors are possible. In the current example, the display 81 provides suggestions, including the number of "privileged" similar reference documents, the number of "non-responsive", similar reference documents, and the absence of other classification codes of similar reference doctrrmtents. Based an the number of privileged"
similar reference documents compared to the member of "non-responsive" similar reference documents, the reviewer ma be.more inclined to classify the selected uncoded documents as "privileged.."
Alternatively, the reviewer may wish to further review the selected uncoded document based. on the multiple classification codes of the similar reference documents. Other classification codes and combinations of classification codes are possible, The reviewer can utilize the suggestions provided by the similar reference documents to assign a classification to the selected encoded document.. In a further embodin-rernt, the now classified and previously unaided document can be added to the set of reference documents for use in classifying other uncoded documents.
S In a further embodiment, similar reference documents can be identified for a cluster or spine to provide suggestions for classifying the cluster and spine. Fora cluster, the similar reference documents are identified based on a comparison of a score vector for the cluster, lvhi:ch is representat.i e of the cluster center and the reference document score vecto s. Meanwhile, identifying similar reference documents for a spine is based on a comparison between the score vector for the spine, which is based on the cluster center of all the clusters along that spine, and the reference document score vectors. Once identified., the sir:_umilar reference documents are used for classifying the cluster or spine.
In an even further embodiment, the rancoded documents, including the selected uncoded document, and the similar reference documents can be displayed as a document list. FIGURE, 6 is a screerishot W showing, by way of example, an alternative visual display of the similar reference documents 85 and u ncoded documents 82. `I`he encoded documents 82 can be provided as a list in an uncoded. document box 81, such as an eni.ail inbox.
The encoded documents 82 can be identified and organized using uncoded document factors, such as file name., subject, date, recipient, sender, creator, and classification category $3, if previously assigned.
At least one of the uncoded documents can be selected and displayed in a document viewing box 84. The selected uncoded document can be identified in the list S
1 using a selection indicator (not shown), including a symbol, font. or highlighting. Other selection indicators and uncoded document factors are possible, Once identified, the selected uncoded document can be compared to a set of reference documents to identify the reference documents $5 most similar.
The identified similar reference documents 85 can be displayed below the document viewing box 84 with an associated classification code 83. The classification code of the similar reference document 85 can be used as a suggestion for classifying the selected uncoded document. After assigning a classification code, a representation 83 of the classification can be provided in the display with the selected encoded docunmrent:. In a further embodiment, the now classified and previously uncoded document can be added to the set of reference docu_rrmrent:s.
Similar reference documents can be used as suggestions to indicate a need -for manual review of the uncoded documents, when review may be unnecessary, and hints for classifying the encoded documents, clusters, or spines. Additional information can be generated to assist a reviewer in making classification decisions for the unc oded. documents.. such as a machine-generated confidence level associated wit hr a suggested classification code, as described in cornnaon assigned i..S. Patent Application Serial No. 12 833,7$9, entitled "Systeni and Method for Providing a Classification Suggestion for Electronically Stored inform atioi " filed on July 9, 2010, pending, the disclosure of which is incorporated by reference.
The machine-generated suggestion for classification and associated confidence level can be determined by a classifier. RG RE 7 is it process flow diagram 90 showing, by a -ay of exan:tpie, a method Ãbr classifying uncoded docu-inenÃs by a classifier for use in. the method of FIGURE 2, An uncoded document is selected front a chaster (block 91) and compared to a neighborhood of x-similar reference documents (block- 92) to identif ! those similar reference documents that are. most relevant to the selected uncoded document. The selected uncoiled document can be the same as the ancoded. document selected for identifying similar reference documents or a different uncoded document. In a further embodiment. a machine-generated suggestion can be provided fora. cluster or spine by selecting and comparing the cluster or spine to a neighborhood of x--reference documents for the. cluster or spine.
The neighborhood of x-similar reference documents is determined separately for each ,selected uncoded document and can include one or more shnilar reference docume its. During neighborhood Ã eneration, a value for x similar reference documents is first determined automatically or by an individual reviewer. The neighborhood of similar reference documents can include the reference documents, which were identified as similar reference documents according to the method of FIGURE 4, or reference documents located in one or more clusters, such as the satire cluster as the selected. uncoded document or in one or more files, such as an email file. Next., the x-number of similar reference documents nearest to the selected uncoded document. are identified. Finally, the identified _x-number of similar reference documents are provided as the neighborhood for the selected. uncoded document. In a further embodiment., the x-number of similar reference documents are, defined for each classification code, rather- than all classification codes. Once generated, the.a-number of similar reference documents in across the neighborhood and the selected uncoded document are analyzed by the classifier to provide a machine-generated classification suggestion for assigning a classification code (block 93), A
confidence level for the machine-generated classification suggestion is also provided (block 94).
The machine- eneratecl analys s of the selected uncoded document and x-nunal er of similar reference documents can be based on one or more routines per-rori-ned by the classifier, such as at nearest neighbor (NN) classifier. 'T'he routines for determining a.
suggested classification code include a minimum distance classification measure, also known as closest neighbor, minimum average distance classification measure., raxrrrattaal count classification measure, and distance weighted ::niaxintum coulat cl<assf f caiion measure.
The mini mane distance classification measure for a selected uncoded document includes ideartifying a neighbor that is the closest distance to the elected uncoiled doca. meat and assigning the classification code of the closest neighbor as the suggested classification code for the selected uncoded document:. The closest neighbor is determined by comparing the score vectors for the selected uncoded document with each of the -number o.f simmmilar x Cererice documents in the neighborhood as the cos (5 to determine a distance metric. The distance metrics for the x-rnutnber of sintila reference documents are compared to identi..f the similar reference document closest to the selected uncoded document as the closest rneighbor.
The minimum. average distance classification measure includes calculating an average distance of the similar reference documents for each classification code. The classification code of the similar reference documents having the closest average distance to the selected aricoded document is assigned as the suggested classification code. .the 3'na' imum fount classification measure, also I nowwwn as the voting classification measure. includes counting a number of similar reference documents for each classification code and assigtning a count or "vote" to the similar reference documents based on the. assigned classification code. The classification code with the highest number of similar reference documents or "votes" is assigned to the selected uncoded document as the suggested classification cede. The distance weighted maximum count 2t1 classification .meastr.re includes identifying a count of all similar reference documents for each classification code and. determining a distance between the selected uuncoded document and each.
of the similar reference documents. Each count assigned to the similar reference documents is weight l based on the distance of the similar reference document from the sselected encoded document.. The classification code with the highest count, after consideration of the weight, .is 2 5 assigned to the selected uncoded document as the suggested classification code.
The i rachiare-generated suggested classification code is provided for the selected u ncocled document with a confidence le el, which can he presented as an absolute value or a percentage.
Other confidence level measures are possible. The reviewer can use the suggested classification code and confidence level to assign a classification to the selected uncoiled document.
0 Alternatively, the t-Nsl classifier can tiart.omat:icrall uassign the suggested classification code. In one emhod:ina.ent, the x -NN classifier only assigns an uncoded document with the suggested classification code if the confidence level is above a threshold value, which can. be set. by the reviewer or the x-NN classifier.

Machine classification can also occur on a cluster or spine level once one or more documents in the cluster have been classified. For instance, for cluster classification, a cluster is selected and a score vector for the center of the cluster is deternuned as described above "%,-ith reference to FIGURE 4. A neighborhood .for the selected cluster can be determined based on a distance metric. The x-r?umber of similar reference documents that are closest to the cluster center can be selected for inclusion in the neighborhood, as described above.
Each document in the selected cluster is associated with a score vector from w hich the cluster center score vector is generated. The distance is then deter? ?mined by comparing the score Vector of the cluster Center with the score vector for each of the similar reference documents to determine an x-number of similar reference documents that are closest to the cluster center, However, other methods for i enerating a neighborhood are possible. Once determined, one of the classification routines is applied to the neighborhood to determine a suggested classification code and confidence 14 el for the selected cluster. The neighborhood of x -number ofref"erence documents is determined for aspire by comparing a spine score vector with the vector for each similar reference document to identify the neighborhood of similar documents that are the most similar.
Providing classification suggestions and suggested classification codes has been described in relation to uncoded documents and reference docu.merits. However, in a further embodiment, classification suggestions and suggested classification codes can be provided for the tt:ncoded. documents based on a particular token identified within the unco=ded documents.
?fl The token can include concepts, r `,grams, rmv terms. and entities, In one example, the encoded tokens, which are extracted .front encoded doctiments, can be clustered...
token can be selected from one of the clusters and compared with refernce tokens. Relationships between the urtcoded token and similar reference tokens can'be displayed to provide classification suggestions for the uncoded token. The uncoded doc.unrents can then be classified based on the classified tokens.
While the invention has been particularly shown and described as referenced to the embodiments thereof-, those skilled in the art will understand that the &r-egoing and other changes in formi and detail may be made therein without departing from the spirit and scope.

Claims

1. A system (11) for providing reference documents (14b) as a suggestion for classifying electronically stored information (13) using nearest neighbor, comprising:
a clustering module (33) to provide reference electronically stored information items (14b) previously classified and a set of uncoded electronically stored information items (14a);
a similarity module (34) to compare at least one uncoded electronically stored information item (14a) with the reference electronically stored information items (14b) and to identify one or more of the reference electronically stored information items (14b) that are similar to the at least one uncoded electronically stored information item (14a); and a display (37) to depict relationships between the at least one uncoded electronically stored information item (14a) and the similar reference electronically stored information items (14b) for classifying the at least one uncoded electronically stored information item (14a).

2. A system (11) according to Claim 1, further comprising:
a reference set module (36) to generate the set of reference electronically stored information items (14b), comprising at least one of:
a comparison module to obtain a set of electronically stored information items (14a), to identify one or more electronically stored information items (14a) that are dissimilar from each other electronically stored information item, and to assign a classification code to each of the dissimilar electronically stored information items (14a), as the reference electronically stored information items (14b); and a reference clustering module to group electronically stored information items (14a) for a document review project into one or more clusters (72), to select one or more of the electronically stored information items (14a) in at least one cluster (72), and to assign a classification code (76) to each of the selected electronically stored information items (14a), as the reference electronically stored information items (14b).

3. A system (11) according to Claim 1, further comprising:

a score module to form a score vector for each uncoded electronically stored information item (14a) and each reference electronically stored information item (14b); and the similarity module to calculate the similarity metric by comparing the score vectors for the uncoded electronically stored information items (14a) and the reference electronically stored information items (14b).

4. A system (11) according to Claim 1, wherein the similarity metric is calculated according to the following equation:
where cos.sigma.AB comprises a similarity between uncoded electronically stored information item A and reference electronically stored information item B, ~A
comprises a score vector for uncoded electronically stored information item A, and ~B comprises a score vector for reference electronically stored information item B.

5. A system (11) according to Claim 1, further comprising:
a classification module to assign a classification code (76) to one or more of the uncoded electronically stored information items (14a).

6. A method (40) for providing reference documents (14b) as a suggestion for classifying electronically stored information (13) using nearest neighbor, comprising:
designating reference electronically stored information items (14b), which are previously classified, and a set of uncoded electronically stored information items (14a);
comparing at least one uncoded electronically stored information item (14a) with the reference electronically stored information items (14b);
identifying one or more of the reference electronically stored information items (14b) that are similar to the at least one uncoded electronically stored information item (14a); and depicting relationships between the at least one uncoded electronically stored information item (14a) and the similar reference electronically stored information items (14b) for classifying the at least one uncoded electronically stored information item (14a).

7. A method (40) according to Claim 6, further comprising:
generating the reference electronically stored information items (14b) from a set of electronically stored information items (14a), comprising at least one of:
identifying the electronically stored information items (14a) that are dissimilar from each other electronically stored information item and assigning a classification code (76) to each of the dissimilar electronically stored information items (14a), as the reference electronically stored information items (14b); and grouping a set of electronically stored information items (14a) associated with a document review project into one or more clusters (72), selecting one or more of the electronically stored information items (14a) in at least one cluster (72), and assigning a classification code (76) to each of the selected electronically stored information items (14a), as the reference electronically stored information items (14b).

8. A method (40) according to Claim 6, further comprising:
forming a score vector for each uncoded electronically stored information item (14a) and each reference electronically stored information item (14b); and calculating a similarity metric by comparing the score vectors for the uncoded electronically stored information items (14a) and the reference electronically stored information items (14b) in the reference set.

9. A method (40) according to Claim 8, wherein the similarity metric is calculated according to the following equation:
where cos.sigma.AB comprises a similarity between uncoded electronically stored information item A and reference electronically stored information item B, ~A
comprises a score vector for uncoded electronically stored information item A, and ~B comprises a score vector for reference electronically stored information item B.

10. A method (40) according to Claim 6, further comprising:
assigning a classification code (76) to one or more of the uncoded electronically stored information items (14a) based on the similar reference electronically stored information items (14b).

11. A system (11) for identifying reference documents (14b) for use in classifying uncoded documents (14a), comprising:
a database (30) to store a set of reference documents (14b) each associated with a classification code (76);
a clustering module (33) to designate a set of clusters (72) each comprising uncoded documents (14a);
a similarity module (34) to select at least one uncoded document (14a), to compare the at least one uncoded document (14a) with each of the reference documents (14b), and to identify one or more reference documents (14b) that satisfy a threshold of similarity with the at least one uncoded document (14a);
and a display (37) to display relationships between the at least one uncoded document (14a) and the similar reference documents (14b) based on the associated classification codes (76) as suggestions for classifying the at least one uncoded document (14a).

12. A system (11) according to Claim 11, further comprising:
a reference set module to generate the set of reference documents (14b), comprising at least one of:
a reference similarity module to obtain a set of documents (14b), to identify the documents (14b) that are dissimilar from each other document (14a), and to assign a classification code (76) to each of the dissimilar documents (14b); and a reference cluster module to generate clusters (72) of documents (14b) for a document review project, to select one or more of the documents (14b) in at least one of the clusters (72), and to assign a classification code (76) to each of the documents (14b).

13. A system (11) according to Claim 11, wherein the one or more reference documents (14b) are selected from at least one of a predefined, customized, or arbitrary reference document set.

14. A system (11) according to Claim 11, wherein the similarity module comprises:
a score module to form a score vector for each uncoded document (14a) and each reference document (14b); and a vector similarity module to calculate a similarity metric between the score vectors for the uncoded documents (14a) and reference documents (14b).

15. A system (11) according to Claim 11, wherein the similarity metric is calculated according to the following equation:
where cos.sigma.AB comprises a similarity between uncoded document A and reference document B, ~A comprises a score vector for uncoded document A, and ~B comprises a score vector for reference document B.

16. A method (40) for identifying reference documents (14b) for use in classifying uncoded documents (14a), comprising:
designating a set of reference documents (14b) each associated with a classification code (76);
designing a set of clusters (72) each comprising uncoded documents (14a);

selecting at least one uncoded document (14a) and comparing the at least one uncoded document (14a) with each of the reference documents (14b);
identifying one or more reference documents (14b) that satisfy a threshold of similarity with the at least one uncoded document (14a); and displaying relationships between the at least one uncoded document (14a) and the similar reference documents (14b) based on the associated classification codes (76) as suggestions for classifying the at least one uncoded document (14a).

17. A method (40) according to Claim 16, further comprising:
generating the set of reference documents (14b), comprising at least one of:
obtaining a set of documents (14b), identifying the documents (14b) that are dissimilar from each other document (14a), and assigning a classification code (76) to each of the dissimilar documents (14b); and generating clusters (72) of documents (14b), selecting one or more of the documents (14b) in at least one of the clusters (72) and assigning a classification code (76) to each of the documents (14b).

18. A method (40) according to Claim 16, wherein the one or more reference documents (14b) are selected from at least one of a predefined, customized, or arbitrary reference document set.

19. A method (40) according to Claim 16, further comprising:
forming a score vector for each uncoded document (14a) and each reference document (14b); and calculating a similarity metric between the score vectors for the uncoded documents (14a) and reference documents (14b).

20. A method (40) according to Claim 16, wherein the similarity metric is calculated according to the following equation:

21 where cos.sigma.AB comprises a similarity between uncoded document A and reference document B, ~A comprises a score vector for uncoded document A, and ~B comprises a score vector for reference document B.

22