CA2723167A1 - Database systems and methods for linking records corresponding to the same individual - Google Patents

Database systems and methods for linking records corresponding to the same individual Download PDF

Info

Publication number
CA2723167A1
CA2723167A1 CA2723167A CA2723167A CA2723167A1 CA 2723167 A1 CA2723167 A1 CA 2723167A1 CA 2723167 A CA2723167 A CA 2723167A CA 2723167 A CA2723167 A CA 2723167A CA 2723167 A1 CA2723167 A1 CA 2723167A1
Authority
CA
Canada
Prior art keywords
record
field
database
field value
records
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CA2723167A
Other languages
French (fr)
Other versions
CA2723167C (en
Inventor
David Bayliss
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
LexisNexis Risk Solutions FL Inc
Original Assignee
LexisNexis Risk and Information Analytics Group Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by LexisNexis Risk and Information Analytics Group Inc filed Critical LexisNexis Risk and Information Analytics Group Inc
Priority to CA2749310A priority Critical patent/CA2749310C/en
Priority to CA2749080A priority patent/CA2749080C/en
Publication of CA2723167A1 publication Critical patent/CA2723167A1/en
Application granted granted Critical
Publication of CA2723167C publication Critical patent/CA2723167C/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/28Databases characterised by their database models, e.g. relational or object models
    • G06F16/284Relational databases
    • G06F16/285Clustering or classification
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/26Visual data mining; Browsing structured data
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/21Design, administration or maintenance of databases
    • G06F16/215Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/22Indexing; Data structures therefor; Storage structures
    • G06F16/2228Indexing structures
    • G06F16/2246Trees, e.g. B+trees
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24Querying
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24Querying
    • G06F16/245Query processing
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24Querying
    • G06F16/245Query processing
    • G06F16/2455Query execution
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/28Databases characterised by their database models, e.g. relational or object models
    • G06F16/282Hierarchical databases, e.g. IMS, LDAP data stores or Lotus Notes
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/30Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
    • G06F16/35Clustering; Classification
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90Details of database functions independent of the retrieved data types
    • G06F16/95Retrieval from the web
    • G06F16/951Indexing; Web crawling techniques

Abstract

Disclosed is a system for, and method of, calculating parameters used to determine whether records and entity representations should be linked. The system and method apply iterative techniques such that parameters from each linking iteration are used in the next linking iteration. The system and method need no human interaction in order to calibrate and utilize record matching formulas used for the linking decisions.

Claims (184)

1. A computer implemented method of linking a first record in a database to a second record in a database upon a determination that the first record and the second record correspond to the same individual, the method comprising:
calculating a plurality of match probabilities using an iterative process, each of the plurality of match probabilities corresponding to a different field common to the first record and the second record;
calculating a match score based on a plurality of terms, each of the plurality of terms corresponding to a different field common to the first record and the second record, each of the plurality of terms comprising: (1) a probability that a field value in a corresponding field of the first record matches a field value in a corresponding field in the second record, and (2) a quantity based on a corresponding one of the plurality of match probabilities;
determining, based on the match score and a size of a population associated with the database, whether there is a sufficiently high confidence level that the first record and the second record correspond to the same individual; and linking, in the database, the first record with the second record based on the determining.
2. The method of claim 1, wherein the determining comprises determining whether the match score exceeds a quantity derived from a threshold probability and the size of the population associated with the database.
3. The method of claim 1 or 2, wherein the size of the population associated with the database is a number of individuals intended to be reflected in the database.
4. The method of claim 1 or 2, wherein the size of the population associated with the database is a number of entity representations in the database.
5. The method of claim 1 or 2, wherein the size of the population associated with the database is a number of records in the database.
6. The method of claim 2 wherein the quantity derived from the threshold probability and the size of the population associated with the database comprises a first logarithm and a second logarithm, wherein an argument of the first logarithm comprises a desired confidence probability and an argument of the second logarithm comprises the size of the population associated with the database.
7. The method of claim 6 wherein the quantity derived from the threshold probability and the size of the population associated with the database comprises log(N) - log(1-P) - 1, wherein P represents the threshold probability and N represents the size of the population associated with the database.
8. The method of claim 1 wherein the plurality of match probabilities comprise at least one field probability.
9. The method of claim 1 wherein the plurality of match probabilities comprise at least one field value probability.
10. The method of claim 1 wherein the match score comprises a weighted sum of probabilities that field values in the first record match field values in the second record, wherein the sum is weighted by match weights derived from the match probabilities using logarithms.
11. The method of claim 1 wherein the calculating the match score comprises calculating wherein p1 is a probability that a field value in a corresponding field of the first record matches a field value in a corresponding field in the second record, w1 is a negative logarithm of a corresponding one of the plurality of match probabilities, and I enumerates at least some fields common to the first record and the second record.
12. A system for linking a first record in a database to a second record in a database upon a determination that the first record and the second record correspond to the same individual, the system comprising:
a first computing apparatus configured to calculate a plurality of match probabilities using an iterative process, each of the plurality of match probabilities corresponding to a different field common to the first record and the second record;
a second computing apparatus configured to calculate a match score based on a plurality of terms, each of the plurality of terms corresponding to a different field common to the first record and the second record, each of the plurality of terms comprising: (1) a probability that a field value in a corresponding field of the first record matches a field value in a corresponding field in the second record, and (2) a quantity based on a corresponding one of the plurality of match probabilities;
a third computing apparatus configured to determine, based on the match score and a size of a population associated with the database, whether there is a sufficiently high confidence level that the first record and the second record correspond to the same individual;
and a fourth computing apparatus configured to link, in the database, the first record with the second record based on the determining.
13. The system of claim 12, wherein the third computing apparatus is configured to determine whether the match score exceeds a quantity derived from a threshold probability and the size of the population associated with the database.
14. The system of claim 12 or 13, wherein the size of the population associated with the database is a number of individuals intended to be reflected in the database.
15. The system of claim 12 or 13, wherein the size of the population associated with the database is a number of entity representations in the database.
16. The system of claim 12 or 13, wherein the size of the population associated with the database is a number of records in the database.
17. The system of claim 13 wherein the quantity derived from the threshold probability and the size of the population associated with the database comprises a first logarithm and a second logarithm, wherein an argument of the first logarithm comprises a desired confidence probability and an argument of the second logarithm comprises the size of the population associated with the database.
18. The system of claim 17 wherein the quantity derived from the threshold probability and the size of the population associated with the database comprises log(N) - log(1-P) - 1, wherein P represents the threshold probability and N represents the size of the population associated with the database.
19. The system of claim 12 wherein the plurality of match probabilities comprise at least one field probability.
20. The system of claim 12 wherein the plurality of match probabilities comprise at least one field value probability.
21. The system of claim 12 wherein the match score comprises a weighted sum of probabilities that field values in the first record match field values in the second record, wherein the sum is weighted by match weights derived from the match probabilities using logarithms.
22. The system of claim 12 wherein the second computer is configured to calculate the match by calculating , wherein p1 is a probability that a field value in a corresponding field of the first record matches a field value in a corresponding field in the second record, w1 is a negative logarithm of a corresponding one of the plurality of match probabilities, and I enumerates at least some fields common to the first record and the second record.
23. The system of claim 12, wherein the first computing apparatus, the second computing apparatus, the third computing apparatus and the fourth computing apparatus are the same computing apparatus.
24. The system of claim 12, wherein the first computing apparatus comprises a plurality of processors operating in parallel.

(II.)
25. A computer implemented iterative process for generating entity representations in a computer implemented database using a record matching formula and for generating parameters for the record matching formula, the database comprising a plurality of records, each record comprising a plurality of fields, each field capable of containing a field value, wherein at least a portion of parameters for the record matching formula are configured for a particular field value appearing in a selected field of at least one record, the process comprising:
calculating a field value weight, the field value weight reflecting a likelihood that an arbitrary record in the database comprises the particular field value in the selected field of the arbitrary record;
forming a plurality of entity representations in the database, each entity representation comprising at least two records linked using a first instance of the record matching formula, at least one entity representation comprising a first record comprising the particular field value, the first record linked to a second record using a first instance of the record matching formula comprising the field value weight;
calculating a revised field value weight, the revised field value weight reflecting a likelihood that an arbitrary entity representation in the database comprises the particular field value in the selected field of a record in the arbitrary entity representation;
linking at least two entity representations in the database based on a second instance of the record matching formula, wherein the second instance of the record matching formula comprises the revised field value weight, whereby a number of entity representations in the database is reduced by the linking at least two entity representations relative to a number of entity representations in the database prior to the linking at least two entity representations; and retrieving information from at least one record in the database.
26. The method of claim 25, further comprising iterating the calculating a revised field value weight and the linking at least two entity representations at least once prior to the retrieving.
27. The method of claim 25, wherein the record matching formula comprises a weighted sum of probabilities that two records match.
28. The method of claim 25, wherein the field value weight comprises a logarithm of a probability that an arbitrary record in the database comprises the particular field value in the selected field of the arbitrary record, and wherein the revised field value weight comprises a logarithm of a probability that an arbitrary entity representation in the database comprises the particular field value in the selected field of a record in the arbitrary entity representation.
29. A computer implemented iterative process for generating entity representations in a computer implemented database using a record matching formula and for generating parameters for the record matching formula, the database comprising a plurality of records, each record comprising a plurality of fields, each field capable of containing a field value, wherein at least a portion of parameters for the record matching formula are configured for a selected field and independent of any field value in the selected field, the process comprising:
calculating a plurality of field value weights, each field value weight reflecting a likelihood that an arbitrary record in the database comprises a different field value in the selected field of the arbitrary record;
calculating a field weight for the selected field, the field weight for the selected field derived from each of the plurality of field value weights;
forming a plurality of entity representations in the database, each entity representation comprising at least two records linked using a first instance of the record matching formula, at least one entity representation comprising a first record linked to a second record using a first instance of the record matching formula comprising the field weight for the selected field;
calculating a plurality of revised field value weights, each revised field value weight reflecting a likelihood that an arbitrary entity representation in the database comprises a different field value in the selected field of a record in the arbitrary entity representation;
calculating a revised field weight for the selected field, the revised field weight for the selected field derived from each of the plurality of revised field value weights;
linking at least two entity representations in the database based on a second instance of the record matching formula, wherein the second instance of the record matching formula comprises the revised field value weight, whereby a number of entity representations in the database is reduced by the linking at least two entity representations relative to a number of entity representations in the database prior to the linking at least two entity representations; and retrieving information from at least one record in the database.
30. The method of claim 29 further comprising iterating the calculating the plurality of revised field value probabilities, the calculating a revised field weight, and the linking at least two entity representations at least once prior to the retrieving.
31. The method of claim 29, wherein the record matching formula comprises a weighted sum of probabilities that two records match.
32. The method of claim 29, wherein each field value weight comprises a logarithm of a probability that an arbitrary record in the database comprises a different particular field value in the selected field of the arbitrary record, and wherein each revised field value weight comprises a logarithm of a probability that an arbitrary entity representation in the database comprises a different particular field value in the selected field of a record in the arbitrary entity representation.
33. A computer system for iteratively generating entity representations in a computer implemented database using a record matching formula and for generating parameters for the record matching formula, the database comprising a plurality of records, each record comprising a plurality of fields, each field capable of containing a field value, wherein at least a portion of parameters for the record matching formula are configured for a particular field value appearing in a selected field of at least one record, the system comprising:
a computer implemented database comprising a plurality of records, each record comprising a plurality of fields, each field capable of containing a field value;
a processor programmed to calculate a field value weight, the field value weight reflecting a likelihood that an arbitrary record in the database comprises the particular field value in the selected field of the arbitrary record;
a processor programmed to form and store a plurality of entity representations in the database, each entity representation comprising at least two records linked using a first instance of the record matching formula, at least one entity representation comprising a first record comprising the particular field value, the first record linked to a second record using a first instance of the record matching formula comprising the field value weight;
a processor programmed to calculate a revised field value weight, the revised field value weight reflecting a likelihood that an arbitrary entity representation in the database comprises the particular field value in the selected field of a record in the arbitrary entity representation; and a processor programmed to link and store at least two entity representations in the database based on a second instance of the record matching formula, wherein the second instance of the record matching formula comprises the revised field value weight, whereby a number of entity representations in the database is reduced by linking the at least two entity representations relative to a number of entity representations in the database prior to the linking at least two entity representations.
34. The system of claim 33, further comprising program logic configured to repeat calculating a revised field value weight and linking at least two entity representations at least once.
35. The system of claim 33, wherein the record matching formula comprises a weighted sum of probabilities that two records match.
36. The system of claim 33, wherein the field value weight comprises a logarithm of a probability that an arbitrary record in the database comprises the particular field value in the selected field of the arbitrary record, and wherein the revised field value weight comprises a logarithm of a probability that an arbitrary entity representation in the database comprises the particular field value in the selected field of a record in the arbitrary entity representation.
37. A computer system for iteratively generating entity representations in a computer implemented database using a record matching formula and for generating parameters for the record matching formula, the database comprising a plurality of records, each record comprising a plurality of fields, each field capable of containing a field value, wherein at least a portion of parameters for the record matching formula are configured for a selected field and independent of any field value in the selected field, the system comprising:
a computer implemented database comprising a plurality of records, each record comprising a plurality of fields, each field capable of containing a field value;
a processor programmed to calculate a plurality of field value weights, each field value weight reflecting a likelihood that an arbitrary record in the database comprises a different field value in the selected field of the arbitrary record;
a processor programmed to calculate a field weight for the selected field, the field weight for the selected field derived from each of the plurality of field value weights;
a processor programmed to form and store a plurality of entity representations in the database, each entity representation comprising at least two records linked using a first instance of the record matching formula, at least one entity representation comprising a first record linked to a second record using a first instance of the record matching formula comprising the field weight for the selected field;
a processor programmed to calculate a plurality of revised field value weights, each revised field value weight reflecting a likelihood that an arbitrary entity representation in the database comprises a different field value in the selected field of a record in the arbitrary entity representation;

a processor programmed to calculate a revised field weight for the selected field, the revised field weight for the selected field derived from each of the plurality of revised field value weights; and a processor programmed to link and store at least two entity representations in the database based on a second instance of the record matching formula, wherein the second instance of the record matching formula comprises the revised field value weight, whereby a number of entity representations in the database is reduced by the linking at least two entity representations relative to a number of entity representations in the database prior to the linking at least two entity representations.
38. The system of claim 37 further comprising program logic configured to iterate calculating the plurality of revised field value probabilities, calculating a revised field weight, and linking and storing at least two entity representations at least once.
39. The system of claim 37, wherein the record matching formula comprises a weighted sum of probabilities that two records match.
40. The system of claim 37, wherein each field value weight comprises a logarithm of a probability that an arbitrary record in the database comprises a different particular field value in the selected field of the arbitrary record, and wherein each revised field value weight comprises a logarithm of a probability that an arbitrary entity representation in the database comprises a different particular field value in the selected field of a record in the arbitrary entity representation.

(III.)
41. A computer implemented iterative process for generating entity representations in a computer implemented database using a record matching formula and for generating parameters for the record matching formula, the database comprising a plurality of records, each record comprising a plurality of fields, each field capable of containing a field value, wherein at least a portion of parameters for the record matching formula are generated using a symmetric and reflexive function and configured for a particular field value appearing in a selected field of at least one record, and wherein the process provides for linking records or entity representations with non-identical field values, the process comprising:
selecting a distance;
calculating a field value weight, the field value weight reflecting a likelihood that an arbitrary record in the database comprises a field value in the selected field of the arbitrary record that lies within the distance of the particular field value as determined by the symmetric and reflexive function;
forming a plurality of entity representations in the database, each entity representation comprising at least two records linked using a first instance of the record matching formula, at least one entity representation comprising a first record comprising the particular field value, the first record linked to a second record using a first instance of the record matching formula comprising the field value weight;
calculating a revised field value weight, the revised field value weight reflecting a likelihood that an arbitrary entity representation in the database comprises a field value, in the selected field of a record in the arbitrary entity representation, that lies within the distance of the particular field value as determined by the symmetric and reflexive function;
linking at least two entity representations in the database based on a second instance of the record matching formula, wherein the second instance of the record matching formula comprises the revised field value weight, whereby a number of entity representations in the database is reduced by the linking at least two entity representations relative to a number of entity representations in the database prior to the linking at least two entity representations; and retrieving information from at least one record in the database.
42. The method of claim 41 further comprising iterating the calculating a revised field value weight and the linking at least two entity representations at least once prior to the retrieving.
43. The method of claim 41, wherein the record matching formula comprises a weighted sum of probabilities that two records match.
44. The method of claim 41, wherein the field value weight comprises a logarithm of a probability that an arbitrary record in the database comprises a field value in the selected field of the arbitrary record that lies within the distance of the particular field value as determined by the symmetric and reflexive function, and wherein the revised field value weight comprises a logarithm of a probability that an arbitrary entity representation in the database comprises a field value, in the selected field of a record in the arbitrary entity representation, that lies within the distance of the particular field value as determined by the symmetric and reflexive function.
45. The method of claim 41, wherein the symmetric, reflexive function is not an edit distance function.
46. A computer implemented iterative process for generating entity representations in a computer implemented database using a record matching formula and for generating parameters for the record matching formula, the database comprising a plurality of records, each record comprising a plurality of fields, each field capable of containing a field value, wherein at least a portion of parameters for the record matching formula are generated using a symmetric and reflexive function, configured for a particular field, and independent of any field value in the selected field, and wherein the process provides for linking records or entity representations with non-identical field values, the process comprising:
selecting a distance;
calculating a plurality of field value weights, each field value weight reflecting a likelihood that an arbitrary record in the database comprises a different field value, in the selected field of the arbitrary record, that lies within the distance of the field value as determined by the symmetric and reflexive function;
calculating a field weight for the selected field, the field weight for the selected field derived from each of the plurality of field value weights;
forming a plurality of entity representations in the database, each entity representation comprising at least two records linked using a first instance of the record matching formula, at least one entity representation comprising a first record linked to a second record using a first instance of the record matching formula comprising the field weight for the selected field;

calculating a plurality of revised field value weights, each revised field value weight reflecting a likelihood that an arbitrary entity representation in the database comprises a different field value, in the selected field of a record in the arbitrary entity representation, that lies within the distance of the particular field value as determined by the symmetric and reflexive function;
calculating a revised field weight for the selected field, the revised field weight for the selected field derived from each of the plurality of revised field value weights;
linking at least two entity representations in the database based on a second instance of the record matching formula, wherein the second instance of the record matching formula comprises the revised field value weight, whereby a number of entity representations in the database is reduced by the linking at least two entity representations relative to a number of entity representations in the database prior to the linking at least two entity representations; and retrieving information from at least one record in the database.
47. The method of claim 46 further comprising iterating the calculating a plurality of revised field value weights, the calculating a revised field weight, and the linking at least two entity representations at least once prior to the retrieving.
48. The method of claim 46, wherein the record matching formula comprises a weighted sum of probabilities that two records match.
49. The method of claim 46, wherein each field value weight comprises a logarithm of a probability that an arbitrary record in the database comprises a field value in the selected field of the arbitrary record that lies within the distance of the particular field value as determined by the symmetric and reflexive function, and wherein each revised field value weight comprises a logarithm of a probability that an arbitrary entity representation in the database comprises a field value, in the selected field of a record in the arbitrary entity representation, that lies within the distance of the particular field value as determined by the symmetric and reflexive function.
50. The method of claim 46, wherein the symmetric, reflexive function is not an edit distance function.
51. A computer system for iteratively generating entity representations in a computer implemented database using a record matching formula and for generating parameters for the record matching formula, the database comprising a plurality of records, each record comprising a plurality of fields, each field capable of containing a field value, wherein at least a portion of parameters for the record matching formula are generated using a symmetric and reflexive function and configured for a particular field value appearing in a selected field of at least one record, and wherein the system provides for linking records or entity representations with non-identical field values, the system comprising:
a computer implemented database comprising a plurality of records, each record comprising a plurality of fields, each field capable of containing a field value;
an electronic memory storing a selected distance;
a processor programmed to calculate a field value weight, the field value weight reflecting a likelihood that an arbitrary record in the database comprises a field value in the selected field of the arbitrary record that lies within the distance of the particular field value as determined by the symmetric and reflexive function;
a processor programmed to form and store a plurality of entity representations in the database, each entity representation comprising at least two records linked using a first instance of the record matching formula, at least one entity representation comprising a first record comprising the particular field value, the first record linked to a second record using a first instance of the record matching formula comprising the field value weight;
a processor programmed to calculate a revised field value weight, the revised field value weight reflecting a likelihood that an arbitrary entity representation in the database comprises a field value, in the selected field of a record in the arbitrary entity representation, that lies within the distance of the particular field value as determined by the symmetric and reflexive function;
and a processor programmed to link and store at least two entity representations in the database based on a second instance of the record matching formula, wherein the second instance of the record matching formula comprises the revised field value weight, whereby a number of entity representations in the database is reduced by the linking at least two entity representations relative to a number of entity representations in the database prior to the linking at least two entity representations.
52. The system of claim 51 further comprising program logic configured to repeat calculating a revised field value weight and linking at least two entity representations at least once.
53. The system of claim 51, wherein the record matching formula comprises a weighted sum of probabilities that two records match.
54. The system of claim 51, wherein the field value weight comprises a logarithm of a probability that an arbitrary record in the database comprises a field value in the selected field of the arbitrary record that lies within the distance of the particular field value as determined by the symmetric and reflexive function, and wherein the revised field value weight comprises a logarithm of a probability that an arbitrary entity representation in the database comprises a field value, in the selected field of a record in the arbitrary entity representation, that lies within the distance of the particular field value as determined by the symmetric and reflexive function.
55. The system of claim 51, wherein the symmetric, reflexive function is not an edit distance function.
56. A computer system for iteratively generating entity representations in a computer implemented database using a record matching formula and for generating parameters for the record matching formula, the database comprising a plurality of records, each record comprising a plurality of fields, each field capable of containing a field value, wherein at least a portion of parameters for the record matching formula are generated using a symmetric and reflexive function, configured for a particular field, and independent of any field value in the selected field, and wherein the process provides for linking records or entity representations with non-identical field values, the process comprising:
a computer implemented database comprising a plurality of records, each record comprising a plurality of fields, each field capable of containing a field value;
an electronic memory storing a selected distance;
a processor programmed to calculate a plurality of field value weights, each field value weight reflecting a likelihood that an arbitrary record in the database comprises a different field value, in the selected field of the arbitrary record, that lies within the distance of the field value as determined by the symmetric and reflexive function;
a processor programmed to calculate a field weight for the selected field, the field weight for the selected field derived from each of the plurality of field value weights;
a processor programmed to form and store a plurality of entity representations in the database, each entity representation comprising at least two records linked using a first instance of the record matching formula, at least one entity representation comprising a first record linked to a second record using a first instance of the record matching formula comprising the field weight for the selected field;
a processor programmed to calculate a plurality of revised field value weights, each revised field value weight reflecting a likelihood that an arbitrary entity representation in the database comprises a different field value, in the selected field of a record in the arbitrary entity representation, that lies within the distance of the particular field value as determined by the symmetric and reflexive function;
a processor programmed to calculate a revised field weight for the selected field, the revised field weight for the selected field derived from each of the plurality of revised field value weights; and a processor programmed to link and store at least two entity representations in the database based on a second instance of the record matching formula, wherein the second instance of the record matching formula comprises the revised field value weight, whereby a number of entity representations in the database is reduced by the linking at least two entity representations relative to a number of entity representations in the database prior to the linking at least two entity representations.
57. The system of claim 56 further comprising program logic configured to iterate calculating the plurality of revised field value probabilities, calculating a revised field weight, and linking and storing at least two entity representations at least once.
58. The system of claim 56, wherein the record matching formula comprises a weighted sum of probabilities that two records match.
59. The system of claim 56, wherein each field value weight comprises a logarithm of a probability that an arbitrary record in the database comprises a field value in the selected field of the arbitrary record that lies within the distance of the particular field value as determined by the symmetric and reflexive function, and wherein each revised field value weight comprises a logarithm of a probability that an arbitrary entity representation in the database comprises a field value, in the selected field of a record in the arbitrary entity representation, that lies within the distance of the particular field value as determined by the symmetric and reflexive function.
60: The system of claim 56, wherein the symmetric, reflexive function is not an edit distance function.

(IV.)
61. A computer implemented iterative process for generating entity representations in a computer implemented database using a record matching formula and for generating parameters for the record matching formula, the database comprising a plurality of records, each record comprising a plurality of fields, each field capable of containing a field value, wherein at least a portion of parameters for the record matching formula are configured for a particular field value associated with a selected field, and wherein the process provides for linking records or entity representations with non-identical field values, the process comprising:
applying a symmetric, reflexive and transitive function to each field value in the selected field of each of a plurality of records in the database, whereby applying the symmetric, reflexive and transitive function to each field value in the selected field of each of a plurality of records in the database defines a partition of the plurality of records, wherein the partition of the plurality of records comprises a plurality of parts, each of the parts associated with at least one field value appearing in the selected field;
calculating a first probability, the first probability reflecting a likelihood that an arbitrary record in the database is in a part associated with the particular field value;
forming a plurality of entity representations in the database, each entity representation comprising at least two records linked using a first instance of the record matching formula, at least one entity representation comprising at least two records linked using a first instance of the record matching formula that comprises a first parameter derived from the first probability;
calculating a second probability, the second probability reflecting a likelihood that an arbitrary entity representation in the database comprises a record that is in the part associated with the particular field value;
linking at least two entity representations in the database based on a second instance of the record matching formula, wherein the second instance of the record matching formula comprises a second parameter derived from the second probability, whereby a number of entity representations in the database is reduced by the linking entity representations relative to a number of entity representations in the database prior to the linking at least two entity representations; and retrieving information from at least one record in the database.
62. The process of claim 61 further comprising repeating the calculating a second probability and the linking at least two entity representations at least once prior to the retrieving.
63. The process of claim 61, wherein the record matching formula comprises a weighted sum of probabilities that two records match.
64. The process of claim 61, wherein the first parameter comprises a logarithm of the first probability and wherein the second parameter comprises a logarithm of the second probability.
65. A computer implemented iterative process for generating entity representations in a computer implemented database using a record matching formula and for generating parameters for the record matching formula, the database comprising a plurality of records, each record comprising a plurality of fields, each field capable of containing a field value, wherein at least a portion of parameters for the record matching formula are configured for a selected field and independent of any particular field value in the selected field, and wherein the process provides for linking records and entity representations with non-identical field values, the process comprising:
applying a symmetric, reflexive and transitive function to each field value in the selected field of each of a plurality of records in the database, whereby applying the symmetric, reflexive and transitive function to each field value in the selected field of each of a plurality of records in the database defines a partition of the plurality of records, wherein the partition of the plurality of records comprises a plurality of parts, each of the parts associated with at least one field value appearing in the particular field;
calculating a first plurality of first probabilities, each of the plurality of first probabilities reflecting a likelihood that an arbitrary record in the database is in a different first part;
calculating a first parameter derived from a weighted sum of the plurality of first probabilities;
forming a plurality of entity representations in the database, each entity representation comprising at least two records linked using a first instance of the record matching formula, at least one entity representation comprising at least two records linked using a first instance of the record matching formula that comprises the first parameter;
calculating a second plurality of second probabilities, each of the second plurality of second probabilities reflecting a likelihood that an arbitrary entity representation in the database comprises a record that is in a different part;
calculating a second parameter derived from a weighted sum of the second plurality of second probabilities;
linking at least two entity representations in the database based on a second instance of the record matching formula, wherein the second instance of the record matching formula comprises the second parameter, whereby a number of entity representations in the database is reduced by the linking entity representations relative to a number of entity representations in the database prior to the linking at least two entity representations; and retrieving information from at least one record in the database.
66. The process of claim 65 further comprising repeating the calculating a second plurality of second probabilities, the calculating a second parameter, and the linking at least two entity representations at least once prior to the retrieving.
67. The process of claim 65, wherein the record matching formula comprises a weighted sum of probabilities that two records match.
68. The process of claim 65, wherein the first parameter comprises a weighted sum of logarithms of the first plurality of first probabilities and wherein the second parameter comprises a weighted sum of a logarithms of the second plurality of second probabilities.
69. A computer system -for iteratively generating entity representations in a computer implemented database using a record matching formula and for generating parameters for the record matching formula, the database comprising a plurality of records, each record comprising a plurality of fields, each field capable of containing a field value, wherein at least a portion of parameters for the record matching formula are configured for a particular field value associated with a selected field, and wherein the process provides for linking records or entity representations with non-identical field values, the system comprising:
a database comprising a plurality of records, each record comprising a plurality of fields, each field capable of containing a field value;
a processor programmed to apply a symmetric, reflexive and transitive function to each field value in the selected field of each of a plurality of records in the database, whereby applying the symmetric, reflexive and transitive function to each field value in the selected field of each of a plurality of records in the database defines a partition of the plurality of records, wherein the partition of the plurality of records comprises a plurality of parts, each of the parts associated with at least one field value appearing in the selected field;

a processor programmed to calculate a first probability, the first probability reflecting a likelihood that an arbitrary record in the database is in a part associated with the particular field value;
a processor programmed to form and store a plurality of entity representations in the database, each entity representation comprising at least two records linked using a first instance of the record matching formula, at least one entity representation comprising at least two records linked using a first instance of the record matching formula that comprises a first parameter derived from the first probability;
a processor programmed to calculate a second probability, the second probability reflecting a likelihood that an arbitrary entity representation in the database comprises a record that is in the part associated with the particular field value; and a processor programmed to link and store at least two entity representations in the database based on a second instance of the record matching formula, wherein the second instance of the record matching formula comprises a second parameter derived from the second probability, whereby a number of entity representations in the database is reduced by the linking entity representations relative to a number of entity representations in the database prior to the linking at least two entity representations.
70. The system of claim 69 further comprising program logic configured to repeat calculating a second probability and linking and storing at least two entity representations at least once.
71. The system of claim 69, wherein the record matching formula comprises a weighted sum of probabilities that two records match.
72. The system of claim 69, wherein the first parameter comprises a logarithm of the first probability and wherein the second parameter comprises a logarithm of the second probability.
73. A computer system for iteratively generating entity representations in a computer implemented database using a record matching formula and for generating parameters for the record matching formula, the database comprising a plurality of records, each record comprising a plurality of fields, each field capable of containing a field value, wherein at least a portion of parameters for the record matching formula are configured for a selected field and independent of any particular field value in the selected field, and wherein the process provides for linking records and entity representations with non-identical field values, the system comprising:
a database comprising a plurality of records, each record comprising a plurality of fields, each field capable of containing a field value;
a processor programmed to apply a symmetric, reflexive and transitive function to each field value in the selected field of each of a plurality of records in the database, whereby applying the symmetric, reflexive and transitive function to each field value in the selected field of each of a plurality of records in the database defines a partition of the plurality of records, wherein the partition of the plurality of records comprises a plurality of parts, each of the parts associated with at least one field value appearing in the particular field;
a processor programmed to calculate a first plurality of first probabilities, each of the plurality of first probabilities reflecting a likelihood that an arbitrary record in the database is in a different first part;
a processor programmed to calculate a first parameter derived from a weighted sum of the plurality of first probabilities;
a processor programmed to form and store a plurality of entity representations in the database, each entity representation comprising at least two records linked using a first instance of the record matching formula, at least one entity representation comprising at least two records linked using a first instance of the record matching formula that comprises the first parameter;
a processor programmed to calculate a second plurality of second probabilities, each of the second plurality of second probabilities reflecting a likelihood that an arbitrary entity representation in the database comprises a record that is in a different part;
a processor programmed to calculate a second parameter derived from a weighted sum of the second plurality of second probabilities; and a processor programmed to link and store at least two entity representations in the database based on a second instance of the record matching formula, wherein the second instance of the record matching formula comprises the second parameter, whereby a number of entity representations in the database is reduced by the linking entity representations relative to a number of entity representations in the database prior to the linking at least two entity representations.
74. The system of claim 73 further comprising program logic configured to repeat calculating a second plurality of second probabilities, calculating a second parameter, and linking and storing at least two entity representations at least once.
75. The system of claim 73, wherein the record matching formula comprises a weighted sum of probabilities that two records match.
76. The system of claim 73, wherein the first parameter comprises a weighted sum of logarithms of the first plurality of first probabilities and wherein the second parameter comprises a weighted sum of a logarithms of the second plurality of second probabilities.

(V.)
77. A computer implemented iterative process for generating entity representations in a computer implemented database using a record matching formula and for generating parameters for the record matching formula, the database comprising a plurality of records, each record comprising a plurality of fields, each field capable of containing a field value, wherein at least a portion of the parameters for the record matching formula are specific to a particular plurality of field values associated with a particular plurality of fields, the process comprising:
adding, in the database, a supplemental field to each of the plurality of records;
populating each supplemental field of each of the plurality of records with a supplemental field value, each supplemental field value representative of field values from the particular plurality of fields of that record;

calculating a plurality of supplemental field value weights, each supplemental field value weight associated with a supplemental field value, each supplemental field value weight reflecting a likelihood that an arbitrary record in the database comprises an associated supplemental field value;
forming a plurality of entity representations in the database, at least one entity representation comprising at least two records linked using a first instance of the record matching formula comprising a supplemental field value weight associated with a field value appearing in the supplemental field of at least one of the at least two records;
calculating a plurality of revised supplemental field value weights, each revised supplemental field value weight associated with a particular supplemental field value, each revised supplemental field value weight reflecting a likelihood that an arbitrary entity representation in the database comprises an associated supplemental field value;
linking at least two entity representations in the database based on a second instance of the record matching formula, wherein the second instance of the record matching formula comprises a revised supplemental field value weight associated with a field value appearing in the supplemental field of at least one of the at least two entity representations, whereby a number of entity representations in the database is reduced by the forming a plurality of linked entity representations; and retrieving information from at least one record in the database.
78. The process of claim 77, further comprising repeating the calculating a plurality of revised supplemental field value weights and the linking at least two entity representations at least once prior to the retrieving.
79. The process of claim 77, wherein the record matching formula comprises a weighted sum of probabilities that two records match.
80. The process of claim 77, wherein each supplemental field value weight comprises a logarithm of a probability and wherein each revised supplemental field value weight comprises a logarithm of a probability.
81. A computer implemented iterative process for generating entity representations in a computer implemented database using a record matching formula and for generating parameters for the record matching formula, the database comprising a plurality of records, each record comprising a plurality of fields, each field capable of containing a field value, wherein at least a portion of the parameters for the record matching formula are specific to a particular plurality of fields, the process comprising:
adding, in the database, a supplemental field to each of the plurality of records;
populating each supplemental field of each of the plurality of records with a supplemental field value, each supplemental field value representative of field values from the particularly plurality of fields of that record;
calculating a plurality of supplemental field value weights, each supplemental field value weight associated with a supplemental field value, each supplemental field value weight reflecting a likelihood that an arbitrary record in the database comprises an associated supplemental field value;
calculating a supplemental field weight, the supplemental field weight derived from each of the plurality of supplemental field value weights;
forming a plurality of entity representations in the database, at least one entity representation comprising at least two records linked using a first instance of the record matching formula comprising the supplemental field weight;
calculating a plurality of revised supplemental field value weights, each revised supplemental field value weight associated with a particular supplemental field value, each revised supplemental field value weight reflecting a likelihood that an arbitrary entity representation in the database comprises an associated supplemental field value;
calculating a revised supplemental field weight, the revised supplemental field weight derived from each of the plurality of revised supplemental field value weights;
linking at least two entity representations in the database based on a second instance of the record matching formula, wherein the second instance of the record matching formula comprises the revised supplemental field weight, whereby a number of entity representations in the database is reduced by the forming a plurality of linked entity representations; and retrieving information from at least one record in the database.
82. The process of claim 81 further comprising repeating the calculating a plurality of revised supplemental field value weights, the calculating a revised supplemental field weight, and the linking at least two entity representations at least once prior to the retrieving.
83. The process of claim 81, wherein the record matching formula comprises a weighted sum of probabilities that two records match.
84. The process of claim 81, wherein each supplemental field value weight comprises a logarithm of a probability and wherein each revised supplemental field value weight comprises a logarithm of a probability.
85. A computer system for iteratively generating entity representations in a computer implemented database using a record matching formula and for generating parameters for the record matching formula, the database comprising a plurality of records, each record comprising a plurality of fields, each field capable of containing a field value, wherein at least a portion of the parameters for the record matching formula are specific to a particular plurality of field values associated with a particular plurality of fields, the system comprising:
a database comprising a plurality of records, each record comprising a plurality of fields, each field capable of containing a field value a processor programmed to add, in the database, a supplemental field to each of the plurality of records;
a processor programmed to populate each supplemental field of each of the plurality of records with a supplemental field value, each supplemental field value representative of field values from the particularly plurality of fields of that record;
a processor programmed to calculate a plurality of supplemental field value weights, each supplemental field value weight associated with a supplemental field value, each supplemental field value weight reflecting a likelihood that an arbitrary record in the database comprises an associated supplemental field value;
a processor programmed to form and store a plurality of entity representations in the database, at least one entity representation comprising at least two records linked using a first instance of the record matching formula comprising a supplemental field value weight associated with a field value appearing in the supplemental field of at least one of the at least two records;
a processor programmed to calculate a plurality of revised supplemental field value weights, each revised supplemental field value weight associated with a particular supplemental field value, each revised supplemental field value weight reflecting a likelihood that an arbitrary entity representation in the database comprises an associated supplemental field value; and a processor programmed to link and store at least two entity representations in the database based on a second instance of the record matching formula, wherein the second instance of the record matching formula comprises a revised supplemental field value weight associated with a field value appearing in the supplemental field of at least one of the at least two entity representations, whereby a number of entity representations in the database is reduced by the forming a plurality of linked entity representations.
86. The system of claim 85, further comprising program logic configured to repeat calculating a plurality of revised supplemental field value weights and linking and storing at least two entity representations at least once prior to retrieving information from at least one record in the database.
87. The system of claim 85, wherein the record matching formula comprises a weighted sum of probabilities that two records match.
88. The system of claim 85, wherein each supplemental field value weight comprises a logarithm of a probability and wherein each revised supplemental field value weight comprises a logarithm of a probability.
89. A computer system for iteratively generating entity representations in a computer implemented database using a record matching formula and for generating parameters for the record matching formula, the database comprising a plurality of records, each record comprising a plurality of fields, each field capable of containing a field value, wherein at least a portion of the parameters for the record matching formula are specific to a particular plurality of fields, the system comprising:
a database comprising a plurality of records, each record comprising a plurality of fields, each field capable of containing a field value a processor programmed to add, in the database, a supplemental field to each of the plurality of records;
a processor programmed to populate each supplemental field of each of the plurality of records with a supplemental field value, each supplemental field value representative of field values from the particularly plurality of fields of that record;
a processor programmed to calculate a plurality of supplemental field value weights, each supplemental field value weight associated with a supplemental field value, each supplemental field value weight reflecting a likelihood that an arbitrary record in the database comprises an associated supplemental field value;
a processor programmed to calculate a supplemental field weight, the supplemental field weight derived from each of the plurality of supplemental field value weights;
a processor programmed to form and store a ,plurality of entity representations in the database, at least one entity representation comprising at least two records linked using a first instance of the record matching formula comprising the supplemental field weight;
a processor programmed to calculate a plurality of revised supplemental field value weights, each revised supplemental field value weight associated with a particular supplemental field value, each revised supplemental field value weight reflecting a likelihood that an arbitrary entity representation in the database comprises an associated supplemental field value;
a processor programmed to calculate a revised supplemental field weight, the revised supplemental field weight derived from each of the plurality of revised supplemental field value weights; and a processor programmed to link and store at least two entity representations in the database based on a second instance of the record matching formula, wherein the second instance of the record matching formula comprises the revised supplemental field weight, whereby a number of entity representations in the database is reduced by the forming a plurality of linked entity representations.
90. The system of claim 89, further comprising program logic configured to repeat calculating a plurality of revised supplemental field value weights, calculating a revised supplemental field weight, and linking and storing at least two entity representations at least once.
91. The system of claim 89, wherein the record matching formula comprises a weighted sum of probabilities that two records match.
92. The system of claim 89, wherein each supplemental field value weight comprises a logarithm of a probability and wherein each revised supplemental field value weight comprises a logarithm of a probability.

(VI.)
93. A computer implemented method of detecting and treating one or more field values as null field values in records of an electronic database, the computer implemented method comprising:
selecting a field of the records;
determining, using a programmed computer, a frequency for each field value of the field, the frequency comprising an amount of the records in which the field value of the field appears;
calculating, using a programmed computer, a critical frequency for the field based on the frequency of each field value of the field; and treating, using a programmed computer, each field value of the field with a frequency that is greater than the critical frequency of the field as a null field value and each field value of the field with a frequency that is less than the critical frequency of the field as a non-null field value.
94. The computer implemented method of claim 93, wherein the calculating comprises ordering the frequencies of the field values of the field according to frequency along a horizontal axis of a histogram of the one or more field values of the field.
95. The computer implemented method of claim 94, wherein the calculating further comprises calculating a difference between adjacent frequencies in the histogram.
96. The computer implemented method of claim 93, wherein the calculating comprises calculating f(x) = g(x) - g(x+1), wherein f is a function of the frequency differences based on a function g that is defined by the frequency of each field value of the field.
97. The computer implemented method of claim 96, wherein the critical frequency of the field is determined by determining the first point at which a derivative of the f function changes from a first sign to a second sign.
98. The computer implemented method of claim 96, wherein the critical frequency of the field is determined by determining the first point at which a derivative of the f function changes from a positive sign to a negative sign.
99. The computer implemented method of claim 96, wherein the critical frequency of the field is determined by determining the first point at which a derivative of the f function changes from a negative sign to a positive sign.
100. The computer implemented method of claim 96, wherein the critical frequency of the field is determined by determining the first point at which a derivative of the f function equals zero.
101. The computer implemented method of claim 96, wherein the critical frequency of the field is determined by determining the first point at which a derivative of the f function decreases below a threshold.
102. The computer implemented method of claim 93, wherein the treating further comprises treating each field value of the field with a frequency that is equal to the critical frequency of the field as a null field value.
103. The computer implemented method of claim 93, further comprising replacing one or more field values treated as null field values with a canonical null field value.
104. A computer implemented system of detecting and treating one or more field values as null field values in records of an electronic database, the computer implemented system comprising:
a first computing apparatus configured to select a field of the records;
a second computing apparatus configured to determine a frequency for each field value of the field, the frequency comprising an amount of the records in which the field value of the field appears;
a third computing apparatus configured to calculate a critical frequency for the field based on the frequency of each field value of the field; and a fourth computing apparatus configured to treat each field value of the field with a frequency that is greater than the critical frequency of the field as a null field value and each field value of the field with a frequency that is less than the critical frequency of the field as a non-null field value.
105. The computer implemented system of claim 104, wherein the third computing apparatus is configured to calculate by ordering the frequencies of the field values of the field according to frequency along a horizontal axis of a histogram of the one or more field values of the field.
106. The computer implemented system of claim 105, wherein the third computing apparatus is further configured to calculate by calculating a difference between adjacent frequencies in the histogram.
107. The computer implemented system of claim 104, wherein the third computing apparatus is configured to calculate by calculating f(x) = g(x) - g(x+1), wherein f is a function of the frequency differences based on a function g that is defined by the frequency of each field value of the field.
108. The computer implemented system of claim 106, wherein the critical frequency of the field is determined by determining the first point at which a derivative of the f function changes from a first sign to a second sign.
109. The computer implemented system of claim 106, wherein the critical frequency of the field is determined by determining the first point at which a derivative of the f function changes from a positive sign to a negative sign.
110. The computer implemented system of claim 106, wherein the critical frequency of the field is determined by determining the first point at which a derivative of the f function changes from a negative sign to a positive sign.
111. The computer implemented system of claim 106, wherein the critical frequency of the field is determined by determining the first point at which a derivative of the f function equals zero.
112. The computer implemented system of claim 106, wherein the critical frequency of the field is determined by determining the first point at which a derivative of the f function decreases below a threshold.
113. The computer implemented system of claim 104, wherein the fourth computing apparatus is further configured to treat each field value of the field with a frequency that is equal to the critical frequency of the field as a null field value.
114. The computer implemented system of claim 104, further comprising a fifth computing apparatus configured to replace one or more field values treated as null field values with a canonical null field value.

(VII.)
115. A computer implemented iterative process for generating entity representations by identifying and linking related records in a computer implemented database using a record matching formula, each record and entity representation electronically stored in the database, each record comprising a plurality of fields, each field capable of containing a field value, the process comprising:
assigning to each pair of records from a plurality of records in the database a match value using a the record matching formula, the match value reflecting a likelihood that the pair of records is related, the match value computed by a programmed processor;
assigning, for each record from the plurality of records, at least one associated preferred record, wherein a match value assigned to a given record together with its associated preferred record is at least as great as a match value assigned to the record together with any other record in the plurality of records;
identifying mutually preferred pairs of records from the plurality of records, each mutually preferred pair of records consisting of a first record and a second record, the first record consisting of a preferred record associated with the second record and the second record consisting of a preferred record associated with the first record;
forming and storing a plurality of entity representations in the database, each entity representation of the plurality of entity representations comprising at least one linked pair of mutually preferred records; and retrieving information from at least one record of a pair of mutually preferred records.
116. The process of claim 115 wherein each preferred record associated with a given record comprises a record that, when paired with the given record, has a maximal assigned match value in comparison to match values assigned to other record pairs comprising the given record.
117. The process of claim 115 further comprising, for at least one mutually preferred pair of records consisting of a first record and a second record, altering at least one field value from the first record based on at least one field value from the second record.
118. The process of claim 115 wherein the match value reflects a number of field values common to the pair of records.
119. The process of claim 115 wherein the database comprises a third record and a fourth record, wherein the third record is an associated preferred record of the fourth record and wherein the fourth record is not an associated preferred record of the third record.
120. The process of claim 115 further comprising, after the step of forming and prior to the step of retrieving:
for a particular record in a particular entity representation, assigning to each pair of records consisting of the particular record and a record from a entity representation different from the particular entity representation, a match value using the record matching formula;
determining a mutually preferred record for the particular record; and linking the particular entity representation to the mutually preferred record for the particular record.
121. The process of claim 115 wherein the record matching formula is of the form
122. A computer implemented iterative process for generating entity representations by identifying and linking related records in a computer implemented database using a record matching formula, each record and entity representation electronically stored in the database, each record comprising a plurality of fields, each field capable of containing a field value, the process comprising:

determining a mutually preferred pair of records consisting of a first record and a second record, wherein a match score of the first record and the second record as computed using the record matching formula is at least as great as a match score of the first record and any other record in the database, and wherein the match score of the first record and the second record as computed using the record matching formula is at least as great as a match score for the second record and any other record in the database;
forming a new entity representation in the database, the new entity representation comprising at least the first record and the second record;
determining a mutually preferred pair of records consisting of the first record and a third record, the third record from a different entity representation than the new entity representation, wherein a match score of the first record and the third record as computed using the record matching formula is at least as great as a match score of the first record and any other record not in the new entity representation, and wherein a match score of the first record and the third record as computed using the record matching formula is at least as great as a match score for the third record and any other record not in the different entity representation;
consolidating the new entity representation with the different entity representation by linking the new entity representation with the different entity representation resulting in a consolidated entity representation; and retrieving information from the consolidation entity representation.
123. The process of claim 122 wherein the record matching formula is of the form
124. A computer system for iteratively generating entity representations in a computer implemented database using a record matching formula, the database comprising a plurality of records, each record comprising a plurality of fields, each field capable of containing a field value, the system comprising:
a computer implemented database comprising a plurality of records, each record comprising a plurality of fields, each field capable of containing a field value;

a processor programmed to assign to each pair of records from a plurality of records in the database a match value using a the record matching formula, the match value reflecting a likelihood that the pair of records is related, the match value computed by a programmed processor;
a processor programmed to assign, for each record from the plurality of records, at least one associated preferred record, wherein a match value assigned to a given record together with its associated preferred record is at least as great as a match value assigned to the record together with any other record in the plurality of records;
a processor programmed to identify mutually preferred pairs of records from the plurality of records, each mutually preferred pair of records consisting of a first record and a second record, the first record consisting of a preferred record associated with the second record and the second record consisting of a preferred record associated with the first record; and a processor programmed to form and store a plurality of entity representations in the database, each entity representation of the plurality of entity representations comprising at least one linked pair of mutually preferred records.
125. The system of claim 124 wherein each preferred record associated with a given record comprises a record that, when paired with the given record, has a maximal assigned match value in comparison to match values assigned to other record pairs comprising the given record.
126. The system of claim 124 further comprising processor programmed to, for at least one mutually preferred pair of records consisting of a first record and a second record, alter at least one field value from the first record based on at least one field value from the second record.
127. The system of claim 124 wherein the match value reflects a number of field values common to the pair of records.
128. The system of claim 124 wherein the database comprises a third record and a fourth record, wherein the third record is an associated preferred record of the fourth record and wherein the fourth record is not an associated preferred record of the third record.
129. The system of claim 124 further comprising:
a processor programmed to, for a particular record in a particular entity representation, assign to each pair of records consisting of the particular record and a record from a entity representation different from the particular entity representation, a match value using the record matching formula;
a processor programmed to determine a mutually preferred record for the particular record; and a processor programmed to link the particular entity representation to the mutually preferred record for the particular record.
130. The system of claim 124 wherein the record matching formula is of the form
131. A computer system for iteratively generating entity representations by identifying and linking related records in a computer implemented database using a record matching formula, each record and entity representation electronically stored in the database, each record comprising a plurality of fields, each field capable of containing a field value, the system comprising:
a computer implemented database comprising a plurality of records, each record comprising a plurality of fields, each field capable of containing a field value;
a processor programmed to determine a mutually preferred pair of records consisting of a first record and a second record, wherein a match score of the first record and the second record as computed using the record matching formula is at least as great as a match score of the first record and any other record in the database, and wherein the match score of the first record and the second record as computed using the record matching formula is at least as great as a match score for the second record and any other record in the database;
a processor programmed to form and store a new entity representation in the database, the new entity representation comprising at least the first record and the second record;
a processor programmed to determine a mutually preferred pair of records consisting of the first record and a third record, the third record from a different entity representation than the new entity representation, wherein a match score of the first record and the third record as computed using the record matching formula is at least as great as a match score of the first record and any other record not in the new entity representation, and wherein a match score of the first record and the third record as computed using the record matching formula is at least as great as a match score for the third record and any other record not in the different entity representation; and a processor programmed to consolidate the new entity representation with the different entity representation by linking the new entity representation with the different entity representation, resulting in a consolidated entity representation.
132. The system of claim 131 wherein the record matching formula is of the form (VIII.)
133. A computer implemented method of identifying a set of fields applicable to partition a plurality of records in an electronic database into one or more blocks based on a desired block size and independent of specific queries against the database, the method comprising:
receiving a desired block size;
calculating field probabilities for a plurality of fields in the database, wherein each field probability represents an average cohort size for a field, each of the field probabilities associated with one of the fields in the database;

determining a set of fields wherein a product of the associated field probabilities and the number of records in the database is approximately equal to the desired block size; and outputting the set of fields, the set of fields independent of specific queries against the database.
134. The method of claim 133, wherein each field probability is calculated as a weighted sum of field value probabilities associated with the associated field in the database.
135. The computer implemented method of claim 133, wherein the determining a set of fields comprises determining a set of fields wherein a product of the associated field probabilities and the number of records in the database is less than or equal to the desired block size.
136. The computer implemented method of claim 133, wherein the determining a set of fields comprises determining a set of fields wherein a product of the associated field probabilities and the number of records in the database is not more than one percent more than the desired block size.
137. The computer implemented method of claim 133, wherein the determining a set of fields comprises determining a set of fields wherein a product of the associated field probabilities and the number of records in the database is not more than five percent more than the desired block size.
138. The method of claim 133, wherein the set of fields is determined by combining the field probabilities of one or more fields by mathematical calculation until the combined field probabilities of the one or more fields in the set of fields multiplied by the plurality of records in the database is equal to or less than the desired block size.
139. The method of claim 138, further comprising ordering the field probabilities from a lowest field probability to a highest field probability, and the lowest field probability is multiplied with successive higher field probabilities until the cumulative field probability multiplied by the number of records in the database is equal to or less than the desired block size.
140. The method of claim 138, further comprising calculating a potential remaining probability for each field probability by multiplying a largest field probability with successive smaller field probabilities.
141. The method of claim 133, where the desired block size is defined as a number of records.
142. The method of claim 133, where the desired block size is defined as a percentage of the total number of records in the database.
143. The method of claim 133, where the desired block size is defined as the size of a block processed by one or more computers.
144. The method of claim 133, further comprising arranging the fields of the database into a search tree and determining a set of fields according to one or more branches of the search tree.
145. The method of claim 144, further comprising calculating a potential remaining probability for the one or more branches by multiplying a largest field probability with successive smaller field probabilities, and removing branches in the search tree that do not yield a subset of records equal to or less than the desired block size based on the potential remaining probability of the combination.
146. The method of claim 133, where more than one set of fields is determined and presented in an output.
147. A computer implemented method of creating blocking criteria based on a desired block size, the method comprising:
calculating, using a programmed computer, one or more field probabilities for one or more fields in an electronic database, wherein each field probability represents an average cohort size for a field, each of the field probabilities associated with one of the fields in the database;
determining, using a programmed computer, one or more fields wherein a product of the associated field probabilities and a number of records in the database is approximately equal to the desired block size;
grouping, using a programmed computer, the one or more fields into one or more blocking criteria;
outputting the one or more blocking criteria; and applying, using a programmed computer, at least one of the one or more blocking criteria to the records of the database create a smaller group of records in the database.
148. The method of claim 147, wherein the set of fields is determined by combining the field probabilities of one or more fields by mathematical calculation until the combined field probabilities of the one or more fields in the set of fields multiplied by the plurality of records in the database is equal to or less than the desired block size.
149. The method of claim 148, wherein the field probabilities are ordered from a lowest field probability to a highest field probability, and the lowest field probability is multiplied with successive higher field probabilities until the cumulative field probability multiplied by the number of records in the database is equal to or less than the desired block size.
150. The method of claim 148, further comprising calculating a potential remaining probability for each field probability by multiplying a largest field probability with successive smaller field probabilities.
151. The method of claim 147, wherein the determining, using a programmed computer, one or more fields comprises determining a set of fields wherein a product of the associated field probabilities and the number of records in the database is not more than one percent more than the desired block size.
152. The method of claim 147, wherein the determining, using a programmed computer, one or more fields comprises determining a set of fields wherein a product of the associated field probabilities and the number of records in the database is not more than five percent more than the desired block size.
153. The method of claim 147, further comprising arranging the fields of the database into a search tree and determining a set of fields according to one or more branches of the search tree.
154. The method of claim 153, further comprising calculating a potential remaining probability for the one or more branches by multiplying a largest field probability with successive smaller field probabilities, and removing branches in the search tree that do not yield a subset of records equal to or less than the desired block size based on the potential remaining probability of the combination.
155. The method of claim 147, further comprising transmitting at least one of the one or more blocking criteria one or more matching algorithms.
156. A system for identifying a set of fields applicable to partition a plurality of records in an electronic database into one or more blocks based on a desired block size and independent of specific queries against the database, comprising:
an electronic processor configured to receive a desired block size;
an electronic processor configured to calculate field probabilities for a plurality of fields in the database, wherein each field probability represents an average cohort size for a field, each of the field probabilities associated with one of the fields in the database;

an electronic processor configured to determine a set of fields wherein a product of the associated field probabilities and the number of records in the database is approximately equal to the desired block size; and an electronic processor configured to output the set of fields, the set of fields independent of specific queries against the database.
157. The system of claim 156, wherein the set of fields is determined by operating an electronic processor to combine the field probabilities of one or more fields by mathematical calculation until the combined field probabilities of the one or more fields in the set of fields multiplied by the plurality of records in the database is equal to or less than the desired block size.
158. The system of claim 157, wherein an electronic processor is used to order the field probabilities from a lowest field probability to a highest field probability, and the lowest field probability is multiplied with successive higher field probabilities until the cumulative field probability multiplied by the number of records in the database is equal to or less than the desired block size.
159. The system of claim 157, further comprising an electronic processor configured to calculate a potential remaining probability for each field probability by multiplying a largest field probability with successive smaller field probabilities.
160. The system of claim 156, wherein the product of the associated field probabilities and the number of records in the database is not more than one percent more than the desired block size.
161. The system of claim 156, wherein the product of the associated field probabilities and the number of records in the database is not more than five percent more than the desired block size.
162. The system of claim 156, further comprising an electronic processor configured to arrange the fields of the database into a search tree and an electronic processor configured to determine a set of fields according to one or more branches of the search tree.
163. The system of claim 162, further comprising an electronic processor configured to calculate a potential remaining probability for the one or more branches by multiplying a largest field probability with successive smaller field probabilities, and an electronic processor configured to remove branches in the search tree that do not yield a subset of records equal to or less than the desired block size based on the potential remaining probability of the combination.

(IX.)
164. A computer implemented iterative process for generating entity representations in a computer implemented database using a record matching formula and for generating parameters for the record matching formula, the database comprising a plurality of records, each record comprising a Plurality of fields, each field capable of containing a field value, the process comprising:
calculating a field weight for a selected field, the field weight for the selected field derived from each of a plurality of field value weights for the selected field;
forming a plurality of entity representations in the database, each entity representation comprising at least two records linked using a first instance of the record matching formula, at least one entity representation comprising a first record linked to a second record using a first instance of the record matching formula wherein the first record comprises a different field value in its selected field than that of the second record, the first instance of the record matching formula comprising a negative of the field weight for the selected field;
calculating a weight parameter for the selected field, the weight parameter for the selected field reflecting a likelihood that an arbitrary entity representation in the database comprises two different records each comprising a different field value in its respective selected field, the weight parameter being a negative number;

linking at least two entity representations in the database based on a second instance of the record matching formula, wherein the second instance of the record matching formula comprises the weight parameter, whereby a number of entity representations in the database is reduced by the linking at least two entity representations relative to a number of entity representations in the database prior to the linking at least two entity representations; and retrieving information from at least one record in the database.
165. The process of claim 164 wherein each of the plurality of field value weights reflects a likelihood that an arbitrary record in the database comprises a different field value in the selected field of the arbitrary record.
166. The process of claim 164 wherein the weight parameter comprises a probability that that an arbitrary entity representation in the database contains a same field value in the selected field of each record comprising the arbitrary entity representation.
167. The process of claim 164, further comprising:
calculating a plurality of revised field value weights, each revised field value weight reflecting a likelihood that an arbitrary entity representation in the database comprises a different field value in the selected field of a record in the arbitrary entity representation; and calculating a revised field weight for the selected field, the revised field weight for the selected field derived from each of the plurality of revised field value weights;
wherein the weight parameter comprises a product of the revised field weight and a probability that an arbitrary entity representation in the database comprises two different records each comprising a different field value in its respective selected field.
168. The process of claim 164 wherein the weight parameter comprises a negative logarithm of a probability that an arbitrary entity representation in the database comprises two different records each comprising a different field value in its respective selected field, the weight parameter being a negative number.
169. A computer system for iteratively generating entity representations in a computer implemented database using a record matching formula and for generating parameters for the record matching formula, the database comprising a plurality of records, each record comprising a plurality of fields, each field capable of containing a field value, the system comprising:
a database comprising a plurality of records, each record comprising a plurality of fields, each field capable of containing a field value;
a processor programmed to calculate a field weight for a selected field, the field weight for the selected field derived from each of a plurality of field value weights for the selected field;
a processor programmed to form and store a plurality of entity representations in the database, each entity representation comprising at least two records linked using a first instance of the record matching formula, at least one entity representation comprising a first record linked to a second record using a first instance of the record matching formula wherein the first record comprises a different field value in its selected field than that of the second record, the first instance of the record matching formula comprising a negative of the field weight for the selected field;
a processor programmed to calculate a weight parameter for the selected field, the weight parameter for the selected field reflecting a likelihood that an arbitrary entity representation in the database comprises two different records each comprising a different field value in its respective selected field, the weight parameter being a negative number; and a processor programmed to link and store at least two entity representations in the database based on a second instance of the record matching formula, wherein the second instance of the record matching formula comprises the weight parameter, whereby a number of entity representations in the database is reduced by the linking at least two entity representations relative to a number of entity representations in the database prior to the linking at least two entity representations.
170. The system of claim 169 wherein each of the plurality of field value weights reflects a likelihood that an arbitrary record in the database comprises a different field value in the selected field of the arbitrary record.
171. The system of claim 169 wherein the weight parameter comprises a probability that that an arbitrary entity representation in the database contains a same field value in the selected field of each record comprising the arbitrary entity representation.
172. The system of claim 169, further comprising:
a processor programmed to calculate a plurality of revised field value weights, each revised field value weight reflecting a likelihood that an arbitrary entity representation in the database comprises a different field value in the selected field of a record in the arbitrary entity representation; and a processor programmed to calculate a revised field weight for the selected field, the revised field weight for the selected field derived from each of the plurality of revised field value weights;
wherein the weight parameter comprises a product of the revised field weight and a probability that an arbitrary entity representation in the database comprises two different records each comprising a different field value in its respective selected field.
173. The system of claim 169 wherein the weight parameter comprises a negative logarithm of a probability that an arbitrary entity representation in the database comprises two different records each comprising a different field value in its respective selected field, the weight parameter being a negative number.

(X.)
174. A computer implemented iterative process for generating entity representations in a computer implemented database using a record matching formula and for generating parameters for the record matching formula, the database comprising a plurality of records, each record comprising a plurality of fields, each field capable of containing a field value, each field value comprising at least one token, wherein at least a portion of parameters for the record matching formula are configured for a first particular field value and a non-identical second particular field value, the first particular field value and a second particular field value associated with a selected field, the first particular field value comprising a plurality of particular tokens, wherein the process provides for linking records and entity representations with non-identical field values, the process comprising:
calculating, for each of the plurality of particular tokens, an associated token weight, each token weight reflecting a likelihood that an arbitrary record in the database comprises the associated particular token in the selected field of the arbitrary record;
calculating a field value weight, the field value weight reflecting a likelihood that an arbitrary record in the database comprises the first particular field value in the selected field of the arbitrary record;
calculating a blended field value weight for the first particular field value and the second particular field value, the blended field value weight for the first particular field value and the second particular field value being derived from at least the token weights associated with each of the plurality of particular tokens and from the field value weight;
forming a plurality of entity representations in the database, each entity representation comprising at least two records linked using a first instance of the record matching formula, at least one entity representation comprising a first record comprising the first particular field value and a second record comprising the second particular field value, the first record linked to the second record using a first instance of the record matching formula that comprises the blended field value weight;
calculating, for each of the plurality of particular tokens, an associated revised token weight, each revised token weight reflecting a likelihood that an arbitrary entity representation in the database comprises a record comprising the associated particular token;
calculating a revised field value weight, the revised field value weight reflecting a likelihood that an arbitrary entity representation in the database comprises the first particular field value in the selected field of a record comprising the arbitrary entity representation;
calculating a revised blended field value weight for the first particular field value and the second particular field value, the revised blended field value weight for the first particular field value and the second particular field value being derived from at least the revised token weights associated with each of the plurality of particular tokens and from the revised field value weight;

linking at least two entity representations in the database based on a second instance of the record matching formula, wherein the second instance of the record matching formula comprises the revised blended field value weight, whereby a number of entity representations in the database is reduced by the linking at least two entity representations relative to a number of entity representations in the database prior to the linking at least two entity representations; and retrieving information from at least one record in the database.
175. The process of claim 174 further comprising repeating the calculating, for each of the plurality of particular tokens, an associated revised token weight, the calculating a revised field value weight, the calculating a revised blended field value weight, and the linking at least two entity representations at least once.
176. The process of claim 174 wherein the calculating the blended field value weight comprises calculating a sum of token weights associated with tokens that appear in the selected field of both of the first record and the second record, and calculating a sum of token weights associated with tokens that appear in the selected field of exactly one of the first record and the second record.
177. The process of claim 174 wherein the calculating the blended field value weight comprises calculating a maximum between (1) a probability reflecting a likelihood that an arbitrary record in the database comprises, in the selected field of the arbitrary record, the first particular field value, and (2) a probability reflecting a likelihood that an arbitrary record in the database comprises, in the selected field of the arbitrary record, the second particular field value.
178. The process of claim 174 wherein the calculating the blended field value weight comprises calculating a minimum between (1) a probability reflecting a likelihood that an arbitrary record in the database comprises, in the selected field of the arbitrary record, the first particular field value, and (2) a probability reflecting a likelihood that an arbitrary record in the database comprises, in the selected field of the arbitrary record, the second particular field value.
179. The process of claim 174 wherein the calculating the blended field value weight comprises calculating at least one of w i,j w i,j, =min(w i,j,w j) x , w i,j =max(w i,j,w j) x and w i,j =
min(w i, w j, M), wherein i is an index for the first record and j is an index for the second record.
180. A computer system for iteratively generating entity representations in a computer implemented database using a record matching formula and for generating parameters for the record matching formula, the database comprising a plurality of records, each record comprising a plurality of fields, each field capable of containing a field value, each field value comprising at least one token, wherein at least a portion of parameters for the record matching formula are configured for a first particular field value and a non-identical second particular field value, the first particular field value and a second particular field value associated with a selected field, the first particular field value comprising a plurality of particular tokens, wherein the process provides for linking records and entity representations with non-identical field values, the system comprising:
a computer implemented database comprising a plurality of records, each record comprising a plurality of fields, each field capable of containing a field value, each field value comprising at least one token;
a processor programmed to calculate, for each of the plurality of particular tokens, an associated token weight, each token weight reflecting a likelihood that an arbitrary record in the database comprises the associated particular token in the selected field of the arbitrary record;
a processor programmed to calculate a field value weight, the field value weight reflecting a likelihood that an arbitrary record in the database comprises the first particular field value in the selected field of the arbitrary record;
a processor programmed to calculate a blended field value weight for the first particular field value and the second particular field value, the blended field value weight for the first particular field value and the second particular field value being derived from at least the token weights associated with each of the plurality of particular tokens and from the field value weight;

a processor programmed to form and store a plurality of entity representations in the database, each entity representation comprising at least two records linked using a first instance of the record matching formula, at least one entity representation comprising a first record comprising the first particular field value and a second record comprising the second particular field value, the first record linked to the second record using a first instance of the record matching formula that comprises the blended field value weight;
a processor programmed to calculate, for each of the plurality of particular tokens, an associated revised token weight, each revised token weight reflecting a likelihood that an arbitrary entity representation in the database comprises a record comprising the associated particular token;
a processor programmed to calculate a revised field value weight, the revised field value weight reflecting a likelihood that an arbitrary entity representation in the database comprises the first particular field value in the selected field of a record comprising the arbitrary entity representation;
a processor programmed to calculate a revised blended field value weight for the first particular field value and the second particular field value, the revised blended field value weight for the first particular field value and the second particular field value being derived from at least the revised token weights associated with each of the plurality of particular tokens and from the revised field value weight; and a processor programmed to link and store at least two entity representations in the database based on a second instance of the record matching formula, wherein the second instance of the record matching formula comprises the revised blended field value weight, whereby a number of entity representations in the database is reduced by linking the at least two entity representations relative to a number of entity representations in the database prior to the linking at least two entity representations.
181. The system of claim 180 wherein the processor programmed to calculate a blended field value weight is programmed to calculate a sum of token weights associated with tokens that appear in the selected field of both of the first record and the second record, and calculate a sum of token weights associated with tokens that appear in the selected field of exactly one of the first record and the second record.
182. The system of claim 180 wherein the processor programmed to calculate the blended field value weight is programmed to calculate a maximum between (1) a probability reflecting a likelihood that an arbitrary record in the database comprises, in the selected field of the arbitrary record, the first particular field value, and (2) a probability reflecting a likelihood that an arbitrary record in the database comprises, in the selected field of the arbitrary record, the second particular field value.
183. The system of claim 180 wherein the processor programmed to calculate the blended field value weight is programmed to calculate a minimum between (1) a probability reflecting a likelihood that an arbitrary record in the database comprises, in the selected field of the arbitrary record, the first particular field value, and (2) a probability reflecting a likelihood that an arbitrary record in the database comprises, in the selected field of the arbitrary record, the second particular field value.
184. The system of claim 180 wherein the processor programmed to calculate the blended field value weight is programmed to calculate at least one of w i,j = max(w i, w j) x , w i,j = min(w i, w j) w i,j = max(w i, w j) and w j = min(w i, w j, M), wherein i is an index for the first record and j is an index for the second record.
CA2723167A 2008-04-24 2009-04-24 Database systems and methods for linking records corresponding to the same individual Active CA2723167C (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
CA2749310A CA2749310C (en) 2008-04-24 2009-04-24 Statistical record linkage calibration at the field and field value levels without the need for human interaction
CA2749080A CA2749080C (en) 2008-04-24 2009-04-24 Adaptive clustering of records and entity representations

Applications Claiming Priority (5)

Application Number Priority Date Filing Date Title
US4757008P 2008-04-24 2008-04-24
US61/047,570 2008-04-24
US5320208P 2008-05-14 2008-05-14
US61/053,202 2008-05-14
PCT/US2009/041649 WO2009132263A2 (en) 2008-04-24 2009-04-24 Database systems and methods

Related Child Applications (2)

Application Number Title Priority Date Filing Date
CA2749310A Division CA2749310C (en) 2008-04-24 2009-04-24 Statistical record linkage calibration at the field and field value levels without the need for human interaction
CA2749080A Division CA2749080C (en) 2008-04-24 2009-04-24 Adaptive clustering of records and entity representations

Publications (2)

Publication Number Publication Date
CA2723167A1 true CA2723167A1 (en) 2009-10-29
CA2723167C CA2723167C (en) 2012-10-30

Family

ID=41215989

Family Applications (3)

Application Number Title Priority Date Filing Date
CA2749080A Active CA2749080C (en) 2008-04-24 2009-04-24 Adaptive clustering of records and entity representations
CA2723167A Active CA2723167C (en) 2008-04-24 2009-04-24 Database systems and methods for linking records corresponding to the same individual
CA2749310A Active CA2749310C (en) 2008-04-24 2009-04-24 Statistical record linkage calibration at the field and field value levels without the need for human interaction

Family Applications Before (1)

Application Number Title Priority Date Filing Date
CA2749080A Active CA2749080C (en) 2008-04-24 2009-04-24 Adaptive clustering of records and entity representations

Family Applications After (1)

Application Number Title Priority Date Filing Date
CA2749310A Active CA2749310C (en) 2008-04-24 2009-04-24 Statistical record linkage calibration at the field and field value levels without the need for human interaction

Country Status (4)

Country Link
US (22) US8266168B2 (en)
CA (3) CA2749080C (en)
GB (1) GB2472335A (en)
WO (1) WO2009132263A2 (en)

Families Citing this family (231)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7657540B1 (en) 2003-02-04 2010-02-02 Seisint, Inc. Method and system for linking and delinking data records
US20050162122A1 (en) 2004-01-22 2005-07-28 Dunn Glenn M. Fuel cell power and management system, and technique for controlling and/or operating same
US7822665B2 (en) 2004-10-29 2010-10-26 American Express Travel Related Services Company, Inc. Using commercial share of wallet in private equity investments
US7792732B2 (en) 2004-10-29 2010-09-07 American Express Travel Related Services Company, Inc. Using commercial share of wallet to rate investments
US8204774B2 (en) 2004-10-29 2012-06-19 American Express Travel Related Services Company, Inc. Estimating the spend capacity of consumer households
US8543499B2 (en) 2004-10-29 2013-09-24 American Express Travel Related Services Company, Inc. Reducing risks related to check verification
US8086509B2 (en) 2004-10-29 2011-12-27 American Express Travel Related Services Company, Inc. Determining commercial share of wallet
US8630929B2 (en) 2004-10-29 2014-01-14 American Express Travel Related Services Company, Inc. Using commercial share of wallet to make lending decisions
US7814004B2 (en) 2004-10-29 2010-10-12 American Express Travel Related Services Company, Inc. Method and apparatus for development and use of a credit score based on spend capacity
US20070016501A1 (en) 2004-10-29 2007-01-18 American Express Travel Related Services Co., Inc., A New York Corporation Using commercial share of wallet to rate business prospects
US8239250B2 (en) 2006-12-01 2012-08-07 American Express Travel Related Services Company, Inc. Industry size of wallet
US8285656B1 (en) 2007-03-30 2012-10-09 Consumerinfo.Com, Inc. Systems and methods for data verification
US20090157426A1 (en) * 2007-12-12 2009-06-18 Mckesson Financial Holdings Limited Methods, apparatuses & computer program products for facilitating efficient distribution of data within a system
US8775441B2 (en) 2008-01-16 2014-07-08 Ab Initio Technology Llc Managing an archive for approximate string matching
US8266168B2 (en) 2008-04-24 2012-09-11 Lexisnexis Risk & Information Analytics Group Inc. Database systems and methods for linking records and entity representations with sufficiently high confidence
US8312033B1 (en) 2008-06-26 2012-11-13 Experian Marketing Solutions, Inc. Systems and methods for providing an integrated identifier
GB2472358A (en) 2008-07-02 2011-02-02 Lexisnexis Risk & Information Analytics Group Inc Database system and methods
AU2009308206B2 (en) 2008-10-23 2015-08-06 Ab Initio Technology Llc Fuzzy data operations
US9135396B1 (en) * 2008-12-22 2015-09-15 Amazon Technologies, Inc. Method and system for determining sets of variant items
US20100169348A1 (en) * 2008-12-31 2010-07-01 Evrichart, Inc. Systems and Methods for Handling Multiple Records
US8196022B2 (en) * 2009-01-16 2012-06-05 International Business Machines Corporation Hamming radius separated deduplication links
US8312370B2 (en) * 2009-03-10 2012-11-13 Lsi Corporation System and method of hardware-assisted assembly of documents
US8171134B2 (en) * 2009-03-18 2012-05-01 At&T Intellectual Property I, L.P. Methods and apparatus to characterize and predict network health status
US8332185B2 (en) * 2009-08-17 2012-12-11 Lockheed Martin Corporation Method and system for calculating elementary symmetric functions of subsets of a set
US20110066446A1 (en) * 2009-09-15 2011-03-17 Arien Malec Method, apparatus and computer program product for providing a distributed registration manager
US8452763B1 (en) * 2009-11-19 2013-05-28 Google Inc. Extracting and scoring class-instance pairs
US9411859B2 (en) 2009-12-14 2016-08-09 Lexisnexis Risk Solutions Fl Inc External linking based on hierarchical level weightings
US8452795B1 (en) 2010-01-15 2013-05-28 Google Inc. Generating query suggestions using class-instance relationships
US8478705B2 (en) * 2010-01-15 2013-07-02 International Business Machines Corporation Portable data management using rule definitions
EP2529324A4 (en) * 2010-01-29 2016-06-01 Lexisnexis Risk Data Man Inc Statistical record linkage calibration for geographic proximity matching
GB201001859D0 (en) * 2010-02-04 2010-03-24 Isis Innovation A method of linking electronic database records
US20110218819A1 (en) * 2010-03-02 2011-09-08 Mckesson Financial Holdings Limited Method, apparatus and computer program product for providing a distributed care planning tool
US8352460B2 (en) * 2010-03-29 2013-01-08 International Business Machines Corporation Multiple candidate selection in an entity resolution system
US8538916B1 (en) 2010-04-09 2013-09-17 Google Inc. Extracting instance attributes from text
EP2558988A4 (en) * 2010-04-14 2016-12-21 The Dun And Bradstreet Corp Ascribing actionable attributes to data that describes a personal identity
US20110289529A1 (en) * 2010-05-18 2011-11-24 Ropvi Technologies Corporation user interface for content browsing and selection in a television portal of a content system
US9189505B2 (en) 2010-08-09 2015-11-17 Lexisnexis Risk Data Management, Inc. System of and method for entity representation splitting without the need for human interaction
US8666998B2 (en) * 2010-09-14 2014-03-04 International Business Machines Corporation Handling data sets
US8918393B2 (en) 2010-09-29 2014-12-23 International Business Machines Corporation Identifying a set of candidate entities for an identity record
US20120089604A1 (en) * 2010-10-08 2012-04-12 Jocelyn Siu Luan Hamilton Computer-Implemented Systems And Methods For Matching Records Using Matchcodes With Scores
US8407215B2 (en) * 2010-12-10 2013-03-26 Sap Ag Text analysis to identify relevant entities
US8949166B2 (en) 2010-12-16 2015-02-03 International Business Machines Corporation Creating and processing a data rule for data quality
CN102024053B (en) * 2010-12-17 2012-12-19 东北大学 Approximate circle matching method for isomorphic and symmetric publish-subscribe system
US8635197B2 (en) * 2011-02-28 2014-01-21 International Business Machines Corporation Systems and methods for efficient development of a rule-based system using crowd-sourcing
JP2012212422A (en) * 2011-03-24 2012-11-01 Sony Corp Information processor, information processing method, and program
US10042895B1 (en) * 2011-03-31 2018-08-07 Upstream Commerce Ltd. Product information integration
US9009124B2 (en) * 2011-06-17 2015-04-14 Microsoft Technology Licensing, Llc Compression aware physical database design
US9547693B1 (en) 2011-06-23 2017-01-17 Palantir Technologies Inc. Periodic database search manager for multiple data sources
US8874616B1 (en) * 2011-07-11 2014-10-28 21Ct, Inc. Method and apparatus for fusion of multi-modal interaction data
US8706732B1 (en) 2011-07-12 2014-04-22 Google Inc. Managing information about entities using clusters of received observations
US8676804B1 (en) 2011-07-12 2014-03-18 Google Inc. Managing information about entities using observations generated from user modified values
US8898104B2 (en) 2011-07-26 2014-11-25 International Business Machines Corporation Auto-mapping between source and target models using statistical and ontology techniques
US10636083B1 (en) * 2011-07-27 2020-04-28 Intuit Inc. Systems methods and articles of manufacture for analyzing on-line banking account data using hybrid edit distance
US8676823B2 (en) 2011-08-16 2014-03-18 National Student Clearinghouse Efficient student record matching
CA2855715C (en) 2011-11-15 2019-02-19 Ab Initio Technology Llc Data clustering based on candidate queries
US9058380B2 (en) 2012-02-06 2015-06-16 Fis Financial Compliance Solutions, Llc Methods and systems for list filtering based on known entity matching
US8473410B1 (en) 2012-02-23 2013-06-25 American Express Travel Related Services Company, Inc. Systems and methods for identifying financial relationships
US8538869B1 (en) 2012-02-23 2013-09-17 American Express Travel Related Services Company, Inc. Systems and methods for identifying financial relationships
US8442886B1 (en) 2012-02-23 2013-05-14 American Express Travel Related Services Company, Inc. Systems and methods for identifying financial relationships
US9477988B2 (en) 2012-02-23 2016-10-25 American Express Travel Related Services Company, Inc. Systems and methods for identifying financial relationships
US8781954B2 (en) 2012-02-23 2014-07-15 American Express Travel Related Services Company, Inc. Systems and methods for identifying financial relationships
US8805900B2 (en) 2012-03-30 2014-08-12 Mckesson Financial Holdings Methods, apparatuses and computer program products for facilitating location and retrieval of health information in a healthcare system
US20130311362A1 (en) * 2012-04-26 2013-11-21 Mastercard International Incorporated Systems and methods for verifying payee information in electronic payments
US20130290177A1 (en) * 2012-04-26 2013-10-31 Amy Christine Milam Systems and methods for facilitating processing of electronic payments
US9262475B2 (en) 2012-06-12 2016-02-16 Melissa Data Corp. Systems and methods for matching records using geographic proximity
US20140122176A1 (en) 2012-06-18 2014-05-01 ServiceSource International, Inc. Predictive model of recurring revenue opportunities
AU2013277314A1 (en) * 2012-06-18 2015-01-22 ServiceSource International, Inc. Service asset management system and method
US9652776B2 (en) 2012-06-18 2017-05-16 Greg Olsen Visual representations of recurring revenue management system data and predictions
US9460200B2 (en) 2012-07-02 2016-10-04 International Business Machines Corporation Activity recommendation based on a context-based electronic files search
US9336302B1 (en) 2012-07-20 2016-05-10 Zuci Realty Llc Insight and algorithmic clustering for automated synthesis
US9262499B2 (en) 2012-08-08 2016-02-16 International Business Machines Corporation Context-based graphical database
US9619580B2 (en) 2012-09-11 2017-04-11 International Business Machines Corporation Generation of synthetic context objects
US8620958B1 (en) 2012-09-11 2013-12-31 International Business Machines Corporation Dimensionally constrained synthetic context objects database
US9251237B2 (en) 2012-09-11 2016-02-02 International Business Machines Corporation User-specific synthetic context object matching
US9223846B2 (en) * 2012-09-18 2015-12-29 International Business Machines Corporation Context-based navigation through a database
US9741138B2 (en) 2012-10-10 2017-08-22 International Business Machines Corporation Node cluster relationships in a graph database
US8931109B2 (en) 2012-11-19 2015-01-06 International Business Machines Corporation Context-based security screening for accessing data
US9275117B1 (en) * 2012-12-06 2016-03-01 Emc Corporation Fast dependency mining using access patterns in a storage system
US9563677B2 (en) 2012-12-11 2017-02-07 Melissa Data Corp. Systems and methods for clustered matching of records using geographic proximity
JP5971115B2 (en) * 2012-12-26 2016-08-17 富士通株式会社 Information processing program, information processing method and apparatus
US8983981B2 (en) 2013-01-02 2015-03-17 International Business Machines Corporation Conformed dimensional and context-based data gravity wells
US9229932B2 (en) 2013-01-02 2016-01-05 International Business Machines Corporation Conformed dimensional data gravity wells
US9053102B2 (en) 2013-01-31 2015-06-09 International Business Machines Corporation Generation of synthetic context frameworks for dimensionally constrained hierarchical synthetic context-based objects
US9069752B2 (en) 2013-01-31 2015-06-30 International Business Machines Corporation Measuring and displaying facets in context-based conformed dimensional data gravity wells
US9292506B2 (en) 2013-02-28 2016-03-22 International Business Machines Corporation Dynamic generation of demonstrative aids for a meeting
US10275778B1 (en) 2013-03-15 2019-04-30 Palantir Technologies Inc. Systems and user interfaces for dynamic and interactive investigation based on automatic malfeasance clustering of related data in various data structures
US9230280B1 (en) 2013-03-15 2016-01-05 Palantir Technologies Inc. Clustering data based on indications of financial malfeasance
US9965937B2 (en) 2013-03-15 2018-05-08 Palantir Technologies Inc. External malware data item clustering and analysis
US8930897B2 (en) 2013-03-15 2015-01-06 Palantir Technologies Inc. Data integration tool
US8855999B1 (en) 2013-03-15 2014-10-07 Palantir Technologies Inc. Method and system for generating a parser and parsing complex data
US8903717B2 (en) 2013-03-15 2014-12-02 Palantir Technologies Inc. Method and system for generating a parser and parsing complex data
US8818892B1 (en) 2013-03-15 2014-08-26 Palantir Technologies, Inc. Prioritizing data clusters with customizable scoring strategies
US10152526B2 (en) 2013-04-11 2018-12-11 International Business Machines Corporation Generation of synthetic context objects using bounded context objects
US9830377B1 (en) * 2013-04-26 2017-11-28 Wal-Mart Stores, Inc. Methods and systems for hierarchical blocking
US9760654B2 (en) 2013-04-26 2017-09-12 Wal-Mart Stores, Inc. Method and system for focused multi-blocking to increase link identification rates in record comparison
US10803102B1 (en) * 2013-04-30 2020-10-13 Walmart Apollo, Llc Methods and systems for comparing customer records
US9767127B2 (en) 2013-05-02 2017-09-19 Outseeker Corp. Method for record linkage from multiple sources
US9348794B2 (en) 2013-05-17 2016-05-24 International Business Machines Corporation Population of context-based data gravity wells
US9195608B2 (en) 2013-05-17 2015-11-24 International Business Machines Corporation Stored data analysis
KR101416261B1 (en) * 2013-05-22 2014-07-09 연세대학교 산학협력단 Method for updating inverted index of flash SSD
EP2808804A1 (en) * 2013-05-29 2014-12-03 Fujitsu Ltd. Database controller, method, and program for handling range queries
US10248875B2 (en) * 2013-06-14 2019-04-02 Aware Inc. Method for automatically detecting and repairing biometric crosslinks
IL227135B (en) * 2013-06-23 2018-05-31 Drori Gideon Method and system for constructing a data base of consolidated entries
US8601326B1 (en) 2013-07-05 2013-12-03 Palantir Technologies, Inc. Data quality monitors
WO2015012991A1 (en) * 2013-07-23 2015-01-29 Mastercard International Incorporated Systems and methods for verifying payee information in electronic payments
US10042911B2 (en) * 2013-07-30 2018-08-07 International Business Machines Corporations Discovery of related entities in a master data management system
US10510440B1 (en) 2013-08-15 2019-12-17 Change Healthcare Holdings, Llc Method and apparatus for identifying matching record candidates
US11114185B1 (en) 2013-08-20 2021-09-07 Change Healthcare Holdings, Llc Method and apparatus for defining a level of assurance in a link between patient records
US9830325B1 (en) * 2013-09-11 2017-11-28 Intuit Inc. Determining a likelihood that two entities are the same
US10956997B2 (en) * 2013-10-10 2021-03-23 E-Legal, Inc. System, method, and process for the automatic generation of documents
US9697240B2 (en) 2013-10-11 2017-07-04 International Business Machines Corporation Contextual state of changed data structures
US9116975B2 (en) 2013-10-18 2015-08-25 Palantir Technologies Inc. Systems and user interfaces for dynamic and interactive simultaneous querying of multiple data stores
US9443015B1 (en) * 2013-10-31 2016-09-13 Allscripts Software, Llc Automatic disambiguation assistance for similar items in a set
US10102536B1 (en) 2013-11-15 2018-10-16 Experian Information Solutions, Inc. Micro-geographic aggregation system
WO2015074079A1 (en) 2013-11-18 2015-05-21 ServiceSource International, Inc. User task focus and guidance for recurring revenue asset management
US10579647B1 (en) 2013-12-16 2020-03-03 Palantir Technologies Inc. Methods and systems for analyzing entity performance
US9552615B2 (en) 2013-12-20 2017-01-24 Palantir Technologies Inc. Automated database analysis to detect malfeasance
US10356032B2 (en) 2013-12-26 2019-07-16 Palantir Technologies Inc. System and method for detecting confidential information emails
US8832832B1 (en) 2014-01-03 2014-09-09 Palantir Technologies Inc. IP reputation
US10026114B2 (en) * 2014-01-10 2018-07-17 Betterdoctor, Inc. System for clustering and aggregating data from multiple sources
US8837835B1 (en) 2014-01-20 2014-09-16 Array Technology, LLC Document grouping system
US10262362B1 (en) 2014-02-14 2019-04-16 Experian Information Solutions, Inc. Automatic generation of code for attributes
US9009827B1 (en) 2014-02-20 2015-04-14 Palantir Technologies Inc. Security sharing system
US10002149B2 (en) * 2014-02-22 2018-06-19 SourceThought, Inc. Relevance ranking for data and transformations
US10127229B2 (en) * 2014-04-23 2018-11-13 Elsevier B.V. Methods and computer-program products for organizing electronic documents
WO2015161899A1 (en) * 2014-04-25 2015-10-29 Hewlett Packard Development Company L.P. Determine relationships between entities in datasets
US9535974B1 (en) 2014-06-30 2017-01-03 Palantir Technologies Inc. Systems and methods for identifying key phrase clusters within documents
US9619557B2 (en) 2014-06-30 2017-04-11 Palantir Technologies, Inc. Systems and methods for key phrase characterization of documents
US9202249B1 (en) 2014-07-03 2015-12-01 Palantir Technologies Inc. Data item clustering and analysis
US9256664B2 (en) 2014-07-03 2016-02-09 Palantir Technologies Inc. System and method for news events detection and visualization
US10572496B1 (en) 2014-07-03 2020-02-25 Palantir Technologies Inc. Distributed workflow system and database with access controls for city resiliency
US9021260B1 (en) 2014-07-03 2015-04-28 Palantir Technologies Inc. Malware data item analysis
US9785773B2 (en) 2014-07-03 2017-10-10 Palantir Technologies Inc. Malware data item analysis
US10387389B2 (en) * 2014-09-30 2019-08-20 International Business Machines Corporation Data de-duplication
US9984166B2 (en) * 2014-10-10 2018-05-29 Salesforce.Com, Inc. Systems and methods of de-duplicating similar news feed items
US10592841B2 (en) 2014-10-10 2020-03-17 Salesforce.Com, Inc. Automatic clustering by topic and prioritizing online feed items
US11488086B2 (en) 2014-10-13 2022-11-01 ServiceSource International, Inc. User interface and underlying data analytics for customer success management
US9043894B1 (en) 2014-11-06 2015-05-26 Palantir Technologies Inc. Malicious software detection in a computing system
US9483546B2 (en) * 2014-12-15 2016-11-01 Palantir Technologies Inc. System and method for associating related records to common entities across multiple lists
US10552994B2 (en) 2014-12-22 2020-02-04 Palantir Technologies Inc. Systems and interactive user interfaces for dynamic retrieval, analysis, and triage of data items
US9367872B1 (en) 2014-12-22 2016-06-14 Palantir Technologies Inc. Systems and user interfaces for dynamic and interactive investigation of bad actor behavior based on automatic clustering of related data in various data structures
US10362133B1 (en) 2014-12-22 2019-07-23 Palantir Technologies Inc. Communication data processing architecture
US9348920B1 (en) 2014-12-22 2016-05-24 Palantir Technologies Inc. Concept indexing among database of documents using machine learning techniques
US9817563B1 (en) 2014-12-29 2017-11-14 Palantir Technologies Inc. System and method of generating data points from one or more data stores of data items for chart creation and manipulation
US20160232606A1 (en) * 2015-02-11 2016-08-11 Mastercard International Inc Systems and Methods for Use in Providing Lending Products to Consumers
EP3268880B1 (en) * 2015-03-10 2022-11-02 Elekta, Inc. Adaptive treatment management system with a workflow management engine
US10103953B1 (en) 2015-05-12 2018-10-16 Palantir Technologies Inc. Methods and systems for analyzing entity performance
US10412465B2 (en) * 2015-06-08 2019-09-10 Sony Corporation System and method for controlling playback of digital content
US10042914B2 (en) 2015-06-10 2018-08-07 International Business Machines Corporation Database index for constructing large scale data level of details
US9454785B1 (en) 2015-07-30 2016-09-27 Palantir Technologies Inc. Systems and user interfaces for holistic, data-driven investigation of bad actor behavior based on clustering and scoring of related data
US9456000B1 (en) 2015-08-06 2016-09-27 Palantir Technologies Inc. Systems, methods, user interfaces, and computer-readable media for investigating potential malicious communications
US10489391B1 (en) 2015-08-17 2019-11-26 Palantir Technologies Inc. Systems and methods for grouping and enriching data items accessed from one or more databases for presentation in a user interface
US10296627B2 (en) 2015-08-18 2019-05-21 Fiserv, Inc. Generating integrated data records by correlating source data records from disparate data sources
US10127289B2 (en) * 2015-08-19 2018-11-13 Palantir Technologies Inc. Systems and methods for automatic clustering and canonical designation of related data in various data structures
US9485265B1 (en) 2015-08-28 2016-11-01 Palantir Technologies Inc. Malicious activity detection system capable of efficiently processing data accessed from databases and generating alerts for display in interactive user interfaces
US10360244B2 (en) 2015-09-24 2019-07-23 Liveramp, Inc. System and method for improving computational efficiency of consumer databases using household links
US10067972B2 (en) 2015-11-17 2018-09-04 International Business Machines Corporation Semantic database driven form validation
CN105893411A (en) * 2015-11-20 2016-08-24 乐视致新电子科技(天津)有限公司 Statistic data processing method and apparatus
US10901996B2 (en) 2016-02-24 2021-01-26 Salesforce.Com, Inc. Optimized subset processing for de-duplication
US10152497B2 (en) * 2016-02-24 2018-12-11 Salesforce.Com, Inc. Bulk deduplication detection
US10956450B2 (en) 2016-03-28 2021-03-23 Salesforce.Com, Inc. Dense subset clustering
US10949395B2 (en) 2016-03-30 2021-03-16 Salesforce.Com, Inc. Cross objects de-duplication
US10452627B2 (en) 2016-06-02 2019-10-22 International Business Machines Corporation Column weight calculation for data deduplication
JP6772606B2 (en) * 2016-07-13 2020-10-21 富士通株式会社 Data processing programs, data processing methods, and data processing equipment
US11106692B1 (en) 2016-08-04 2021-08-31 Palantir Technologies Inc. Data record resolution and correlation system
JP6734538B2 (en) * 2016-10-06 2020-08-05 富士通株式会社 Evaluation program, evaluation method, and evaluation device
US10545963B2 (en) * 2016-10-31 2020-01-28 Servicenow, Inc. Generating a priority list of records using indicator data
US20180137667A1 (en) 2016-11-14 2018-05-17 Oracle International Corporation Graph Visualization Tools With Summary Visualization For Very Large Labeled Graphs
US10318630B1 (en) 2016-11-21 2019-06-11 Palantir Technologies Inc. Analysis of large bodies of textual data
US11205103B2 (en) 2016-12-09 2021-12-21 The Research Foundation for the State University Semisupervised autoencoder for sentiment analysis
CN106777070B (en) * 2016-12-12 2020-06-26 江苏师范大学 Web record link system and method based on block
US10620618B2 (en) 2016-12-20 2020-04-14 Palantir Technologies Inc. Systems and methods for determining relationships between defects
US20180181646A1 (en) * 2016-12-26 2018-06-28 Infosys Limited System and method for determining identity relationships among enterprise data entities
US9805073B1 (en) * 2016-12-27 2017-10-31 Palantir Technologies Inc. Data normalization system
US10552388B2 (en) * 2017-01-10 2020-02-04 Sap Se Null replacing attributes
US10558817B2 (en) * 2017-01-30 2020-02-11 Foley & Lardner LLP Establishing a link between identifiers without disclosing specific identifying information
US11227001B2 (en) 2017-01-31 2022-01-18 Experian Information Solutions, Inc. Massive scale heterogeneous data ingestion and user resolution
US10614102B2 (en) * 2017-02-23 2020-04-07 Innoplexus Ag Method and system for creating entity records using existing data sources
US10614083B2 (en) * 2017-02-28 2020-04-07 Innoplexus Ag Method and system for identifying incipient field-specific entity records
US10614136B2 (en) * 2017-03-02 2020-04-07 Innoplexus Ag Method and system for managing primary field-specific entity records required by user
US10325224B1 (en) 2017-03-23 2019-06-18 Palantir Technologies Inc. Systems and methods for selecting machine learning training data
US10606866B1 (en) 2017-03-30 2020-03-31 Palantir Technologies Inc. Framework for exposing network activities
US10235461B2 (en) 2017-05-02 2019-03-19 Palantir Technologies Inc. Automated assistance for generating relevant and valuable search results for an entity of interest
US10482382B2 (en) 2017-05-09 2019-11-19 Palantir Technologies Inc. Systems and methods for reducing manufacturing failure rates
US10585575B2 (en) 2017-05-31 2020-03-10 Oracle International Corporation Visualizing UI tool for graph construction and exploration with alternative action timelines
CN107515908A (en) * 2017-08-11 2017-12-26 新智数通(北京)技术服务有限公司 A kind of data processing method and device
WO2019035903A1 (en) 2017-08-16 2019-02-21 Walmart Apollo, Llc Systems and methods for distributed data validation
WO2019089888A1 (en) 2017-11-01 2019-05-09 Walmart Apollo, Llc Systems and methods for dynamic hierarchical metadata storage and retrieval
CN107784115B (en) * 2017-11-09 2019-01-22 福建省特种设备检验研究院 A kind of special equipment failure analysis methods and system based on editing distance algorithm
CN110019218B (en) * 2017-12-08 2023-08-25 阿里巴巴集团控股有限公司 Data storage and query method and equipment
US10838987B1 (en) 2017-12-20 2020-11-17 Palantir Technologies Inc. Adaptive and transparent entity screening
US11605018B2 (en) 2017-12-27 2023-03-14 Cerner Innovation, Inc. Ontology-guided reconciliation of electronic records
US10810258B1 (en) * 2018-01-04 2020-10-20 Amazon Technologies, Inc. Efficient graph tree based address autocomplete and autocorrection
US10949465B1 (en) 2018-01-04 2021-03-16 Amazon Technologies, Inc. Efficient graph tree based address autocomplete and autocorrection
CN108334570A (en) * 2018-01-19 2018-07-27 上海达梦数据库有限公司 Method, apparatus, server and the storage medium of hierarchical query
CN108334571A (en) * 2018-01-19 2018-07-27 上海达梦数据库有限公司 Method, apparatus, server and the storage medium of hierarchical query
CA3036664A1 (en) * 2018-03-14 2019-09-14 Royal Bank Of Canada Method for data structure relationship detection
GB2572546A (en) * 2018-03-27 2019-10-09 Innoplexus Ag System and method of creating entity records
US11120082B2 (en) 2018-04-18 2021-09-14 Oracle International Corporation Efficient, in-memory, relational representation for heterogeneous graphs
SG10201803391SA (en) * 2018-04-23 2019-11-28 Trident Corporate Services Singapore Pte Ltd As Trustee For The Oneplace Unit Trust Method and system for managing a new record
US11288294B2 (en) * 2018-04-26 2022-03-29 Accenture Global Solutions Limited Natural language processing and artificial intelligence based search system
US11170046B2 (en) * 2018-05-29 2021-11-09 American Express Travel Related Services Company, Inc. Network node consolidation
US11119630B1 (en) 2018-06-19 2021-09-14 Palantir Technologies Inc. Artificial intelligence assisted evaluations and user interface for same
US11392627B2 (en) 2018-08-03 2022-07-19 Kilpatrick Townsend & Stockton Llp Identifying missing nodes within a graphically represented family
US11222050B2 (en) 2018-08-03 2022-01-11 Kilpatrick Townsend & Stockton Llp Graphically representing related patent families using a phantom parent node
US10963434B1 (en) 2018-09-07 2021-03-30 Experian Information Solutions, Inc. Data architecture for supporting multiple search models
US11210293B2 (en) * 2018-11-27 2021-12-28 Sap Se Systems and methods for associating data entries
US10922337B2 (en) * 2019-04-30 2021-02-16 Amperity, Inc. Clustering of data records with hierarchical cluster IDs
US11003643B2 (en) * 2019-04-30 2021-05-11 Amperity, Inc. Multi-level conflict-free entity clusterings
US10922481B2 (en) * 2019-06-14 2021-02-16 International Business Machines Corporation Visual user attention redirection while form filling to enhance auto-fill accuracy
US11520764B2 (en) 2019-06-27 2022-12-06 International Business Machines Corporation Multicriteria record linkage with surrogate blocking keys
US11086940B1 (en) * 2019-09-30 2021-08-10 Amazon Technologies, Inc. Scalable parallel elimination of approximately subsumed sets
US11113254B1 (en) 2019-09-30 2021-09-07 Amazon Technologies, Inc. Scaling record linkage via elimination of highly overlapped blocks
US11675805B2 (en) * 2019-12-16 2023-06-13 Cerner Innovation, Inc. Concept agnostic reconcilation and prioritization based on deterministic and conservative weight methods
US11360971B2 (en) * 2020-01-16 2022-06-14 Capital One Services, Llc Computer-based systems configured for entity resolution for efficient dataset reduction
US11328796B1 (en) 2020-02-25 2022-05-10 Vignet Incorporated Techniques for selecting cohorts for decentralized clinical trials for pharmaceutical research
US11605038B1 (en) 2020-05-18 2023-03-14 Vignet Incorporated Selecting digital health technology to achieve data collection compliance in clinical trials
US11461216B1 (en) 2020-05-18 2022-10-04 Vignet Incorporated Monitoring and improving data collection using digital health technology
US20220100857A1 (en) * 2020-09-28 2022-03-31 Elasticsearch B.V. Systems and Methods of Anomalous Pattern Discovery and Mitigation
US11789837B1 (en) 2021-02-03 2023-10-17 Vignet Incorporated Adaptive data collection in clinical trials to increase the likelihood of on-time completion of a trial
US11521714B1 (en) 2021-02-03 2022-12-06 Vignet Incorporated Increasing diversity of participants in health research using adaptive methods
US11361846B1 (en) 2021-02-03 2022-06-14 Vignet Incorporated Systems and methods for customizing monitoring programs involving remote devices
US11316941B1 (en) 2021-02-03 2022-04-26 Vignet Incorporated Remotely managing and adapting monitoring programs using machine learning predictions
US11296971B1 (en) 2021-02-03 2022-04-05 Vignet Incorporated Managing and adapting monitoring programs
US11196656B1 (en) 2021-02-03 2021-12-07 Vignet Incorporated Improving diversity in cohorts for health research
CN113037165B (en) * 2021-03-12 2023-01-03 上海金脉电子科技有限公司 Method and device for correcting flux linkage coefficient of permanent magnet synchronous motor
US11880377B1 (en) 2021-03-26 2024-01-23 Experian Information Solutions, Inc. Systems and methods for entity resolution
US11694276B1 (en) * 2021-08-27 2023-07-04 Bottomline Technologies, Inc. Process for automatically matching datasets
US20230110661A1 (en) * 2021-09-30 2023-04-13 T-Rex Solutions, Llc Method of linking records between databases
US20230334025A1 (en) * 2022-04-18 2023-10-19 Salesforce, Inc. Record management for database systems using fuzzy field matching
US11907652B2 (en) * 2022-06-02 2024-02-20 On Time Staffing, Inc. User interface and systems for document creation

Family Cites Families (186)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4814979A (en) * 1981-04-01 1989-03-21 Teradata Corporation Network to transmit prioritized subtask pockets to dedicated processors
US4445171A (en) * 1981-04-01 1984-04-24 Teradata Corporation Data processing systems and methods
US4769772A (en) * 1985-02-28 1988-09-06 Honeywell Bull, Inc. Automated query optimization method using both global and parallel local optimizations for materialization access planning for distributed databases
US4977599A (en) * 1985-05-29 1990-12-11 International Business Machines Corporation Speech recognition employing a set of Markov models that includes Markov models representing transitions to and from silence
US4925311A (en) * 1986-02-10 1990-05-15 Teradata Corporation Dynamically partitionable parallel processors
US4870568A (en) * 1986-06-25 1989-09-26 Thinking Machines Corporation Method for searching a database system including parallel processors
US4860201A (en) * 1986-09-02 1989-08-22 The Trustees Of Columbia University In The City Of New York Binary tree parallel processor
US6108763A (en) * 1986-09-18 2000-08-22 Grondalski; Robert S. Simultaneous parity generating/reading circuit for massively parallel processing systems
US5146590A (en) * 1989-01-13 1992-09-08 International Business Machines Corporation Method for sorting using approximate key distribution in a distributed system
US5471622A (en) 1989-10-04 1995-11-28 Paralogic, Inc. Run-time system having nodes for identifying parallel tasks in a logic program and searching for available nodes to execute the parallel tasks
DE69108900D1 (en) * 1990-01-30 1995-05-18 Johnson Service Co NETWORKED RESOURCE MANAGEMENT SYSTEM.
US5321813A (en) * 1991-05-01 1994-06-14 Teradata Corporation Reconfigurable, fault tolerant, multistage interconnect network and protocol
US5251131A (en) * 1991-07-31 1993-10-05 Thinking Machines Corporation Classification of data records by comparison of records to a training database using probability weights
US5555404A (en) * 1992-03-17 1996-09-10 Telenor As Continuously available database server having multiple groups of nodes with minimum intersecting sets of database fragment replicas
US5423037A (en) * 1992-03-17 1995-06-06 Teleserve Transaction Technology As Continuously available database server having multiple groups of nodes, each group maintaining a database copy with fragments stored on multiple nodes
EP0562251A2 (en) 1992-03-24 1993-09-29 Universities Research Association, Inc. Parallel data transfer network controlled by a dynamically reconfigurable serial network
FI97919C (en) 1992-06-05 1997-03-10 Nokia Mobile Phones Ltd Speech recognition method and system for a voice-controlled telephone
US5845113A (en) 1992-10-27 1998-12-01 International Business Machines Corporation Method for external sorting in shared-nothing parallel architectures
US5392430A (en) * 1992-10-30 1995-02-21 International Business Machines Hierarchical scheduling method for processing tasks having precedence constraints on a parallel processing system
JP2583010B2 (en) * 1993-01-07 1997-02-19 インターナショナル・ビジネス・マシーンズ・コーポレイション Method of maintaining consistency between local index table and global index table in multi-tier index structure
JP3266351B2 (en) * 1993-01-20 2002-03-18 株式会社日立製作所 Database management system and query processing method
US6556988B2 (en) * 1993-01-20 2003-04-29 Hitachi, Ltd. Database management apparatus and query operation therefor, including processing plural database operation requests based on key range of hash code
US5408649A (en) * 1993-04-30 1995-04-18 Quotron Systems, Inc. Distributed data access system including a plurality of database access processors with one-for-N redundancy
BR9405417A (en) 1993-06-07 1999-09-08 Radio Local Area Networks Inc Network interconnection controller, methods of designating a master node on a network, changing the frequency on a network, choosing an optimal transmission power level for a first node and transmitting a plurality of data packets
DE4323241A1 (en) * 1993-07-12 1995-02-02 Ibm Method and computer system for finding incorrect character strings in a text
EP0746816B1 (en) 1993-08-03 2001-10-24 Sun Microsystems, Inc. Flexible multi-platform partitioning for computer applications
DE4497320T1 (en) * 1993-09-27 1996-11-21 Oracle Corp Method and device for parallel processing in a database system
US5495606A (en) * 1993-11-04 1996-02-27 International Business Machines Corporation System for parallel processing of complex read-only database queries using master and slave central processor complexes
US7599910B1 (en) * 1993-11-16 2009-10-06 Hitachi, Ltd. Method and system of database divisional management for parallel database system
US5590319A (en) 1993-12-15 1996-12-31 Information Builders, Inc. Query processor for parallel processing in homogenous and heterogenous databases
US5742806A (en) 1994-01-31 1998-04-21 Sun Microsystems, Inc. Apparatus and method for decomposing database queries for database management system including multiprocessor digital data processing system
JP2708386B2 (en) * 1994-03-18 1998-02-04 インターナショナル・ビジネス・マシーンズ・コーポレイション Method and apparatus for recovering duplicate database through simultaneous update and copy procedure
DE4420659C2 (en) * 1994-06-14 1997-03-13 Agfa Gevaert Ag Device for conveying photosensitive photographic films from cartridges
EP0691623B1 (en) * 1994-07-04 2001-10-04 Hewlett-Packard Company, A Delaware Corporation Scribble matching
US5694593A (en) 1994-10-05 1997-12-02 Northeastern University Distributed computer database system and method
US5732400A (en) * 1995-01-04 1998-03-24 Citibank N.A. System and method for a risk-based purchase of goods
US5799323A (en) * 1995-01-24 1998-08-25 Tandem Computers, Inc. Remote duplicate databased facility with triple contingency protection
US5754941A (en) * 1995-02-06 1998-05-19 Broadband Technologies, Inc. Point-to-multipoint broadband services drop with multiple time slot return channel for customer premises equipment served by fiber optic telecommunication system employing STS-based transmission format containing asynchronous transfer mode cells
US5655080A (en) * 1995-08-14 1997-08-05 International Business Machines Corporation Distributed hash group-by cooperative processing
US5745746A (en) * 1996-06-24 1998-04-28 International Business Machines Corporation Method for localizing execution of subqueries and determining collocation of execution of subqueries in a parallel database
CA2159269C (en) 1995-09-27 2000-11-21 Chaitanya K. Baru Method and apparatus for achieving uniform data distribution in a parallel database system
US5692174A (en) 1995-10-05 1997-11-25 International Business Machines Corporation Query parallelism in a shared data DBMS system
US5754841A (en) * 1995-10-20 1998-05-19 Ncr Corporation Method and apparatus for parallel execution of user-defined functions in an object-relational database management system
US5758314A (en) * 1996-05-21 1998-05-26 Sybase, Inc. Client/server database system with methods for improved soundex processing in a heterogeneous language environment
US5806032A (en) * 1996-06-14 1998-09-08 Lucent Technologies Inc. Compilation of weighted finite-state transducers from decision trees
US5877840A (en) * 1996-09-20 1999-03-02 Sanyo Electric Co., Ltd. Binocular view function inspecting apparatus and inspecting method
GB9623472D0 (en) * 1996-11-12 1997-01-08 Int Computers Ltd Data transfer mechanism
US7634529B2 (en) * 1996-11-29 2009-12-15 Ellis Iii Frampton E Personal and server computers having microchips with multiple processing units and internal firewalls
US5878408A (en) * 1996-12-06 1999-03-02 International Business Machines Corporation Data management system and process
GB2323944A (en) * 1997-04-01 1998-10-07 Ibm Parallel data processing system
US6026304A (en) * 1997-01-08 2000-02-15 U.S. Wireless Corporation Radio transmitter location finding for wireless communication network services and management
US5884299A (en) * 1997-02-06 1999-03-16 Ncr Corporation Optimization of SQL queries involving aggregate expressions using a plurality of local and global aggregation operations
JPH10232875A (en) 1997-02-19 1998-09-02 Hitachi Ltd Data base managing method and parallel data base managing system
WO1998036712A1 (en) * 1997-02-20 1998-08-27 The Cleveland Clinic Foundation Cranial remodeling headpiece
US6295533B2 (en) * 1997-02-25 2001-09-25 At&T Corp. System and method for accessing heterogeneous databases
US5920845A (en) * 1997-04-04 1999-07-06 Risemberg; Rafael Date matching methods
US6192391B1 (en) * 1997-05-30 2001-02-20 Nec Corporation Process stop method and apparatus for a distributed memory multi-processor system
US5897638A (en) * 1997-06-16 1999-04-27 Ab Initio Software Corporation Parallel virtual file system
US6081801A (en) * 1997-06-30 2000-06-27 International Business Machines Corporation Shared nothing parallel execution of procedural constructs in SQL
US6523041B1 (en) * 1997-07-29 2003-02-18 Acxiom Corporation Data linking system and method using tokens
US6006249A (en) 1997-08-19 1999-12-21 The Chase Manhattan Bank Method and apparatus for concurrent data processing
US6199069B1 (en) * 1997-08-25 2001-03-06 International Business Machines Corporation System and method for switching between databases without disruption to applications
US6026398A (en) * 1997-10-16 2000-02-15 Imarket, Incorporated System and methods for searching and matching databases
US6266804B1 (en) * 1997-12-23 2001-07-24 Ab Initio Software Corporation Method for analyzing capacity of parallel processing systems
US6963871B1 (en) 1998-03-25 2005-11-08 Language Analysis Systems, Inc. System and method for adaptive multi-cultural searching and matching of personal names
US6426947B1 (en) * 1998-10-21 2002-07-30 Kim K. Banker Apparatus and method for unilateral topology discovery in network management
US6304882B1 (en) 1998-05-05 2001-10-16 Informix Software, Inc. Data replication system and method
US6311169B2 (en) 1998-06-11 2001-10-30 Consumer Credit Associates, Inc. On-line consumer credit data reporting system
US6742003B2 (en) * 2001-04-30 2004-05-25 Microsoft Corporation Apparatus and accompanying methods for visualizing clusters of data and hierarchical cluster classifications
AU5233099A (en) * 1998-07-24 2000-02-14 Jarg Corporation Search system and method based on multiple ontologies
CN1324464A (en) * 1998-08-26 2001-11-28 西姆泰克有限公司 Method and devices for mapping data files
US6330552B1 (en) 1998-09-28 2001-12-11 Compaq Database query cost model optimizer
US6427148B1 (en) * 1998-11-09 2002-07-30 Compaq Computer Corporation Method and apparatus for parallel sorting using parallel selection/partitioning
US6535728B1 (en) * 1998-11-18 2003-03-18 Lightbridge, Inc. Event manager for use in fraud detection
US6430552B1 (en) * 1998-12-24 2002-08-06 Microsoft Corporation Method for converting queries with logical operators into free text queries
US6360361B1 (en) * 1999-03-15 2002-03-19 Microsoft Corporation Field reordering to optimize cache utilization
US6374241B1 (en) * 1999-03-31 2002-04-16 Verizon Laboratories Inc. Data merging techniques
US6252946B1 (en) * 1999-06-08 2001-06-26 David A. Glowny System and method for integrating call record information
US6594640B1 (en) * 1999-06-23 2003-07-15 Richard Postrel System for electronic barter, trading and redeeming points accumulated in frequent use reward programs
US6658412B1 (en) * 1999-06-30 2003-12-02 Educational Testing Service Computer-based method and system for linking records in data files
US20010032029A1 (en) 1999-07-01 2001-10-18 Stuart Kauffman System and method for infrastructure design
US6389429B1 (en) * 1999-07-30 2002-05-14 Aprimo, Inc. System and method for generating a target database from one or more source databases
GB2371901B (en) * 1999-09-21 2004-06-23 Andrew E Borthwick A probabilistic record linkage model derived from training data
US6523019B1 (en) * 1999-09-21 2003-02-18 Choicemaker Technologies, Inc. Probabilistic record linkage model derived from training data
US20020007284A1 (en) * 1999-12-01 2002-01-17 Schurenberg Kurt B. System and method for implementing a global master patient index
US6594651B2 (en) * 1999-12-22 2003-07-15 Ncr Corporation Method and apparatus for parallel execution of SQL-from within user defined functions
US6546403B1 (en) * 2000-01-19 2003-04-08 International Business Machines Corporation Mechanism to resubmit queries in a parallel database system
US6983317B1 (en) * 2000-02-28 2006-01-03 Microsoft Corporation Enterprise management system
US20040006447A1 (en) * 2000-06-22 2004-01-08 Jacky Gorin Methods and apparatus for test process enhancement
US20020065870A1 (en) * 2000-06-30 2002-05-30 Tom Baehr-Jones Method and apparatus for heterogeneous distributed computation
US6423041B1 (en) * 2000-07-31 2002-07-23 Candace Lynn Grant Catheter caddy
US20050154664A1 (en) * 2000-08-22 2005-07-14 Guy Keith A. Credit and financial information and management system
WO2002025588A2 (en) * 2000-09-21 2002-03-28 Md Online Inc. Medical image processing systems
US20020073138A1 (en) * 2000-12-08 2002-06-13 Gilbert Eric S. De-identification and linkage of data records
US20020073099A1 (en) * 2000-12-08 2002-06-13 Gilbert Eric S. De-identification and linkage of data records
US6938053B2 (en) * 2001-03-02 2005-08-30 Vality Technology Incorporated Categorization based on record linkage theory
US7076485B2 (en) * 2001-03-07 2006-07-11 The Mitre Corporation Method and system for finding similar records in mixed free-text and structured data
US6615211B2 (en) * 2001-03-19 2003-09-02 International Business Machines Corporation System and methods for using continuous optimization for ordering categorical data sets
US6912540B2 (en) 2001-05-04 2005-06-28 Ncr Corporation Chaining database records that represent a single customer or multiple customers living in a household
US7246125B2 (en) * 2001-06-21 2007-07-17 Microsoft Corporation Clustering of databases having mixed data attributes
US7028024B1 (en) * 2001-07-20 2006-04-11 Vignette Corporation Information retrieval from a collection of information objects tagged with hierarchical keywords
US20030033263A1 (en) 2001-07-31 2003-02-13 Reel Two Limited Automated learning system
US7251625B2 (en) * 2001-10-02 2007-07-31 Best Buy Enterprise Services, Inc. Customer identification system and method
US7295995B1 (en) * 2001-10-30 2007-11-13 A9.Com, Inc. Computer processes and systems for adaptively controlling the display of items
US6859803B2 (en) * 2001-11-13 2005-02-22 Koninklijke Philips Electronics N.V. Apparatus and method for program selection utilizing exclusive and inclusive metadata searches
US6826568B2 (en) * 2001-12-20 2004-11-30 Microsoft Corporation Methods and system for model matching
US20030126156A1 (en) * 2001-12-21 2003-07-03 Stoltenberg Jay A. Duplicate resolution system and method for data management
WO2003067497A1 (en) 2002-02-04 2003-08-14 Cataphora, Inc A method and apparatus to visually present discussions for data mining purposes
US20030154285A1 (en) * 2002-02-13 2003-08-14 International Business Machines Corporation Method and system for assigning network addreses
US6934714B2 (en) * 2002-03-04 2005-08-23 Intelesis Engineering, Inc. Method and system for identification and maintenance of families of data records
US7152060B2 (en) * 2002-04-11 2006-12-19 Choicemaker Technologies, Inc. Automated database blocking and record matching
US6990503B1 (en) * 2002-04-12 2006-01-24 Ncr Corporation Rescheduling transactions in a database system
US20040064447A1 (en) * 2002-09-27 2004-04-01 Simske Steven J. System and method for management of synonymic searching
US7246113B2 (en) 2002-10-02 2007-07-17 General Electric Company Systems and methods for selecting a material that best matches a desired set of properties
GB0224589D0 (en) * 2002-10-22 2002-12-04 British Telecomm Method and system for processing or searching user records
US20040088322A1 (en) * 2002-10-31 2004-05-06 International Business Machines Corporation System and method for determining connections between information aggregates
US7293024B2 (en) * 2002-11-14 2007-11-06 Seisint, Inc. Method for sorting and distributing data among a plurality of nodes
US8676843B2 (en) 2002-11-14 2014-03-18 LexiNexis Risk Data Management Inc. Failure recovery in a parallel-processing database system
US6968335B2 (en) * 2002-11-14 2005-11-22 Sesint, Inc. Method and system for parallel processing of database queries
US7240059B2 (en) * 2002-11-14 2007-07-03 Seisint, Inc. System and method for configuring a parallel-processing database system
US7945581B2 (en) * 2002-11-14 2011-05-17 Lexisnexis Risk Data Management, Inc. Global-results processing matrix for processing queries
US7185003B2 (en) * 2002-11-14 2007-02-27 Seisint, Inc. Query scheduling in a parallel-processing database system
US20040107205A1 (en) * 2002-12-03 2004-06-03 Lockheed Martin Corporation Boolean rule-based system for clustering similar records
US7403942B1 (en) * 2003-02-04 2008-07-22 Seisint, Inc. Method and system for processing data records
US7657540B1 (en) 2003-02-04 2010-02-02 Seisint, Inc. Method and system for linking and delinking data records
US7912842B1 (en) * 2003-02-04 2011-03-22 Lexisnexis Risk Data Management Inc. Method and system for processing and linking data records
US8166033B2 (en) * 2003-02-27 2012-04-24 Parity Computing, Inc. System and method for matching and assembling records
US8103753B2 (en) * 2003-04-22 2012-01-24 Microsoft Corporation Distributing membership information for multi-party application layer sessions
US7792746B2 (en) * 2003-07-25 2010-09-07 Oracle International Corporation Method and system for matching remittances to transactions based on weighted scoring and fuzzy logic
US7391885B2 (en) 2003-07-30 2008-06-24 Xerox Corporation Method for determining overall effectiveness of a document
US7305404B2 (en) * 2003-10-21 2007-12-04 United Parcel Service Of America, Inc. Data structure and management system for a superset of relational databases
JP4419531B2 (en) * 2003-11-20 2010-02-24 日産自動車株式会社 VEHICLE DRIVE OPERATION ASSISTANCE DEVICE AND VEHICLE HAVING VEHICLE DRIVE OPERATION ASSISTANCE DEVICE
US7343366B2 (en) * 2004-03-30 2008-03-11 International Business Machines Corporation Group-By result size estimation
US7519587B2 (en) * 2004-07-02 2009-04-14 Goldman Sachs & Co. Method, system, apparatus, program code, and means for determining a relevancy of information
US8892571B2 (en) * 2004-10-12 2014-11-18 International Business Machines Corporation Systems for associating records in healthcare database with individuals
US20080077570A1 (en) * 2004-10-25 2008-03-27 Infovell, Inc. Full Text Query and Search Systems and Method of Use
US7343369B2 (en) * 2004-11-18 2008-03-11 International Business Machines Corporation Method and apparatus for predicting selectivity of database query join conditions using hypothetical query predicates having skewed value constants
JP4403971B2 (en) * 2005-01-13 2010-01-27 オムロン株式会社 Planar antenna
US20060184584A1 (en) * 2005-02-11 2006-08-17 Microsoft Corporation Contact merge auto-suggest
JP4721740B2 (en) 2005-03-23 2011-07-13 富士通株式会社 Program for managing articles or topics
AU2006252583B2 (en) * 2005-05-31 2010-04-22 Siemens Medical Solutions Usa, Inc. System and method for data sensitive filtering of patient demographic record queries
JP4335849B2 (en) * 2005-06-13 2009-09-30 富士通マイクロエレクトロニクス株式会社 Imaging device capable of flicker detection
US8112441B2 (en) * 2005-07-15 2012-02-07 Indxit Sytems Inc. Systems and methods for data indexing and processing
US7539723B2 (en) * 2005-07-28 2009-05-26 International Business Machines Corporation System for configuring a cellular telephone to operate according to policy guidelines of a group of users
GB2430507A (en) * 2005-09-21 2007-03-28 Stephen Robert Ives System for managing the display of sponsored links together with search results on a mobile/wireless device
US7657506B2 (en) * 2006-01-03 2010-02-02 Microsoft International Holdings B.V. Methods and apparatus for automated matching and classification of data
WO2007084790A2 (en) 2006-01-20 2007-07-26 Glenbrook Associates, Inc. System and method for context-rich database optimized for processing of concepts
US7672942B2 (en) * 2006-05-01 2010-03-02 Sap, Ag Method and apparatus for matching non-normalized data values
US7664718B2 (en) 2006-05-16 2010-02-16 Sony Corporation Method and system for seed based clustering of categorical data using hierarchies
US7526486B2 (en) * 2006-05-22 2009-04-28 Initiate Systems, Inc. Method and system for indexing information about entities with respect to hierarchies
US8332366B2 (en) * 2006-06-02 2012-12-11 International Business Machines Corporation System and method for automatic weight generation for probabilistic matching
US7725421B1 (en) * 2006-07-26 2010-05-25 Google Inc. Duplicate account identification and scoring
US7627550B1 (en) * 2006-09-15 2009-12-01 Initiate Systems, Inc. Method and system for comparing attributes such as personal names
US20080133502A1 (en) * 2006-12-01 2008-06-05 Elena Gurevich System and method for utilizing multiple values of a search criteria
US7555478B2 (en) * 2006-12-05 2009-06-30 Yahoo! Inc. Search results presented as visually illustrative concepts
US20080140653A1 (en) * 2006-12-08 2008-06-12 Matzke Douglas J Identifying Relationships Among Database Records
US8160984B2 (en) * 2007-01-26 2012-04-17 Symphonyiri Group, Inc. Similarity matching of a competitor's products
EP2111593A2 (en) * 2007-01-26 2009-10-28 Information Resources, Inc. Analytic platform
US9262503B2 (en) * 2007-01-26 2016-02-16 Information Resources, Inc. Similarity matching of products based on multiple classification schemes
US7870151B2 (en) * 2007-02-07 2011-01-11 Fair Issac Corporation Fast accurate fuzzy matching
US20080208780A1 (en) * 2007-02-28 2008-08-28 Caterpillar Inc. System and method for evaluating documents
JP5042667B2 (en) 2007-03-05 2012-10-03 株式会社日立製作所 Information output device, information output method, and information output program
JP5088668B2 (en) * 2007-03-08 2012-12-05 日本電気株式会社 Computer load estimation system, computer load estimation method, computer load estimation program
US8144920B2 (en) * 2007-03-15 2012-03-27 Microsoft Corporation Automated location estimation using image analysis
WO2008121700A1 (en) * 2007-03-29 2008-10-09 Initiate Systems, Inc. Method and system for managing entities
WO2008121824A1 (en) 2007-03-29 2008-10-09 Initiate Systems, Inc. Method and system for data exchange among data sources
WO2008121170A1 (en) * 2007-03-29 2008-10-09 Initiate Systems, Inc. Method and system for parsing languages
US20090006431A1 (en) * 2007-06-29 2009-01-01 International Business Machines Corporation System and method for tracking database disclosures
US8019760B2 (en) * 2007-07-09 2011-09-13 Vivisimo, Inc. Clustering system and method
JP5306359B2 (en) * 2007-09-28 2013-10-02 インターナショナル・ビジネス・マシーンズ・コーポレーション Method and system for associating data records in multiple languages
WO2009042941A1 (en) * 2007-09-28 2009-04-02 Initiate Systems, Inc. Method and system for analysis of a system for matching data records
US8131759B2 (en) * 2007-10-18 2012-03-06 Asurion Corporation Method and apparatus for identifying and resolving conflicting data records
US20110001604A1 (en) * 2007-11-05 2011-01-06 Nelson Ludlow Automatic incident reporting in an access control system
US20090132514A1 (en) * 2007-11-16 2009-05-21 Iac Search & Media, Inc. method and system for building text descriptions in a search database
US7925652B2 (en) * 2007-12-31 2011-04-12 Mastercard International Incorporated Methods and systems for implementing approximate string matching within a database
US8019737B2 (en) * 2008-03-13 2011-09-13 Harris Corporation Synchronization of metadata
US7895171B2 (en) * 2008-03-27 2011-02-22 International Business Machines Corporation Compressibility estimation of non-unique indexes in a database management system
US8266168B2 (en) 2008-04-24 2012-09-11 Lexisnexis Risk & Information Analytics Group Inc. Database systems and methods for linking records and entity representations with sufficiently high confidence
US20090319515A1 (en) * 2008-06-02 2009-12-24 Steven Minton System and method for managing entity knowledgebases
GB2472358A (en) 2008-07-02 2011-02-02 Lexisnexis Risk & Information Analytics Group Inc Database system and methods
US8838549B2 (en) * 2008-07-07 2014-09-16 Chandra Bodapati Detecting duplicate records
US8422781B2 (en) * 2008-12-03 2013-04-16 Industrial Technology Research Institute Methods and systems for creating a hierarchical appearance model
US8150855B2 (en) * 2008-12-30 2012-04-03 International Business Machines Corporation Performing an efficient implicit join of multiple mixed-type records
US8554742B2 (en) * 2009-07-06 2013-10-08 Intelligent Medical Objects, Inc. System and process for record duplication analysis
US8583571B2 (en) * 2009-07-30 2013-11-12 Marchex, Inc. Facility for reconciliation of business records using genetic algorithms
US20110106807A1 (en) * 2009-10-30 2011-05-05 Janya, Inc Systems and methods for information integration through context-based entity disambiguation

Also Published As

Publication number Publication date
US9031979B2 (en) 2015-05-12
US20120290585A1 (en) 2012-11-15
US9836524B2 (en) 2017-12-05
US20140032594A1 (en) 2014-01-30
US20120173548A1 (en) 2012-07-05
CA2749310A1 (en) 2009-10-29
US20090271397A1 (en) 2009-10-29
US8498969B2 (en) 2013-07-30
US8495077B2 (en) 2013-07-23
US20130297635A1 (en) 2013-11-07
US8046362B2 (en) 2011-10-25
CA2749310C (en) 2015-06-02
US8489617B2 (en) 2013-07-16
WO2009132263A3 (en) 2010-01-07
US20140032557A1 (en) 2014-01-30
US8316047B2 (en) 2012-11-20
US20140250111A1 (en) 2014-09-04
US20150199418A1 (en) 2015-07-16
US8195670B2 (en) 2012-06-05
US20090287689A1 (en) 2009-11-19
US9043359B2 (en) 2015-05-26
US20090271404A1 (en) 2009-10-29
US8676838B2 (en) 2014-03-18
US20090271405A1 (en) 2009-10-29
US20090271694A1 (en) 2009-10-29
GB2472335A (en) 2011-02-02
US20090271359A1 (en) 2009-10-29
US20090292695A1 (en) 2009-11-26
US9037606B2 (en) 2015-05-19
US20090292694A1 (en) 2009-11-26
US9384262B2 (en) 2016-07-05
US20120173546A1 (en) 2012-07-05
GB201017876D0 (en) 2010-12-01
US8250078B2 (en) 2012-08-21
US8484168B2 (en) 2013-07-09
CA2749080A1 (en) 2009-10-29
US8135681B2 (en) 2012-03-13
US8135679B2 (en) 2012-03-13
US8572052B2 (en) 2013-10-29
US8266168B2 (en) 2012-09-11
CA2749080C (en) 2015-08-18
WO2009132263A2 (en) 2009-10-29
US8135719B2 (en) 2012-03-13
CA2723167C (en) 2012-10-30
US20160283575A1 (en) 2016-09-29
US20090271424A1 (en) 2009-10-29
US20120173545A1 (en) 2012-07-05
US8135680B2 (en) 2012-03-13
US20090271363A1 (en) 2009-10-29
US20140032556A1 (en) 2014-01-30
US20120278340A1 (en) 2012-11-01
US8275770B2 (en) 2012-09-25
US9020971B2 (en) 2015-04-28

Similar Documents

Publication Publication Date Title
CA2723167A1 (en) Database systems and methods for linking records corresponding to the same individual
US8112421B2 (en) Query selection for effectively learning ranking functions
US8489502B2 (en) Methods and systems for multi-credit reporting agency data modeling
Gal et al. Tuning the ensemble selection process of schema matchers
US6988090B2 (en) Prediction analysis apparatus and program storage medium therefor
CN103927615B (en) Entity is associated with classification
US8862638B2 (en) Interpolation data template to normalize analytic runs
CN109634924B (en) File system parameter automatic tuning method and system based on machine learning
US8768942B2 (en) Systems and methods for generating interpolated data sets converging to optimized results using iterative overlapping inputs
CN111612041A (en) Abnormal user identification method and device, storage medium and electronic equipment
CN108846338A (en) Polarization characteristic selection and classification method based on object-oriented random forest
Sabzevari et al. A comparison between statistical and data mining methods for credit scoring in case of limited available data
CN110310114A (en) Object classification method, device, server and storage medium
Khoshnevis et al. Prioritizing ground‐motion validation metrics using semisupervised and supervised learning
CN115686432B (en) Document evaluation method for retrieval sorting, storage medium and terminal
WO2023098034A1 (en) Business data report classification method and apparatus
Garciarena et al. Evolving imputation strategies for missing data in classification problems with TPOT
CN116235158A (en) System and method for implementing automated feature engineering
Kelly et al. A methodology for software cost estimation using machine learning techniques
CN113034264A (en) Method and device for establishing customer loss early warning model, terminal equipment and medium
Aliehyaei et al. Ant colony optimization, genetic programming and a hybrid approach for credit scoring: a comparative study
CN117435580B (en) Database parameter screening method and related equipment
TW202107348A (en) System and method for recommending audit criteria based on integration of qualitative data and quantitative data
Zhou Comparative Analysis of Machine Learning Performance in House Price Prediction for Ames Iowa
Aung et al. Classification of Rank for Distributors of Multi-Level Marketing Company by Using Decision Tree Induction

Legal Events

Date Code Title Description
EEER Examination request