US20060074973A1 - Managing media objects in a database - Google Patents
Managing media objects in a database Download PDFInfo
- Publication number
- US20060074973A1 US20060074973A1 US11/282,610 US28261005A US2006074973A1 US 20060074973 A1 US20060074973 A1 US 20060074973A1 US 28261005 A US28261005 A US 28261005A US 2006074973 A1 US2006074973 A1 US 2006074973A1
- Authority
- US
- United States
- Prior art keywords
- media object
- media objects
- database
- captured
- media
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/40—Information retrieval; Database structures therefor; File system structures therefor of multimedia data, e.g. slideshows comprising image and additional audio data
- G06F16/41—Indexing; Data structures therefor; Storage structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
- G06F16/58—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/40—Information retrieval; Database structures therefor; File system structures therefor of multimedia data, e.g. slideshows comprising image and additional audio data
- G06F16/48—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
- G06F16/53—Querying
- G06F16/535—Filtering based on additional data, e.g. user or group profiles
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
- G06F16/58—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
- G06F16/587—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using geographical or spatial information, e.g. location
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y10—TECHNICAL SUBJECTS COVERED BY FORMER USPC
- Y10S—TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y10S707/00—Data processing: database and file management or data structures
- Y10S707/99941—Database schema or data structure
- Y10S707/99944—Object-oriented database structure
Definitions
- the present invention relates to managing media objects in a database. More particularly, the present invention is directed to a method and apparatus for cataloging, searching and retrieving media objects in a database using contextual information to render inferences based upon known data, categories, indexes and searches. In addition, the present invention is directed to a method and apparatus for organizing media objects that automatically organizes the media objects. A clustering algorithm is used to automatically generate information whereby a user may easily identify media objects.
- media data includes any type of media or multimedia data including but not limited to still images, audio, video, graphics and the like. Because of this growth, there has been an increasing demand for methods and systems that enable a user to easily catalog, index and access the huge amounts of media data. However, unlike textual data, media data cannot be easily organized and searched using phrases, authors, and other traditional search terms. Techniques have been developed to address this problem with respect to still images.
- QBICTM query by content
- the QBICTM technique relies upon classifying an image according to a small number of pre-defined fixed image features, such as distribution of color across an image, shapes in an image, and textures in an image. Once these attributes are measured for each image, a sequence of numeric values is generated for that image.
- queries are made by providing an example of an image similar to that which the user desires, and then setting a weight for each characteristic that the user believes accurately reflects the presence of each attribute in the desired image as compared to that in the test image.
- the system compares the vector for the test image, modified by weights provided by the user, to the vector for each of the images in the database.
- Another technique for automating image retrieval includes mathematical techniques for looking for similarities in images.
- Yet another technique includes using the distribution of colors in an image to create a histogram of frequency of occurrences across a query image. The histogrammed distribution is measured for each image in the database, with an abstract-type distance measure used to compare the histogrammed results between each database image and the query image.
- the method and apparatus according to the present invention utilizes contextual information for a media object and known media objects, categories, indexes and searches, to arrive at an inference for cataloging the media object in a database and for creating an index of a collection.
- the media object may then be cataloged in the database according to the inference.
- a method and apparatus for searching and retrieving media objects in a database are provided.
- the method and apparatus for cataloging, searching and retrieving media objects from the database may be used in PC (Personal Computer) or Web-based (World Wide Web), or any network or computing applications that require the organization and accessibility of large amounts of media objects.
- the present invention can be used in any computer system.
- a method and apparatus for clustering media objects in which unlabeled data is grouped based on a similarity to other data.
- a distance metric may be used to identify related media objects.
- Media objects may be clustered based on time-based clustering, content-based clustering or based on a combination of both.
- FIG. 1 a block diagram of a general-purpose digital computing environment that may be used to implement various aspects of the present invention
- FIG. 2 is a flowchart illustrating a method of cataloging media objects in a database according to an aspect of the present invention
- FIG. 3 is a flowchart illustrating a method of cataloging media objects through categorization in a database according to another aspect of the present invention
- FIG. 4 is a flowchart illustrating a method of cataloging media objects through categorization in a database according to an aspect of the present invention
- FIG. 5 is a flowchart illustrating a method of cataloging media objects through categorization according to another aspect of the present invention
- FIG. 6 is a flowchart illustrating a method of cataloging media objects through categorization according to another aspect of the present invention.
- FIG. 7 is a flowchart illustrating a method of cataloging media objects through categorization according to another aspect of the present invention.
- FIG. 8 is a flowchart illustrating a method of searching a media database according to an aspect of the present invention.
- FIG. 9 is a flowchart illustrating a method of organizing media objects via clustering according to an aspect of the present invention.
- FIG. 10 is a flowchart illustrating a color clustering algorithm for media objects according to an aspect of the present invention.
- FIG. 11 is a flowchart illustrating temporal and color clustering of media objects according to an aspect of the present invention.
- FIG. 12 illustrates another graphical user interface according to an aspect of the present invention.
- FIG. 13 illustrates another graphical user interface according to an aspect of the present invention.
- FIG. 1 illustrates an example of a suitable computing system environment 100 on which the invention may be implemented.
- the computing system environment 100 is only one example of a suitable computing environment and is not intended to suggest any limitation as to the scope of use or functionality of the invention. Neither should the computing environment 100 be interpreted as having any dependency or requirement relating to any one or combination of components illustrated in the exemplary operating environment 100 .
- the invention is operational with numerous other general purpose or special purpose computing system environments or configurations.
- Examples of well known computing systems, environments, and/or configurations that may be suitable for use with the invention include, but are not limited to, personal computers (PCs), server computers, hand-held or laptop devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above-systems or devices, and the like.
- the invention may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer.
- program modules include routines, programs, objects, components, data structures, etc. that may perform particular tasks or implement particular abstract data types.
- the invention may also be practiced in distributed computing environments where tasks are preformed by remote processing devices that are linked through a communications network.
- program modules may be located in both local and remote computer storage media including memory storage devices.
- computer 100 includes a processing unit 110 , a system memory 120 and a system bus 130 that couples various system components including the system memory to the processing unit 110 .
- System bus 130 may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures.
- System memory 120 includes a read only memory (ROM) 140 and a random access memory (RAM) 150 .
- Computer 100 also includes a hard disk drive 170 for reading from and writing to a hard disk (not shown), a magnetic disk drive 180 for reading from or writing to a removable magnetic disk 190 , and an optical disk drive 191 for reading from or writing to a removable optical disk 192 , such as a CD ROM or other optical media.
- Hard disk drive 170 , magnetic disk drive 180 , and optical disk drive 191 are respectively connected to the system bus 130 by a hard disk drive interface 192 , a magnetic disk drive interface 193 , and an optical disk drive interface 194 .
- the drives and their associated computer-readable media provide nonvolatile storage of computer readable instructions, data structures, program modules and other data for personal computer 100 . It will be appreciated by those skilled in the art that other types of computer readable media which can store data that is accessible by a computer, such as magnetic cassettes, flash memory cards, digital video disks, Bernoulli cartridges, random access memories (RAMs), read only memories (ROMs), and the like, may also be used in the exemplary operating environment.
- RAMs random access memories
- ROMs read only memories
- a number of program modules can be stored on the hard disk, magnetic disk 190 , optical disk 192 , ROM 140 or RAM 150 , including an operating system 195 , one or more application programs 196 , other program modules 197 , and program data 198 .
- a user can enter commands and information into computer 100 through input devices, such as a keyboard 101 and a pointing device 102 .
- Other input devices may include a microphone, joystick, game pad, satellite dish, scanner, or the like.
- These and other input devices are often connected to the processing unit 110 through a serial port interface 106 that is coupled to the system bus, but may be connected by other interfaces, such as a parallel port, a game port or a universal serial bus (USB).
- USB universal serial bus
- a monitor 107 or other type of display device is also connected to system bus 130 via an interface, such as a video adapter 108 .
- personal computers typically include other peripheral output devices (not shown), such as speakers and printers.
- Computer 100 can operate in a networked environment using logical connections to one or more remote computers, as is well known in the art.
- Media objects may be supplied from a storage medium, such as a CD-ROM or a PC card, etc., or supplied directly from a capture device such as a camera or a scanner, or from the Internet or some other supply.
- Media objects are often stored in a database. Any database suitable for storing media data may be used.
- Cataloging refers to forming an index of a collection of media objects. Indexing refers to building a table or index that includes pointers to data. These pointers permit quick access to data without having to search and sort through information. Examples of index types that may be useful in the present invention include an index of metadata sorted by date, by photographer or other capturer, by longitude, by latitude, by zip code, by proximity to other objects, by subject matter or any other useful piece of information.
- Typical indexing methods include those used by databases, for example the known Indexed Sequential Access Method (ISAM).
- a catalog can be built as part of a file system or a database as appropriate for the desired method of access.
- a database Access by Microsoft Corp. can be used which provides for a data store, indexing on multiple keys in order to provide quick access to sets of data and methods of performing a query based on multiple factors to generate a report.
- the categories for the database may be standard categories within the cataloging system and/or categories that are set by the user.
- the database may include categories for cataloging the media objects such as particular holidays, vacations, locations, dates, and the like.
- Categorization contemplates the situation where a user pre-defines a set of labels representing a collection or category that a media object lies in such that a new object is automatically placed into one of those collections or categories. If a user does not pre-define a set of labels representing collections or categories, but all objects are automatically placed, the term for the placement is known as clustering.
- An index may be built by categorization or by clustering.
- the media objects in the database may be arranged in groups called “collections” where the media objects in a collection have similar content or relate to similar subject matter.
- the database may include collections such as a Holiday collection, a Vacation collection, a Birthday collection, a People collection, etc.
- Each collection may include a number of categories.
- the Holiday collection may have a Christmas category, a New Year's category, a Fourth of July category, etc. Keywords may be used to indicate what information is found in a media object stored under a particular collection and/or category or to index a media object.
- media objects may be cataloged (i.e., indexed and put into an organized format for future reference) based upon correlation of the data for the media object and/or data describing the media object (metadata) to be stored with known data and metadata.
- the data correlation may be used to automatically catalog large amounts of information. For example, a user may have accumulated a large number of images taken during a vacation. When an image is captured by a digital camera, for example, it may include a date and time stamp indicating the date and time that the image was captured. If the date of the image is the same as the date of other images that have been grouped under a particular collection/category, then an inference may be made that this particular image should also be grouped under the same particular collection/category.
- the images were taken on the same date, then it is likely that they represent the same event. Therefore, it may be inferred that the images be cataloged in the same collection/category of the database. The same may be true of any media objects.
- the location data for an image indicates that it is within a few feet of a location of images that were all cataloged under a category for Grandmother's house, an inference may be made that this image should also be cataloged under the category for Grandmother's house.
- Commonality between media objects may also be used in arriving at an inference as to how a media object should be cataloged. For example, if a number of images were captured on a particular day and a group of them include images of a particular person, then an inference may be made that these images should be grouped together.
- Cataloging according to the present invention involves taking known data that has been associated with attributes and labeled, for example, with “Grandmother's house” and using that known data to infer a label for other data.
- the invention utilizes image data and metadata for a media object to make inferences based on known data and metadata, and then performs cataloging based upon the inferences.
- the cataloging may, on occasion, require input from the user to confirm a particular inference. Therefore, according to the present invention, cataloging of media objects in a database may be substantially automatic.
- the cataloging method according to the present invention essentially involves four phases: data collection, data correlation, data refinement and data compaction. Each of these phases may be performed independently or in combination, at the same time, or at different times, and on one or more devices or databases. They may also be performed multiple times and in different orders of performance to achieve the desired results.
- Data collection includes gathering ancillary data before, during or after the capture of a media object. This may be done automatically or manually, and may be related to or result from another media object capture.
- the data may include information added by the user via an input device such as a keyboard, or it may be supplied from other external devices. For example, information from a GPS device indicating the location where the image was captured may be added, or audio may be added from memory or an external device. The user may add a textual description of the image.
- the ancillary data may be stored along with the media object.
- the media object database may be physically the same or different from the database for storing the information concerning the media object. Underlying database technologies can include any general database, for example SQL Server, Access or Exchange provided by Microsoft Corp. and as known in the art.
- a simple example of data collection may be recording the date and time of a photograph being taken.
- audio may be sampled before, during, and after the photograph for later analysis. For instance, a photograph taken at a birthday party may have “Happy Birthday” sung before and after a photograph is taken.
- the ancillary data may be obtained from another media object based upon an inference, as discussed below.
- Data correlation is the mechanism by which one or more pieces of data from the data collection phase are integrated and synthesized to form new data that may be used later to identify, organize, and search media objects. For example, by correlating GPS and date information, those photographs that were taken at a similar time and place may be detected. This may involve using an inexact search technique. Inexact searches can be performed using a variety of technologies, as is known to one skilled in the art. For example, techniques from information retrieval, inexact logic, or probabilistic searches may be used. In the birthday example noted above, data correlation may assist in determining which photograph would be associated with the “Happy Birthday” recording. In a batch-type process, each media object may be processed to determine how it should be cataloged. On the other hand, the media objects may be processed by determining all media objects relating to a particular event, time, location, etc. The correlation may be performed in a network environment to accommodate access to known information.
- Data refinement relates to the process of managing and extending the data correlation process to find more meaningful data and to reject false correlations. This may involve querying the user to determine the most appropriate correlation. For example, if audio is captured in the time between two meetings, a query to the user may request identification of whether the audio is related to the first or second meeting, or neither. Another example would be to reject the correlation between images taken by two cameras during the same period of time, but at very different locations. Data refinement may also be performed based on the past action of the user on similar data. Another possibility includes enabling the user to manually set the appropriate correlation. According to the present invention, information concerning whether the media object was manually or automatically cataloged may also be stored. When a media object has been cataloged manually, it may be assumed that it has been processed correctly.
- Data compaction relates to the process of removing redundant and/or useless data. Once an image is properly cataloged, indexed, and stored, much of the data used for cataloging may no longer be useful. As there can be a large amount of captured and inferred data, this phase may save a significant amount of storage space.
- the four most common means of doing this are linking (i.e., replacing duplicate information with a reference to that information in a common location), tokenizing or compressing (i.e., replacing data with a smaller, lossless representation), and hashing (replacing data with a unique identifier, from which the original data cannot be reconstructed). Each of these techniques is well known in the art.
- FIG. 2 is a flowchart that illustrates an example of the steps for performing a method according to an aspect of the present invention.
- Collections and categories of the database may be standard as provided with the system.
- the user may add collections and categories to personalize the database.
- a media object is captured, including any metadata and/or ancillary data that may be associated with the media object.
- the cataloging process is initiated in step S 201 .
- the cataloging method may be any one of a number of known methods that may be enhanced by the method according to the present invention.
- the processing may be performed on a server, on the capture device, personal computer and the like.
- the cataloging process according to the present invention may include processing the media object with respect to time, date, and/or location, for example.
- the processing may include comparing the relevant data with threshold data or performing an inexact search using known techniques, as shown in step S 202 .
- the data correlation process performed in step S 202 results in a pruned list of media objects in step S 203 .
- the pruning may require querying the user to determine the most appropriate correlation of the data.
- an inference may be made with respect to cataloging the media object based upon the data for the media objects in the pruned list.
- the media object may be cataloged in step S 205 based upon the inference made in step S 204 .
- the filename may be used to index the media object in the database. For example, the terms of the filename may be used as keywords to search the database. Alternatively, any appropriate known indexing may be used to search the database.
- step S 206 may be performed to remove redundant or useless data in order to reduce the amount of required storage space for the media object.
- any one of a number of known techniques may be used to perform step S 206 , such as linking, tokenizing or compressing and hashing.
- step S 207 the information for the image may be updated to indicate that the image was cataloged based upon an inference.
- FIG. 3 shows an additional level of processing according to another aspect of the present invention. More particularly, steps S 300 -S 303 in FIG. 3 correspond to steps S 200 -S 203 in FIG. 2 .
- step S 304 of FIG. 3 information in a global calendar may be used to catalog the media object.
- the processing in steps S 300 -S 303 may generate a pruned list of images at a particular location. Therefore, it may be inferred that the media object to be cataloged is also an image taken at the particular location.
- date information for the image may be compared with dates in the global calendar lookup in step S 304 .
- step S 305 If the date of the image corresponds to January 1st, it may be further inferred in step S 305 that the image at the particular location was taken on New Year's Day. The image may then be properly cataloged in step S 306 based upon this additional inference. Once the media object is cataloged in step S 306 , redundant or useless information may be removed in step S 307 .
- the cataloging may be performed with respect to media objects that are stored in the user's database of media objects. However, it may be desirable to utilize information other than that found in the user's database to automatically catalog a media object.
- the use of the global calendar lookup enables the cataloging process to utilize information other than that found in the user's database to catalog the image.
- information in a global map lookup, global address book, phonebook etc. may be useful in cataloging and indexing a media object.
- local lookups may also be possible. It will be appreciated by those of ordinary skill in the art that many other types of “lookups” may be possible for performing the method according to the present invention.
- FIG. 4 shows another aspect of the present invention.
- Steps S 400 -S 403 in FIG. 4 correspond to steps S 200 -S 203 in FIG. 2 .
- the media object may be processed to determine common features between the media object and media objects obtained in step S 403 .
- the pruned list in step S 403 may represent the images obtained in the last day.
- the image to be cataloged may be processed with respect to the images in the pruned list to find common features. In this example, it may be determined that the images all contain a particular person. An inference may then be made in step S 405 that the image should be cataloged under the people collection in the category corresponding to the particular person.
- step S 406 the image may be cataloged according to the inference in step S 405 .
- step S 407 redundant or useless information may be removed.
- FIG. 5 A particular example will be described with reference to FIG. 5 .
- the example is directed to cataloging images through the use of categorization algorithms.
- the examples set forth herein are for illustrative purposes only and those of ordinary skill in the art will appreciate that any media object(s) may be cataloged according to the present invention.
- a user may have accumulated a large number of images taken during the Christmas holidays. Information for these images may be used to distinguish them from other images so that they can be organized together in a database.
- a holiday collection may be arranged in the database having categories corresponding to particular holidays, respectively.
- One holiday category may correspond to “Christmas.” Images that were captured on December 24th or December 25th may be organized under the “Christmas” category.
- step S 500 an image is captured along with any data that may be associated with the image, including data added by the user or data supplied from some other external source or device.
- the date will be used for organizing the images.
- step S 501 the cataloging process is initiated. More particularly, in step S 502 , the date on which the image was captured may be detected.
- step S 503 the date is compared with the date in a global calendar lookup.
- step S 504 it is determined whether the date on which the image was captured is December 24 th . If the image was captured on December 24 th , in step S 505 an inference may be made that the image should be cataloged in the Christmas category of the Holiday collection.
- step S 506 the image may be cataloged based upon the inference.
- step S 507 redundant or useless information may be removed. If the date of the image does not correspond to December 24 th , it is determined in step 508 whether the date corresponds to December 25 th . If the answer is Yes in step S 508 , then steps S 505 -S 507 are performed. If the answer in step S 508 is No, then other processing is performed. In this example, in step S 509 , other media objects captured on the same date on which the image was captured are detected to arrive at a pruned list in step S 510 . An inference may be made in step S 511 as to how to catalog the image based upon the information for the media objects in the pruned list.
- step S 512 The image may then be cataloged in step S 512 based upon the inference. Finally, step S 507 may be performed to remove redundant or useless data.
- the filename Once the image is stored in a particular category, the filename may be used as keyword(s) for indexing the image in the database. Of course, one of ordinary skill in the art will appreciate that any appropriate indexing method may be used.
- step S 606 the image may be properly cataloged in step S 606 .
- the information for the image may be updated to indicate that the image was cataloged based upon an inference in step S 607 .
- the identification of inferred data may indicate that there is a certain level of unreliability of the information as compared to manually indexing the media object.
- Steps S 600 -S 604 and steps S 609 -S 613 are the same as steps S 500 -S 504 and S 508 -S 512 in FIG. 5 .
- the date and location may be correlated to organize images, as shown in FIG. 7 .
- step S 700 an image is captured along with any data that may be associated with the image, including data added by the user or data supplied from some other external source or device.
- the cataloging process is initiated in step S 701 .
- step S 702 it is determined whether date information is available. If the answer in step S 702 is Yes, then the date is detected in step S 703 . The date is then compared with a date range corresponding to the user's vacation, for example, in step S 704 . If the date is within the vacation range, then processing continues to step S 705 . In step S 705 , all of the images having the same date as the detected date are obtained from the database and added to the pruned list in step S 706 . If the answer in step S 704 is No, then processing ends.
- step S 713 it is determined whether location information is available. If the answer in step S 713 is Yes, then the location information for the image is detected in step S 714 . In step S 715 , it is determined whether the location information is within fifty miles of the vacation location. If the answer in step S 715 is Yes, then processing continues to step SS 716 . If the answer in step S 715 is No, the process ends. In step S 716 , all of the images at the detected location are detected. In step S 706 , all of the detected images are added to the pruned list. If the answer in step S 713 is No, then other cataloging processes are performed.
- the image may be compared with other media objects to detect common features between the image and any of the stored images in step S 717 .
- the images that include common features may be added to a pruned list in step S 718 .
- An inference may be made in step S 719 as to how to catalog the image based upon the information for the media objects in the pruned list.
- the image may then cataloged in step S 720 based upon the inference, and processing continues at step S 711 .
- step S 707 the images in the pruned list are processed to determine those images having common features with the image to be cataloged.
- step S 708 an inference is made on how the image should be cataloged based upon the information obtained in step S 707 .
- step S 709 date or location information may be added to information for the media object if either of this data is missing.
- step S 710 the image may be cataloged.
- step S 711 redundant or useless data may be removed.
- step S 712 the media object may be flagged as having been cataloged based upon inference(s).
- files in an operating system may be retrieved by searching by date, by file size, last modified, etc. It would be helpful to enable a user to search a database by location or category, for example.
- a user may want to search a database of images by searching all of the images taken at Grandparents' house.
- the search may be performed by going through all of the images and determining which images have a GPS location corresponding to the location of Grandparents' house.
- the search may be performed by determining which photos were taken during a trip to the Grandparents' house as listed in a local calendar. Therefore, according to an aspect of the present invention, searches may be done on the fly just as an indexing system or done as a way of setting up a search.
- a search term is identified, e.g., St. Patrick's Day, March 17 th , etc.
- a reference lookup is performed via a table of references. In the present example, the reference lookup may include looking up March 17, Leprechaun.
- a full search may be performed based on references obtained from reference lookup on step S 801 .
- the list of hits is displayed in step S 803 .
- the list may then be pruned in step S 804 according to known pruning techniques. For example, the list may be pruned to remove duplicates.
- the resulting search list is displayed in step S 805 .
- media objects such as still images
- the images in each collection often tend to be thematically related, in addition to temporally related.
- a user interface in this embodiment may include one image per collection, where the image is shown to the user. If the user is searching for an image, the user views the images respectively representing collections of images, and selects a collection that appears to relate to the desired image. Once a collection is selected, the images corresponding to the collection are shown to the user. In this way, the user can quickly find a desired image.
- step 900 the images in some predetermined directory structure, such as the “My Pictures” directory under the Windows® operating system (OS), for example, are sorted in ascending order by creation date.
- step S 901 a variable denoted “lasttime” is set to effective negative infinity (e.g., a quantity less than the earliest date represented under the OS).
- An empty current collection is created in step S 902 .
- the first (earliest created) image is selected for processing in step S 903 . Of course, processing could be performed beginning with the latest (last created) image.
- the creation time of the current image is compared to the lasttime variable in step S 904 .
- step S 904 If the difference between these two times is less than X hours, for some value of X, the two images are placed in the same collection.
- the value of X may be set by the user to any suitable time such as 1 hour, 24 hours, 15 minutes, etc. The shorter the time set for X, the more collections are created, which will tend to be more thematically related, but will require the user to scan through more collections. A default value for X may be set and the user may adjust the default value as desired. If the answer in step S 904 is Yes, then the image is stored in the current collection in step S 905 . If the answer in step S 904 is No, then the current image is not stored in the current collection. Rather, a new empty current collection is created in step S 910 , and processing continues to step S 905 .
- step S 906 the lasttime variable is updated to reflect the creation date of the current image stored in the current collection.
- step S 907 it is determined whether the current image is the last image in the sorted list of images. When the answer in step S 907 is No, then the next image in the sorted list is selected in step S 911 and processing returns to step S 904 . If the answer in step S 907 is Yes, then a representative image for the current collection is selected in step S 908 .
- the images in each collection may be sorted in order of creation time.
- the representative image for each album may be the image with the median creation time, i.e., the picture in the middle of the list of images in the collection sorted based upon creation time.
- step S 909 the view of the collection is generated.
- the threshold X not be a constant, but rather be computed adaptively based on the differences in creation time of temporally nearby images. Allowing the threshold X to be computed adaptively provides an advantage that the threshold will adapt to the typical behavior of the user and will identify the unusual temporal gaps between photographs.
- i indexes the media objects, (for example, photographs) in order of creation time
- t i is the creation time of photograph i
- X n is the threshold that is compared to the difference between t n and t n+1 .
- Formula 1 thus computes a running average of the log time gaps between photographs, in the example. If the indices i in the sum access creation times that do not exist for a media object, then the sum only runs over indices i that do, in fact exist. As a result, the value in the denominator, 20, is reduced to the number of terms in the sum.
- a flowchart which describes a color clustering algorithm for media objects (photographs in the present example). Color clustering is useful when the creation time of the media objects are either unreliable or unavailable.
- a color histogram for all images is created. The process of extracting color histograms is well known in the art and therefore will not be described in detail here.
- the color histograms are taken over all pixels of images that are downsampled by a factor of 4 or 8, thus increasing the speed of the histogram computation.
- the color histogram is taken in the CIE u′ v′ color space.
- the histogram By taking the color histogram in the CIE u′ v′ color space, pixel luminance is ignored. This is desirable because the pixel luminance does not predict the topic of the media object, here a photograph. Ideally, the histogram has 16 ⁇ 16 equally spaced bins in color space, spanning from 0.1612 to 0.2883 in u′ and 0.4361 to 0.5361 in v′. Pixels falling outside this area in color spaced are mapped to the nearest bin.
- the histogram may be estimated in a Bayesian manner, using maximum a posteriori methods and a uniform prior, as is well known in the art.
- the histogram is initialized with so-called “phantom data.” Before all pixels of an image are scanned, every bin is initialized to 1/256.
- a “number of pixel” counter is required to compute a histogram; the number of pixel counter is initialized the 1. By initializing the number of pixel counter to 1, no bin is allowed to attain a value of 0 which is desirable in the computation of the Jensen-Shannon distance in equation 4 of S 1004 , described below.
- the histogram is best estimated in a smoothed manner as is further known in the art.
- the histogram may be smoothed via so-called bilinear interpolation wherein every scanned pixel increases up to four histogram bins, according to standard bi-linear interpolation rules.
- the creation time for all images is extracted.
- JPEG images produced by digital cameras have image metadata stored in a format known as the EXIF format.
- the creation time for the photograph in the present example is stored as a filed in the EXIF format.
- Techniques for extracting the creation time are well known in the art.
- the creation time in the EXIF format is missing or incorrect. This can occur, for example, when the internal clock of a digital camera gets reset to an initial value when the internal battery of the camera runs down. By way of example, this situation might be detected by an EXIF creation time that occurred before a certain date, such as Jan. 1, 1999.
- the EXIF metadata might be entirely deleted after image capture by way of image manipulation tools.
- the file creation of modification date can be used as a substitute when that information is available, such as when a personal computer is being used.
- the ordering step S 1003 can be derived form the innate ordering of JPEG filenames produced by digital cameras, for example, P000006.JPG would be known to have been taken after P000005.JPG.
- the pairwise distance between color histograms of adjacent pairs of ordered images is computed.
- p i and q i are the ith color histogram bin values of two adjacent images (or collections). These probabilities are computed by taking the ratio between the number of pixels that fall into the ith bin (as modified by the known techniques of Bayesian inference and smoothing) to the total number of pixels that fall into all bins.
- r i n i + m i N + M ( 3 )
- the distance between the two images or clusters can thus be determined as follows: ⁇ i ⁇ ⁇ - ( n i + m i ) ⁇ log ⁇ ( r i ) + n i ⁇ log ⁇ ( p i ) + m i ⁇ log ⁇ ( q i ) ( 4 ) where a term that evaluates to 0 log 0 is treated as 0.
- other distance metrics can be used, including known techniques of chi-squared or L 1 norm.
- these distance metrics are computed between two images, but in step S 1012 , the same distance metric used in step S 1004 is computed between two clusters.
- Omohundro's best-first model merging technique is applied to clustering photographs.
- Best-first model merging is a clustering technique that starts with a set of items that are sorted into a sequence. The technique starts with each item in its own cluster. Then, at every step, the distance between every pair of adjacent clusters is computed. The pair with the smallest distance is merged together to yield a new cluster. This new cluster replaces the two old clusters in the sequence, and the clustering continues until the desired number of clusters is reached.
- all of the distances are placed into a heap data structure which permits finding the least element in O(logN) time, where N are the number of elements stored in the heap, as is well known in the art.
- the algorithm checks for termination by comparing the size of the heap (i.e., the number of elements in the heap) to the number of desired clusters.
- the number of desired clusters can be, for example, 1/12 the total number of photographs.
- the ratio between the number of clusters and the number of photographs is the compression ratio of representative images to all photographs in the user interface.
- Step S 1011 combines these two adjacent clusters (or images) to yield a new cluster.
- the new cluster contains pointers to all images contained in the previous two clusters.
- the histogram for the combined cluster is computed, as previously shown.
- Step S 1012 computes the distance from the new combined cluster to its two new neighboring clusters (or neighboring images). Again, the distance is computed between two adjacent clusters and the speculative cluster that results by combining the two adjacent clusters.
- step S 1013 the heap structure is rearranged in order to maintain the heap property, as is known in the art. Control then flows back to step S 1006 .
- step S 1006 if the correct number of clusters has been found, then control flows to step S 1007 where representative images are chosen from each cluster. For every cluster, the overall histogram that describes all images have been formed at either step S 1004 or S 1012 .
- the representative image is chosen from the images that belong to the cluster to minimize the Kulback-Leibler (KL) distance between the overall histogram r i and the histogram for the representative image s i : ⁇ i ⁇ r i ⁇ log ⁇ ( r i s i ) ( 5 )
- the representative image is the image whose color best mimics the overall color of the cluster.
- step S 1008 all of the representative images are displayed in a user interface.
- step S 1101 the pictures are sorted by creation date, similar to steps S 1101 and S 1102 in FIG. 11 .
- the temporal clustering algorithm is applied, as shown in FIG. 9 (except for S 908 and S 909 ) resulting in a number of clusters. These clusters are then scanned in steps S 1103 -S 1107 to check for failure of the temporal clustering. The existence of overly large clusters is evidence of temporal clustering failure.
- step S 1103 first sets the current cluster to the first cluster and step S 1104 counts the number of images in the current cluster. If the number is larger than a threshold X, step S 1105 will split the current cluster into sub-clusters according to the color algorithm described in FIG. 10 , except for steps S 1007 and S 1008 .
- the threshold X is 23. If the number is larger than the threshold X, the desired number of sub-clusters is preferably 1/12 of the number of photographs in the current cluster, rounded to the nearest integer.
- step S 1107 makes the current cluster the next cluster and control returns to step S 1104 . Otherwise, control flows to step S 1108 which computes representative images for all of the clusters (including both temporal and color clusters). This step is analogous to step S 1007 in FIG. 10 . In step S 1108 , the representative images are shown in the user interface. It is desirable to sort the representative images by increasing creation time prior to display.
- FIG. 12 shows a possible user interface.
- the user is presented with a user interface containing two panes.
- the left pane consists of thumbnails of representative images, represented collectively by 1201 in FIG. 12 .
- These thumbnails of representative images are of all of the clusters of the user's photographs, in the present example. These thumbnails are sorted by increasing creation time.
- the user can scroll through the representative images by using scroll bar 1202 as is known in the art.
- dates can be shown intermixed with or below each of the thumbnails.
- an optional scroll bar 1204 can be used to scroll through the contents of the cluster.
- a user clicks on a thumbnail in 1203 the user is shown a full-sized version of the image.
- the full-sized image can be a drop candidate or can be put on a clipboard, respectively.
- FIG. 13 A preferred user interface is shown in FIG. 13 . Again, two panes are presented in the user interface. In the right pane, thumbnails 1303 of all of the user's photographs are shown. The thumbnails in 1303 are sorted by increasing creation time. There is a scrollbar 1304 to allow the user to scroll through his or her photographs. Representative images are shown as thumbnails in 1301 in the left pane. The thumbnails in 1301 are sorted by increasing creation time also. These representative images form a table of contents for the entire collection on the right.
- the scrollbar 1304 When a user clicks on (or otherwise selects by known techniques) a representative thumbnail in 1301 in the left pane, the scrollbar 1304 is scrolled so that the same thumbnail is vertically centered in the right pane and highlighted with a colored boarder, so as to be noticeable.
- the left pane containing representative thumbnails 1301 are scrolled thus allowing the user to quickly scan the collection.
- the user interface of FIG. 13 is desirable because the user may not always realize which cluster a photograph belongs to.
- the scrollbar 1304 provides the user with a mechanism to easily correct for incorrect cluster guesses.
Abstract
A method and apparatus are provided for organizing media objects in a database using contextual information for a media object and known media objects, categories, indexes and searches, to arrive at an inference for cataloging the media object in a database. The media object may then be cataloged in the database according to the inference. A method and apparatus are provided for clustering media objects by forming groups of unlabeled data and applying a distance metric to said group.
Description
- This application is a continuation of pending U.S. application Ser. No. 10/021,255, filed Dec, 19, 2001, having the same title, which claims priority to U.S. Provisional No. 60/274,234, filed Mar. 9, 2001, entitled “Method And Apparatus For Cataloging And Searching Objects In A Database,” whose contents are expressly incorporated by reference.
- The present invention relates to managing media objects in a database. More particularly, the present invention is directed to a method and apparatus for cataloging, searching and retrieving media objects in a database using contextual information to render inferences based upon known data, categories, indexes and searches. In addition, the present invention is directed to a method and apparatus for organizing media objects that automatically organizes the media objects. A clustering algorithm is used to automatically generate information whereby a user may easily identify media objects.
- Advances in digital technology have resulted in the rapid growth of electronic media data such as still images, audio, video, graphics, and the like. For the purposes of this invention, media data includes any type of media or multimedia data including but not limited to still images, audio, video, graphics and the like. Because of this growth, there has been an increasing demand for methods and systems that enable a user to easily catalog, index and access the huge amounts of media data. However, unlike textual data, media data cannot be easily organized and searched using phrases, authors, and other traditional search terms. Techniques have been developed to address this problem with respect to still images.
- One semi-automated grouping and retrieval technique for still images developed by IBM Corporation is based on image content and involves a “query by content (QBIC™)” paradigm. The QBIC™ technique relies upon classifying an image according to a small number of pre-defined fixed image features, such as distribution of color across an image, shapes in an image, and textures in an image. Once these attributes are measured for each image, a sequence of numeric values is generated for that image. When searching the QBIC™ database, queries are made by providing an example of an image similar to that which the user desires, and then setting a weight for each characteristic that the user believes accurately reflects the presence of each attribute in the desired image as compared to that in the test image. To retrieve the image, the system compares the vector for the test image, modified by weights provided by the user, to the vector for each of the images in the database.
- Another technique for automating image retrieval includes mathematical techniques for looking for similarities in images. Yet another technique includes using the distribution of colors in an image to create a histogram of frequency of occurrences across a query image. The histogrammed distribution is measured for each image in the database, with an abstract-type distance measure used to compare the histogrammed results between each database image and the query image.
- The foregoing techniques suffer from the disadvantage of only addressing retrieval of still images. In addition, these techniques require significant user input and are time intensive.
- Therefore, there is a need for a method and apparatus for cataloging media objects in a database by forming an index to a collection of media objects. There is a need for a method and apparatus that provides organization of media objects using contextual information from which inferences may be drawn based upon known media objects, categories, indexes and searches. Further, a method and apparatus for searching and retrieving media objects from a database are needed. Moreover, there is a need for a method and apparatus for clustering media objects whereby media objects are automatically placed in a collection.
- The deficiencies and limitations of the prior art are overcome by the present invention which provides methods and apparatuses for managing media objects in a database.
- The method and apparatus according to the present invention utilizes contextual information for a media object and known media objects, categories, indexes and searches, to arrive at an inference for cataloging the media object in a database and for creating an index of a collection. The media object may then be cataloged in the database according to the inference.
- According to another aspect of the present invention, a method and apparatus for searching and retrieving media objects in a database are provided.
- According to yet another aspect of the present invention, the method and apparatus for cataloging, searching and retrieving media objects from the database may be used in PC (Personal Computer) or Web-based (World Wide Web), or any network or computing applications that require the organization and accessibility of large amounts of media objects. In addition, the present invention can be used in any computer system.
- According to another aspect of the present invention, a method and apparatus is provided for clustering media objects in which unlabeled data is grouped based on a similarity to other data. A distance metric may be used to identify related media objects. Media objects may be clustered based on time-based clustering, content-based clustering or based on a combination of both.
- These and other objects and features of the present invention will be apparent upon consideration of the following detailed description presented in connection with the following drawings in which like reference numerals identify like elements throughout.
-
FIG. 1 a block diagram of a general-purpose digital computing environment that may be used to implement various aspects of the present invention; -
FIG. 2 is a flowchart illustrating a method of cataloging media objects in a database according to an aspect of the present invention; -
FIG. 3 is a flowchart illustrating a method of cataloging media objects through categorization in a database according to another aspect of the present invention; -
FIG. 4 is a flowchart illustrating a method of cataloging media objects through categorization in a database according to an aspect of the present invention; -
FIG. 5 is a flowchart illustrating a method of cataloging media objects through categorization according to another aspect of the present invention; -
FIG. 6 is a flowchart illustrating a method of cataloging media objects through categorization according to another aspect of the present invention; -
FIG. 7 is a flowchart illustrating a method of cataloging media objects through categorization according to another aspect of the present invention; -
FIG. 8 is a flowchart illustrating a method of searching a media database according to an aspect of the present invention; -
FIG. 9 is a flowchart illustrating a method of organizing media objects via clustering according to an aspect of the present invention; -
FIG. 10 is a flowchart illustrating a color clustering algorithm for media objects according to an aspect of the present invention; -
FIG. 11 is a flowchart illustrating temporal and color clustering of media objects according to an aspect of the present invention; -
FIG. 12 illustrates another graphical user interface according to an aspect of the present invention; and -
FIG. 13 illustrates another graphical user interface according to an aspect of the present invention. - In order to facilitate the description of the present invention, the invention will be described with respect to the particular examples. The present invention, however, is not limited to any particular media object or limited by the examples described herein. Therefore, the description of the embodiment that follows is for purposes of illustration and not limitation.
-
FIG. 1 illustrates an example of a suitablecomputing system environment 100 on which the invention may be implemented. Thecomputing system environment 100 is only one example of a suitable computing environment and is not intended to suggest any limitation as to the scope of use or functionality of the invention. Neither should thecomputing environment 100 be interpreted as having any dependency or requirement relating to any one or combination of components illustrated in theexemplary operating environment 100. - The invention is operational with numerous other general purpose or special purpose computing system environments or configurations. Examples of well known computing systems, environments, and/or configurations that may be suitable for use with the invention include, but are not limited to, personal computers (PCs), server computers, hand-held or laptop devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above-systems or devices, and the like.
- The invention may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that may perform particular tasks or implement particular abstract data types. The invention may also be practiced in distributed computing environments where tasks are preformed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.
- In
FIG. 1 ,computer 100 includes aprocessing unit 110, asystem memory 120 and asystem bus 130 that couples various system components including the system memory to theprocessing unit 110.System bus 130 may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures.System memory 120 includes a read only memory (ROM) 140 and a random access memory (RAM) 150. - A basic input/output system (BIOS) 160 containing the basic routines that help to transfer information between elements within the
computer 100, such as during start-up, is stored in ROM 140.Computer 100 also includes ahard disk drive 170 for reading from and writing to a hard disk (not shown), amagnetic disk drive 180 for reading from or writing to a removablemagnetic disk 190, and anoptical disk drive 191 for reading from or writing to a removableoptical disk 192, such as a CD ROM or other optical media.Hard disk drive 170,magnetic disk drive 180, andoptical disk drive 191 are respectively connected to thesystem bus 130 by a harddisk drive interface 192, a magneticdisk drive interface 193, and an opticaldisk drive interface 194. The drives and their associated computer-readable media provide nonvolatile storage of computer readable instructions, data structures, program modules and other data forpersonal computer 100. It will be appreciated by those skilled in the art that other types of computer readable media which can store data that is accessible by a computer, such as magnetic cassettes, flash memory cards, digital video disks, Bernoulli cartridges, random access memories (RAMs), read only memories (ROMs), and the like, may also be used in the exemplary operating environment. - A number of program modules can be stored on the hard disk,
magnetic disk 190,optical disk 192, ROM 140 orRAM 150, including anoperating system 195, one ormore application programs 196,other program modules 197, andprogram data 198. A user can enter commands and information intocomputer 100 through input devices, such as akeyboard 101 and apointing device 102. Other input devices (not shown) may include a microphone, joystick, game pad, satellite dish, scanner, or the like. These and other input devices are often connected to theprocessing unit 110 through aserial port interface 106 that is coupled to the system bus, but may be connected by other interfaces, such as a parallel port, a game port or a universal serial bus (USB). Amonitor 107 or other type of display device is also connected tosystem bus 130 via an interface, such as avideo adapter 108. In addition to the monitor, personal computers typically include other peripheral output devices (not shown), such as speakers and printers.Computer 100 can operate in a networked environment using logical connections to one or more remote computers, as is well known in the art. - Media objects may be supplied from a storage medium, such as a CD-ROM or a PC card, etc., or supplied directly from a capture device such as a camera or a scanner, or from the Internet or some other supply. Media objects are often stored in a database. Any database suitable for storing media data may be used. Cataloging refers to forming an index of a collection of media objects. Indexing refers to building a table or index that includes pointers to data. These pointers permit quick access to data without having to search and sort through information. Examples of index types that may be useful in the present invention include an index of metadata sorted by date, by photographer or other capturer, by longitude, by latitude, by zip code, by proximity to other objects, by subject matter or any other useful piece of information. Typical indexing methods include those used by databases, for example the known Indexed Sequential Access Method (ISAM). A catalog can be built as part of a file system or a database as appropriate for the desired method of access. As an example of a database, Access by Microsoft Corp. can be used which provides for a data store, indexing on multiple keys in order to provide quick access to sets of data and methods of performing a query based on multiple factors to generate a report. The categories for the database may be standard categories within the cataloging system and/or categories that are set by the user. For example, the database may include categories for cataloging the media objects such as particular holidays, vacations, locations, dates, and the like.
- Categorization according to the present invention contemplates the situation where a user pre-defines a set of labels representing a collection or category that a media object lies in such that a new object is automatically placed into one of those collections or categories. If a user does not pre-define a set of labels representing collections or categories, but all objects are automatically placed, the term for the placement is known as clustering. An index may be built by categorization or by clustering.
- For purposes of describing the present invention, the following terms will be used to represent the cataloging of the database. The media objects in the database may be arranged in groups called “collections” where the media objects in a collection have similar content or relate to similar subject matter. For example, the database may include collections such as a Holiday collection, a Vacation collection, a Birthday collection, a People collection, etc. Each collection may include a number of categories. For example, the Holiday collection may have a Christmas category, a New Year's category, a Fourth of July category, etc. Keywords may be used to indicate what information is found in a media object stored under a particular collection and/or category or to index a media object. Of course, it will be appreciated by one of ordinary skill in the art that data may be arranged differently in databases and other terminology may be used in known systems. However, it will also be appreciated by those of ordinary skill in the art, that the present cataloging method may be applied to any organizing system. The language used herein is only for illustrative purposes and is not intended to limit the invention.
- According to the present invention, media objects may be cataloged (i.e., indexed and put into an organized format for future reference) based upon correlation of the data for the media object and/or data describing the media object (metadata) to be stored with known data and metadata. The data correlation may be used to automatically catalog large amounts of information. For example, a user may have accumulated a large number of images taken during a vacation. When an image is captured by a digital camera, for example, it may include a date and time stamp indicating the date and time that the image was captured. If the date of the image is the same as the date of other images that have been grouped under a particular collection/category, then an inference may be made that this particular image should also be grouped under the same particular collection/category. In other words, if the images were taken on the same date, then it is likely that they represent the same event. Therefore, it may be inferred that the images be cataloged in the same collection/category of the database. The same may be true of any media objects. In another example, if the location data for an image indicates that it is within a few feet of a location of images that were all cataloged under a category for Grandmother's house, an inference may be made that this image should also be cataloged under the category for Grandmother's house. Commonality between media objects may also be used in arriving at an inference as to how a media object should be cataloged. For example, if a number of images were captured on a particular day and a group of them include images of a particular person, then an inference may be made that these images should be grouped together.
- Cataloging according to the present invention involves taking known data that has been associated with attributes and labeled, for example, with “Grandmother's house” and using that known data to infer a label for other data. As demonstrated by the foregoing examples, the invention utilizes image data and metadata for a media object to make inferences based on known data and metadata, and then performs cataloging based upon the inferences. The cataloging may, on occasion, require input from the user to confirm a particular inference. Therefore, according to the present invention, cataloging of media objects in a database may be substantially automatic.
- The cataloging method according to the present invention essentially involves four phases: data collection, data correlation, data refinement and data compaction. Each of these phases may be performed independently or in combination, at the same time, or at different times, and on one or more devices or databases. They may also be performed multiple times and in different orders of performance to achieve the desired results.
- Data collection includes gathering ancillary data before, during or after the capture of a media object. This may be done automatically or manually, and may be related to or result from another media object capture. The data may include information added by the user via an input device such as a keyboard, or it may be supplied from other external devices. For example, information from a GPS device indicating the location where the image was captured may be added, or audio may be added from memory or an external device. The user may add a textual description of the image. The ancillary data may be stored along with the media object. The media object database may be physically the same or different from the database for storing the information concerning the media object. Underlying database technologies can include any general database, for example SQL Server, Access or Exchange provided by Microsoft Corp. and as known in the art.
- A simple example of data collection may be recording the date and time of a photograph being taken. In another example, audio may be sampled before, during, and after the photograph for later analysis. For instance, a photograph taken at a birthday party may have “Happy Birthday” sung before and after a photograph is taken. In addition, the ancillary data may be obtained from another media object based upon an inference, as discussed below.
- Data correlation is the mechanism by which one or more pieces of data from the data collection phase are integrated and synthesized to form new data that may be used later to identify, organize, and search media objects. For example, by correlating GPS and date information, those photographs that were taken at a similar time and place may be detected. This may involve using an inexact search technique. Inexact searches can be performed using a variety of technologies, as is known to one skilled in the art. For example, techniques from information retrieval, inexact logic, or probabilistic searches may be used. In the birthday example noted above, data correlation may assist in determining which photograph would be associated with the “Happy Birthday” recording. In a batch-type process, each media object may be processed to determine how it should be cataloged. On the other hand, the media objects may be processed by determining all media objects relating to a particular event, time, location, etc. The correlation may be performed in a network environment to accommodate access to known information.
- Data refinement relates to the process of managing and extending the data correlation process to find more meaningful data and to reject false correlations. This may involve querying the user to determine the most appropriate correlation. For example, if audio is captured in the time between two meetings, a query to the user may request identification of whether the audio is related to the first or second meeting, or neither. Another example would be to reject the correlation between images taken by two cameras during the same period of time, but at very different locations. Data refinement may also be performed based on the past action of the user on similar data. Another possibility includes enabling the user to manually set the appropriate correlation. According to the present invention, information concerning whether the media object was manually or automatically cataloged may also be stored. When a media object has been cataloged manually, it may be assumed that it has been processed correctly.
- Data compaction relates to the process of removing redundant and/or useless data. Once an image is properly cataloged, indexed, and stored, much of the data used for cataloging may no longer be useful. As there can be a large amount of captured and inferred data, this phase may save a significant amount of storage space. The four most common means of doing this are linking (i.e., replacing duplicate information with a reference to that information in a common location), tokenizing or compressing (i.e., replacing data with a smaller, lossless representation), and hashing (replacing data with a unique identifier, from which the original data cannot be reconstructed). Each of these techniques is well known in the art.
-
FIG. 2 is a flowchart that illustrates an example of the steps for performing a method according to an aspect of the present invention. Collections and categories of the database may be standard as provided with the system. In addition, the user may add collections and categories to personalize the database. In step S200, a media object is captured, including any metadata and/or ancillary data that may be associated with the media object. The cataloging process is initiated in step S201. The cataloging method may be any one of a number of known methods that may be enhanced by the method according to the present invention. The processing may be performed on a server, on the capture device, personal computer and the like. The cataloging process according to the present invention may include processing the media object with respect to time, date, and/or location, for example. The processing may include comparing the relevant data with threshold data or performing an inexact search using known techniques, as shown in step S202. The data correlation process performed in step S202 results in a pruned list of media objects in step S203. The pruning may require querying the user to determine the most appropriate correlation of the data. In step S204, an inference may be made with respect to cataloging the media object based upon the data for the media objects in the pruned list. The media object may be cataloged in step S205 based upon the inference made in step S204. The filename may be used to index the media object in the database. For example, the terms of the filename may be used as keywords to search the database. Alternatively, any appropriate known indexing may be used to search the database. Once the media object is properly stored, cataloged and indexed, step S206 may be performed to remove redundant or useless data in order to reduce the amount of required storage space for the media object. As noted above, any one of a number of known techniques may be used to perform step S206, such as linking, tokenizing or compressing and hashing. Finally, in step S207, the information for the image may be updated to indicate that the image was cataloged based upon an inference. -
FIG. 3 shows an additional level of processing according to another aspect of the present invention. More particularly, steps S300-S303 inFIG. 3 correspond to steps S200-S203 inFIG. 2 . However, in step S304 ofFIG. 3 , information in a global calendar may be used to catalog the media object. For example, the processing in steps S300-S303 may generate a pruned list of images at a particular location. Therefore, it may be inferred that the media object to be cataloged is also an image taken at the particular location. In addition, date information for the image may be compared with dates in the global calendar lookup in step S304. If the date of the image corresponds to January 1st, it may be further inferred in step S305 that the image at the particular location was taken on New Year's Day. The image may then be properly cataloged in step S306 based upon this additional inference. Once the media object is cataloged in step S306, redundant or useless information may be removed in step S307. - In
FIG. 2 , the cataloging may be performed with respect to media objects that are stored in the user's database of media objects. However, it may be desirable to utilize information other than that found in the user's database to automatically catalog a media object. InFIG. 3 , the use of the global calendar lookup enables the cataloging process to utilize information other than that found in the user's database to catalog the image. Similarly, information in a global map lookup, global address book, phonebook etc., may be useful in cataloging and indexing a media object. Of course, local lookups may also be possible. It will be appreciated by those of ordinary skill in the art that many other types of “lookups” may be possible for performing the method according to the present invention. -
FIG. 4 shows another aspect of the present invention. Steps S400-S403 inFIG. 4 correspond to steps S200-S203 inFIG. 2 . In step S404, the media object may be processed to determine common features between the media object and media objects obtained in step S403. For example, the pruned list in step S403 may represent the images obtained in the last day. In step S404, the image to be cataloged may be processed with respect to the images in the pruned list to find common features. In this example, it may be determined that the images all contain a particular person. An inference may then be made in step S405 that the image should be cataloged under the people collection in the category corresponding to the particular person. In step S406, the image may be cataloged according to the inference in step S405. Finally, in step S407, redundant or useless information may be removed. - A particular example will be described with reference to
FIG. 5 . Once again, the example is directed to cataloging images through the use of categorization algorithms. However, as stated above, the examples set forth herein are for illustrative purposes only and those of ordinary skill in the art will appreciate that any media object(s) may be cataloged according to the present invention. Referring to the example inFIG. 5 , a user may have accumulated a large number of images taken during the Christmas holidays. Information for these images may be used to distinguish them from other images so that they can be organized together in a database. A holiday collection may be arranged in the database having categories corresponding to particular holidays, respectively. One holiday category may correspond to “Christmas.” Images that were captured on December 24th or December 25th may be organized under the “Christmas” category. - In step S500, an image is captured along with any data that may be associated with the image, including data added by the user or data supplied from some other external source or device. In the current example, the date will be used for organizing the images. In step S501, the cataloging process is initiated. More particularly, in step S502, the date on which the image was captured may be detected. In step S503, the date is compared with the date in a global calendar lookup. In step S504, it is determined whether the date on which the image was captured is December 24th. If the image was captured on December 24th, in step S505 an inference may be made that the image should be cataloged in the Christmas category of the Holiday collection. In step S506, the image may be cataloged based upon the inference. In step S507, redundant or useless information may be removed. If the date of the image does not correspond to December 24th, it is determined in
step 508 whether the date corresponds to December 25th. If the answer is Yes in step S508, then steps S505-S507 are performed. If the answer in step S508 is No, then other processing is performed. In this example, in step S509, other media objects captured on the same date on which the image was captured are detected to arrive at a pruned list in step S510. An inference may be made in step S511 as to how to catalog the image based upon the information for the media objects in the pruned list. The image may then be cataloged in step S512 based upon the inference. Finally, step S507 may be performed to remove redundant or useless data. Once the image is stored in a particular category, the filename may be used as keyword(s) for indexing the image in the database. Of course, one of ordinary skill in the art will appreciate that any appropriate indexing method may be used. - Referring to
FIG. 6 , once the inference is made in step S605 that the image should be organized with Christmas images, the image may be properly cataloged in step S606. The information for the image may be updated to indicate that the image was cataloged based upon an inference in step S607. The identification of inferred data may indicate that there is a certain level of unreliability of the information as compared to manually indexing the media object. Steps S600-S604 and steps S609-S613 are the same as steps S500-S504 and S508-S512 inFIG. 5 . - In another example, the date and location may be correlated to organize images, as shown in
FIG. 7 . In step S700, an image is captured along with any data that may be associated with the image, including data added by the user or data supplied from some other external source or device. The cataloging process is initiated in step S701. In step S702, it is determined whether date information is available. If the answer in step S702 is Yes, then the date is detected in step S703. The date is then compared with a date range corresponding to the user's vacation, for example, in step S704. If the date is within the vacation range, then processing continues to step S705. In step S705, all of the images having the same date as the detected date are obtained from the database and added to the pruned list in step S706. If the answer in step S704 is No, then processing ends. - If the answer in step S702 is No, then in step S713, it is determined whether location information is available. If the answer in step S713 is Yes, then the location information for the image is detected in step S714. In step S715, it is determined whether the location information is within fifty miles of the vacation location. If the answer in step S715 is Yes, then processing continues to step SS716. If the answer in step S715 is No, the process ends. In step S716, all of the images at the detected location are detected. In step S706, all of the detected images are added to the pruned list. If the answer in step S713 is No, then other cataloging processes are performed. In this example, the image may be compared with other media objects to detect common features between the image and any of the stored images in step S717. The images that include common features may be added to a pruned list in step S718. An inference may be made in step S719 as to how to catalog the image based upon the information for the media objects in the pruned list. The image may then cataloged in step S720 based upon the inference, and processing continues at step S711.
- In step S707, the images in the pruned list are processed to determine those images having common features with the image to be cataloged. In step S708, an inference is made on how the image should be cataloged based upon the information obtained in step S707. In step S709, date or location information may be added to information for the media object if either of this data is missing. In step S710, the image may be cataloged. In step S711, redundant or useless data may be removed. Finally, in step S712, the media object may be flagged as having been cataloged based upon inference(s).
- At present, files in an operating system may be retrieved by searching by date, by file size, last modified, etc. It would be helpful to enable a user to search a database by location or category, for example. For example, a user may want to search a database of images by searching all of the images taken at Grandparents' house. According to an aspect of the present invention, even if there is no index for images taken at Grandparents' house, the search may be performed by going through all of the images and determining which images have a GPS location corresponding to the location of Grandparents' house. Alternatively, the search may be performed by determining which photos were taken during a trip to the Grandparents' house as listed in a local calendar. Therefore, according to an aspect of the present invention, searches may be done on the fly just as an indexing system or done as a way of setting up a search.
- Referring to
FIG. 8 , an example of searching and retrieving data from a database according to an aspect of the present invention will be described. The search may be performed using a general search engine, or it may be limited to a local database. In step S800, a search term is identified, e.g., St. Patrick's Day, March 17th, etc. In step S801, a reference lookup is performed via a table of references. In the present example, the reference lookup may include looking up March 17, Leprechaun. In step S802, a full search may be performed based on references obtained from reference lookup on step S801. The list of hits is displayed in step S803. The list may then be pruned in step S804 according to known pruning techniques. For example, the list may be pruned to remove duplicates. The resulting search list is displayed in step S805. - Another example of automatically organizing media objects will now be described. Once again, the embodiment will be described with respect to the example of still images, and organization is based upon creation date or color. However, the embodiment may be implemented with any media object and based upon some other criteria other than time or color.
- According to this aspect of the invention, media objects, such as still images, are automatically organized into various collections by clustering images that are taken near each other in time. The images in each collection often tend to be thematically related, in addition to temporally related. A user interface in this embodiment may include one image per collection, where the image is shown to the user. If the user is searching for an image, the user views the images respectively representing collections of images, and selects a collection that appears to relate to the desired image. Once a collection is selected, the images corresponding to the collection are shown to the user. In this way, the user can quickly find a desired image.
- The process for automatically organizing media objects, such as still images, into various collections according their creation times will be described with reference to
FIG. 9 . Instep 900, the images in some predetermined directory structure, such as the “My Pictures” directory under the Windows® operating system (OS), for example, are sorted in ascending order by creation date. In step S901, a variable denoted “lasttime” is set to effective negative infinity (e.g., a quantity less than the earliest date represented under the OS). An empty current collection is created in step S902. The first (earliest created) image is selected for processing in step S903. Of course, processing could be performed beginning with the latest (last created) image. The creation time of the current image is compared to the lasttime variable in step S904. If the difference between these two times is less than X hours, for some value of X, the two images are placed in the same collection. The value of X may be set by the user to any suitable time such as 1 hour, 24 hours, 15 minutes, etc. The shorter the time set for X, the more collections are created, which will tend to be more thematically related, but will require the user to scan through more collections. A default value for X may be set and the user may adjust the default value as desired. If the answer in step S904 is Yes, then the image is stored in the current collection in step S905. If the answer in step S904 is No, then the current image is not stored in the current collection. Rather, a new empty current collection is created in step S910, and processing continues to step S905. - In step S906, the lasttime variable is updated to reflect the creation date of the current image stored in the current collection. In step S907, it is determined whether the current image is the last image in the sorted list of images. When the answer in step S907 is No, then the next image in the sorted list is selected in step S911 and processing returns to step S904. If the answer in step S907 is Yes, then a representative image for the current collection is selected in step S908. The images in each collection may be sorted in order of creation time. The representative image for each album may be the image with the median creation time, i.e., the picture in the middle of the list of images in the collection sorted based upon creation time. Finally, in step S909, the view of the collection is generated.
- More specifically, referring again to S903, it is desirable that the threshold X not be a constant, but rather be computed adaptively based on the differences in creation time of temporally nearby images. Allowing the threshold X to be computed adaptively provides an advantage that the threshold will adapt to the typical behavior of the user and will identify the unusual temporal gaps between photographs.
- The adaptive threshold is determined by the following formula:
- In Formula 1, i indexes the media objects, (for example, photographs) in order of creation time, ti is the creation time of photograph i, and Xn is the threshold that is compared to the difference between tn and tn+1. Formula 1 thus computes a running average of the log time gaps between photographs, in the example. If the indices i in the sum access creation times that do not exist for a media object, then the sum only runs over indices i that do, in fact exist. As a result, the value in the denominator, 20, is reduced to the number of terms in the sum.
- Referring to
FIG. 10 , a flowchart is provided which describes a color clustering algorithm for media objects (photographs in the present example). Color clustering is useful when the creation time of the media objects are either unreliable or unavailable. In S1001, a color histogram for all images is created. The process of extracting color histograms is well known in the art and therefore will not be described in detail here. Preferably, the color histograms are taken over all pixels of images that are downsampled by a factor of 4 or 8, thus increasing the speed of the histogram computation. Optimally, the color histogram is taken in the CIE u′ v′ color space. By taking the color histogram in the CIE u′ v′ color space, pixel luminance is ignored. This is desirable because the pixel luminance does not predict the topic of the media object, here a photograph. Ideally, the histogram has 16×16 equally spaced bins in color space, spanning from 0.1612 to 0.2883 in u′ and 0.4361 to 0.5361 in v′. Pixels falling outside this area in color spaced are mapped to the nearest bin. - The histogram may be estimated in a Bayesian manner, using maximum a posteriori methods and a uniform prior, as is well known in the art. The histogram is initialized with so-called “phantom data.” Before all pixels of an image are scanned, every bin is initialized to 1/256. As is also known in the art, a “number of pixel” counter is required to compute a histogram; the number of pixel counter is initialized the 1. By initializing the number of pixel counter to 1, no bin is allowed to attain a value of 0 which is desirable in the computation of the Jensen-Shannon distance in equation 4 of S1004, described below. The histogram is best estimated in a smoothed manner as is further known in the art. The histogram may be smoothed via so-called bilinear interpolation wherein every scanned pixel increases up to four histogram bins, according to standard bi-linear interpolation rules.
- In S1002, the creation time for all images is extracted. As is known in the art, JPEG images produced by digital cameras have image metadata stored in a format known as the EXIF format. The creation time for the photograph in the present example is stored as a filed in the EXIF format. Techniques for extracting the creation time are well known in the art. Occasionally, the creation time in the EXIF format is missing or incorrect. This can occur, for example, when the internal clock of a digital camera gets reset to an initial value when the internal battery of the camera runs down. By way of example, this situation might be detected by an EXIF creation time that occurred before a certain date, such as Jan. 1, 1999. In addition, it is possible that the EXIF metadata might be entirely deleted after image capture by way of image manipulation tools. Where the EXIF creation time is incorrect or missing, the file creation of modification date can be used as a substitute when that information is available, such as when a personal computer is being used.
- In S1003, all of the photographs in the example are ordered by imputed creation time. In the alternative, the ordering step S1003 can be derived form the innate ordering of JPEG filenames produced by digital cameras, for example, P000006.JPG would be known to have been taken after P000005.JPG.
- Turning to S1004, the pairwise distance between color histograms of adjacent pairs of ordered images is computed. By way of example, assume pi and qi are the ith color histogram bin values of two adjacent images (or collections). These probabilities are computed by taking the ratio between the number of pixels that fall into the ith bin (as modified by the known techniques of Bayesian inference and smoothing) to the total number of pixels that fall into all bins. Thus, pi and qi are computed as follows:
for bin values ni and mi and total number of pixels N and M for the first and second images, respectively. If ri if the ith color histogram bin of a tentative cluster of images consisting of two adjacent clusters or images, then the value ri can be computed as follows: - The distance between the two images or clusters can thus be determined as follows:
where a term that evaluates to 0 log 0 is treated as 0. As is known in the art, other distance metrics can be used, including known techniques of chi-squared or L1 norm. In step S1004, these distance metrics are computed between two images, but in step S1012, the same distance metric used in step S1004 is computed between two clusters. - In S1005, S1006, and S1010 through S1013 in
FIG. 10 , Omohundro's best-first model merging technique is applied to clustering photographs. Best-first model merging is a clustering technique that starts with a set of items that are sorted into a sequence. The technique starts with each item in its own cluster. Then, at every step, the distance between every pair of adjacent clusters is computed. The pair with the smallest distance is merged together to yield a new cluster. This new cluster replaces the two old clusters in the sequence, and the clustering continues until the desired number of clusters is reached. In S1005, all of the distances are placed into a heap data structure which permits finding the least element in O(logN) time, where N are the number of elements stored in the heap, as is well known in the art. In S1006, the algorithm checks for termination by comparing the size of the heap (i.e., the number of elements in the heap) to the number of desired clusters. The number of desired clusters can be, for example, 1/12 the total number of photographs. The ratio between the number of clusters and the number of photographs is the compression ratio of representative images to all photographs in the user interface. If S1006 indicates that fewer clusters must be found, then S1010 finds the smallest Jensen-Shannon distance (equation 4 in S1004) in the heap and removes it from the heap. The two clusters (or images) which result in this smallest distance are also found. Step S1011 combines these two adjacent clusters (or images) to yield a new cluster. The new cluster contains pointers to all images contained in the previous two clusters. In addition, the histogram for the combined cluster is computed, as previously shown. Step S1012 computes the distance from the new combined cluster to its two new neighboring clusters (or neighboring images). Again, the distance is computed between two adjacent clusters and the speculative cluster that results by combining the two adjacent clusters. In step S1013, the heap structure is rearranged in order to maintain the heap property, as is known in the art. Control then flows back to step S1006. - Referring to step S1006, if the correct number of clusters has been found, then control flows to step S1007 where representative images are chosen from each cluster. For every cluster, the overall histogram that describes all images have been formed at either step S1004 or S1012. The representative image is chosen from the images that belong to the cluster to minimize the Kulback-Leibler (KL) distance between the overall histogram ri and the histogram for the representative image si:
- The representative image is the image whose color best mimics the overall color of the cluster. In step S1008, all of the representative images are displayed in a user interface.
- Referring to
FIG. 11 , a flowchart is shown depicting a method which combines temporal and color clustering of photographs. This combination uses the better clustering of the temporal algorithm but falls back to color clustering when the temporal data is incorrect. In step S1101, the pictures are sorted by creation date, similar to steps S1101 and S1102 inFIG. 11 . In step S1102, the temporal clustering algorithm is applied, as shown inFIG. 9 (except for S908 and S909) resulting in a number of clusters. These clusters are then scanned in steps S1103-S1107 to check for failure of the temporal clustering. The existence of overly large clusters is evidence of temporal clustering failure. Thus, step S1103 first sets the current cluster to the first cluster and step S1104 counts the number of images in the current cluster. If the number is larger than a threshold X, step S1105 will split the current cluster into sub-clusters according to the color algorithm described inFIG. 10 , except for steps S1007 and S1008. Preferably, the threshold X is 23. If the number is larger than the threshold X, the desired number of sub-clusters is preferably 1/12 of the number of photographs in the current cluster, rounded to the nearest integer. Upon successful splitting of the current cluster into sub-clusters, or if the current cluster has less than or equal to X images, control flows to step S1106 which can terminate the loop if the current cluster is the last cluster. If it is not, step S1107 makes the current cluster the next cluster and control returns to step S1104. Otherwise, control flows to step S1108 which computes representative images for all of the clusters (including both temporal and color clusters). This step is analogous to step S1007 inFIG. 10 . In step S1108, the representative images are shown in the user interface. It is desirable to sort the representative images by increasing creation time prior to display. -
FIG. 12 shows a possible user interface. The user is presented with a user interface containing two panes. The left pane consists of thumbnails of representative images, represented collectively by 1201 inFIG. 12 . These thumbnails of representative images are of all of the clusters of the user's photographs, in the present example. These thumbnails are sorted by increasing creation time. The user can scroll through the representative images by usingscroll bar 1202 as is known in the art. In addition, dates can be shown intermixed with or below each of the thumbnails. When a user clicks on a representative image in 1201, the contents of that cluster are shown in the right pane as thumbnails in 1203. If there are too many photographs in the selected cluster to show in the right pane, an optional scroll bar 1204 (not shown) can be used to scroll through the contents of the cluster. When a user clicks on a thumbnail in 1203, the user is shown a full-sized version of the image. Alternatively, if the user drags or copies a thumbnail from 1203, the full-sized image can be a drop candidate or can be put on a clipboard, respectively. - A preferred user interface is shown in
FIG. 13 . Again, two panes are presented in the user interface. In the right pane,thumbnails 1303 of all of the user's photographs are shown. The thumbnails in 1303 are sorted by increasing creation time. There is ascrollbar 1304 to allow the user to scroll through his or her photographs. Representative images are shown as thumbnails in 1301 in the left pane. The thumbnails in 1301 are sorted by increasing creation time also. These representative images form a table of contents for the entire collection on the right. When a user clicks on (or otherwise selects by known techniques) a representative thumbnail in 1301 in the left pane, thescrollbar 1304 is scrolled so that the same thumbnail is vertically centered in the right pane and highlighted with a colored boarder, so as to be noticeable. Whenscroll bar 1302 is activated, the left pane containingrepresentative thumbnails 1301 are scrolled thus allowing the user to quickly scan the collection. The user interface ofFIG. 13 is desirable because the user may not always realize which cluster a photograph belongs to. Thus, thescrollbar 1304 provides the user with a mechanism to easily correct for incorrect cluster guesses. - while particular embodiments of the invention have been shown and described, it is recognized that various modifications thereof will occur to those skilled in the art without departing from the apirit and scope of the invention. More particularly, it will be appreciated by those of ordinary skill in the art that the present application may be useful in many applications. The described embodiments are to be considered in all respects only as illustrative and not restrictive. Therefore, the scope of the herein-described invention shall be limited solely by the claims appended hereto.
Claims (15)
1. A method of organizing media objects in a database, comprising:
capturing a media object;
detecting at least one of a latitude or longitude of the captured media object indicated in metadata for the captured media object;
identifying media objects stored in the database that are related to the captured media object based on at least one of a latitude or longitude of the captured media object;
inferring organization information for the captured media object based upon information obtained from each of the stored media objects that are related to the captured media object; and
organizing the captured media object in the database based upon the inferred organization information.
2. The method according to claim 1 , wherein the organizing step includes storing the captured media object in the database, and wherein the method further comprises assigning at least one attribute to the metadata for the captured media object prior to storing the captured media object.
3. The method according to claim 1 , further comprising:
assigning at least one attribute to the metadata for the captured media object based upon the inferred organization information.
4. The method according to claim 1 , further comprising:
adding information to the metadata of the captured media object indicating that the inferred organization information for the captured media object was determined based upon an inference.
5. A method of organizing media objects in a database, comprising:
detecting a capture time for each of the media objects to be organized;
sorting the media objects based upon the capture time to generate a sorted list;
comparing the capture time of each of the media objects with a reference value;
updating the reference value according to the media objects in the sorted list; and
storing the media objects in the database based upon the comparison.
6. The method according to claim 5 , wherein the comparing step comprises:
determining whether the capture time of the each of the media objects is within a predetermined time period from the reference value; and
wherein the storing step comprises grouping at least one of the media objects into a collection when the capture time of the at least one of the media objects is within the predetermined time period from the reference value.
7. The method according to claim 6 , further comprising:
selecting a representative media object from the at least one of the media objects grouped in the collection for use as a user interface.
8. The method according to claim 7 , further comprising:
repeating the comparing step, the storing step and the selecting step for each of the media objects in the sorted list.
9. The method according to claim 5 , wherein the step of updating the reference value according to the media objects in the sorted list further comprises:
setting the reference value to a predetermined value;
determining whether the capture time of a first one of the media objects in the sorted list is within a predetermined time period from the reference value;
grouping the first one of the media objects into a collection when the capture time of the first one of the media objects is within the predetermined time period from the reference value;
updating the reference value to the capture time of the first one of the media objects in the sorted list to generate an updated reference value; and
repeating the determining step, the grouping step and the updating step for each of the media objects in the sorted list.
10. The method according to claim 9 , further comprising:
creating a new collection when the capture time of any one of the media objects from the sorted list is not within the predetermined time period from the updated reference value.
11. The method according to claim 10 , further comprising:
selecting a representative media object from the collection and from each new collection for use as a user interface.
12. A computer-readable medium having computer-executable instructions for performing the steps of:
capturing a media object;
detecting at least one of a latitude or longitude of the captured media object indicated in metadata for the captured media object;
identifying media objects stored in the database that are related to the captured media object based on at least one of a latitude or longitude of the captured media object;
inferring organization information for the captured media object based upon information obtained from each of the stored media objects that are related to the captured media object; and
organizing the captured media object in the database based upon the inferred organization information.
13. The computer-readable medium according to claim 12 , wherein the organizing step includes storing the captured media object in the database, and wherein the method further comprises assigning at least one attribute to the metadata for the captured media object prior to storing the captured media object.
14. The computer-readable medium according to claim 12 , having further computer-executable instructions for performing the step of:
assigning at least one attribute to the metadata for the captured media object based upon the inferred organization information.
15. The computer-readable medium according to claim 12 , having further computer-executable instructions for performing the step of:
adding information to the metadata of the captured media object indicating that the inferred organization information for the captured media object was determined based upon an inference.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US11/282,610 US20060074973A1 (en) | 2001-03-09 | 2005-11-21 | Managing media objects in a database |
Applications Claiming Priority (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US27423401P | 2001-03-09 | 2001-03-09 | |
US10/021,255 US7076503B2 (en) | 2001-03-09 | 2001-12-19 | Managing media objects in a database |
US11/282,610 US20060074973A1 (en) | 2001-03-09 | 2005-11-21 | Managing media objects in a database |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US10/021,255 Continuation US7076503B2 (en) | 2001-03-09 | 2001-12-19 | Managing media objects in a database |
Publications (1)
Publication Number | Publication Date |
---|---|
US20060074973A1 true US20060074973A1 (en) | 2006-04-06 |
Family
ID=26694479
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US10/021,255 Expired - Fee Related US7076503B2 (en) | 2001-03-09 | 2001-12-19 | Managing media objects in a database |
US11/282,610 Abandoned US20060074973A1 (en) | 2001-03-09 | 2005-11-21 | Managing media objects in a database |
Family Applications Before (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US10/021,255 Expired - Fee Related US7076503B2 (en) | 2001-03-09 | 2001-12-19 | Managing media objects in a database |
Country Status (1)
Country | Link |
---|---|
US (2) | US7076503B2 (en) |
Cited By (17)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20050278311A1 (en) * | 2004-06-07 | 2005-12-15 | Trent Moore | System and method for generating advertisements utilizing a database of stock imagery |
US20060200475A1 (en) * | 2005-03-04 | 2006-09-07 | Eastman Kodak Company | Additive clustering of images lacking individualized date-time information |
US20070143300A1 (en) * | 2005-12-20 | 2007-06-21 | Ask Jeeves, Inc. | System and method for monitoring evolution over time of temporal content |
US20070260586A1 (en) * | 2006-05-03 | 2007-11-08 | Antonio Savona | Systems and methods for selecting and organizing information using temporal clustering |
US20070270985A1 (en) * | 2006-05-16 | 2007-11-22 | Canon Kabushiki Kaisha | Method for navigating large image sets using sort orders |
US20080052321A1 (en) * | 2006-08-23 | 2008-02-28 | Canon Kabushiki Kaisha | Information display apparatus and method |
US20080313119A1 (en) * | 2007-06-15 | 2008-12-18 | Microsoft Corporation | Learning and reasoning from web projections |
US20090070346A1 (en) * | 2007-09-06 | 2009-03-12 | Antonio Savona | Systems and methods for clustering information |
US20100257252A1 (en) * | 2009-04-01 | 2010-10-07 | Microsoft Corporation | Augmented Reality Cloud Computing |
US20120106859A1 (en) * | 2009-06-24 | 2012-05-03 | Philip Cheatle | Image Album Creation |
US20120201465A1 (en) * | 2011-02-04 | 2012-08-09 | Olympus Corporation | Image processing apparatus |
US20130222696A1 (en) * | 2012-02-28 | 2013-08-29 | Sony Corporation | Selecting between clustering techniques for displaying images |
US8923629B2 (en) | 2011-04-27 | 2014-12-30 | Hewlett-Packard Development Company, L.P. | System and method for determining co-occurrence groups of images |
US8972490B1 (en) * | 2012-06-29 | 2015-03-03 | Emc Corporation | User discovery in socially-aware data storage systems |
US20150088890A1 (en) * | 2013-09-23 | 2015-03-26 | Spotify Ab | System and method for efficiently providing media and associated metadata |
US20160065443A1 (en) * | 2014-08-26 | 2016-03-03 | Sugarcrm Inc. | Retroreflective object tagging |
US10482109B2 (en) * | 2015-09-29 | 2019-11-19 | Canon Kabushiki Kaisha | Object indexing method, object searching method, and object indexing system |
Families Citing this family (126)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7366562B2 (en) * | 2003-10-17 | 2008-04-29 | Medtronic Navigation, Inc. | Method and apparatus for surgical navigation |
US20020087546A1 (en) * | 2000-01-31 | 2002-07-04 | Michael Slater | Apparatus, methods, and systems for digital photo management |
US7593035B2 (en) * | 2000-04-14 | 2009-09-22 | Fujifilm Corporation | Image data transmitting device and method |
US8924383B2 (en) * | 2001-04-06 | 2014-12-30 | At&T Intellectual Property Ii, L.P. | Broadcast video monitoring and alerting system |
US7296032B1 (en) | 2001-05-17 | 2007-11-13 | Fotiva, Inc. | Digital media organization and access |
JP4749628B2 (en) * | 2001-09-07 | 2011-08-17 | パナソニック株式会社 | Album creating apparatus, album creating method, and album creating program |
US7240333B2 (en) * | 2001-09-21 | 2007-07-03 | Intel Corporation | Display of a digital information content and method of selection |
US20030061490A1 (en) * | 2001-09-26 | 2003-03-27 | Abajian Aram Christian | Method for identifying copyright infringement violations by fingerprint detection |
US20030105743A1 (en) * | 2001-11-16 | 2003-06-05 | Sonicblue, Inc. | Use of database queries for manipulation of media content |
US20030128389A1 (en) | 2001-12-26 | 2003-07-10 | Eastman Kodak Company | Method for creating and using affective information in a digital imaging system cross reference to related applications |
JP2003216621A (en) * | 2002-01-23 | 2003-07-31 | Fuji Photo Film Co Ltd | Program and image control device and method |
US7142225B1 (en) * | 2002-01-31 | 2006-11-28 | Microsoft Corporation | Lossless manipulation of media objects |
US7009643B2 (en) * | 2002-03-15 | 2006-03-07 | Canon Kabushiki Kaisha | Automatic determination of image storage location |
JP2003298991A (en) * | 2002-03-29 | 2003-10-17 | Fuji Photo Film Co Ltd | Image arranging method and apparatus, and program |
US20030191776A1 (en) * | 2002-04-05 | 2003-10-09 | Pere Obrador | Media object management |
KR100502710B1 (en) * | 2002-05-24 | 2005-07-20 | 주식회사 아이큐브 | Optical disk regenerative apparatus |
US7949689B2 (en) * | 2002-07-18 | 2011-05-24 | Accenture Global Services Limited | Media indexing beacon and capture device |
US7149755B2 (en) * | 2002-07-29 | 2006-12-12 | Hewlett-Packard Development Company, Lp. | Presenting a collection of media objects |
US8417678B2 (en) * | 2002-07-30 | 2013-04-09 | Storediq, Inc. | System, method and apparatus for enterprise policy management |
US7801894B1 (en) | 2004-10-28 | 2010-09-21 | Stored IQ | Method and apparatus for harvesting file system metadata |
US8612404B2 (en) * | 2002-07-30 | 2013-12-17 | Stored Iq, Inc. | Harvesting file system metsdata |
WO2004012379A2 (en) * | 2002-07-30 | 2004-02-05 | Deepfile Corporation | Method and apparatus for managing file systems and file-based data storage |
DE60231844D1 (en) * | 2002-12-20 | 2009-05-14 | Nokia Corp | NEW RELEASE INFORMATION WITH META INFORMATION |
JP2004213129A (en) * | 2002-12-27 | 2004-07-29 | Fuji Photo Film Co Ltd | Method, device and program for classifying picture |
JP2004214759A (en) * | 2002-12-27 | 2004-07-29 | Fuji Photo Film Co Ltd | Method and apparatus of classifying image, and program |
US7131059B2 (en) | 2002-12-31 | 2006-10-31 | Hewlett-Packard Development Company, L.P. | Scalably presenting a collection of media objects |
US7593915B2 (en) * | 2003-01-07 | 2009-09-22 | Accenture Global Services Gmbh | Customized multi-media services |
US7383497B2 (en) * | 2003-01-21 | 2008-06-03 | Microsoft Corporation | Random access editing of media |
US7509321B2 (en) * | 2003-01-21 | 2009-03-24 | Microsoft Corporation | Selection bins for browsing, annotating, sorting, clustering, and filtering media objects |
US7117453B2 (en) * | 2003-01-21 | 2006-10-03 | Microsoft Corporation | Media frame object visualization system |
US7904797B2 (en) * | 2003-01-21 | 2011-03-08 | Microsoft Corporation | Rapid media group annotation |
JP3984175B2 (en) * | 2003-01-31 | 2007-10-03 | 富士フイルム株式会社 | Photo image sorting apparatus and program |
US8392834B2 (en) * | 2003-04-09 | 2013-03-05 | Hewlett-Packard Development Company, L.P. | Systems and methods of authoring a multimedia file |
US7434170B2 (en) * | 2003-07-09 | 2008-10-07 | Microsoft Corporation | Drag and drop metadata editing |
US20050015389A1 (en) * | 2003-07-18 | 2005-01-20 | Microsoft Corporation | Intelligent metadata attribute resolution |
US20050015405A1 (en) * | 2003-07-18 | 2005-01-20 | Microsoft Corporation | Multi-valued properties |
US7747625B2 (en) * | 2003-07-31 | 2010-06-29 | Hewlett-Packard Development Company, L.P. | Organizing a collection of objects |
US7313574B2 (en) * | 2003-10-02 | 2007-12-25 | Nokia Corporation | Method for clustering and querying media items |
US7636733B1 (en) * | 2003-10-03 | 2009-12-22 | Adobe Systems Incorporated | Time-based image management |
US7159188B2 (en) * | 2003-10-23 | 2007-01-02 | Microsoft Corporation | System and method for navigating content in an item |
US20050091254A1 (en) * | 2003-10-23 | 2005-04-28 | Stabb Charles W. | Shell browser preview display |
US7555165B2 (en) * | 2003-11-13 | 2009-06-30 | Eastman Kodak Company | Method for semantic scene classification using camera metadata and content-based cues |
US8010579B2 (en) * | 2003-11-17 | 2011-08-30 | Nokia Corporation | Bookmarking and annotating in a media diary application |
US7109848B2 (en) * | 2003-11-17 | 2006-09-19 | Nokia Corporation | Applications and methods for providing a reminder or an alert to a digital media capture device |
US8990255B2 (en) * | 2003-11-17 | 2015-03-24 | Nokia Corporation | Time bar navigation in a media diary application |
US20050105374A1 (en) * | 2003-11-17 | 2005-05-19 | Nokia Corporation | Media diary application for use with digital device |
US20050108643A1 (en) * | 2003-11-17 | 2005-05-19 | Nokia Corporation | Topographic presentation of media files in a media diary application |
US20050108234A1 (en) * | 2003-11-17 | 2005-05-19 | Nokia Corporation | Speed browsing of media items in a media diary application |
US7774718B2 (en) * | 2003-12-17 | 2010-08-10 | Nokia Corporation | Time handle in a media diary application for accessing media files |
US7441186B2 (en) * | 2004-01-23 | 2008-10-21 | Microsoft Corporation | System and method for automatically grouping items |
US20050166149A1 (en) * | 2004-01-23 | 2005-07-28 | Microsoft Corporation | Table of contents display |
US20050187943A1 (en) * | 2004-02-09 | 2005-08-25 | Nokia Corporation | Representation of media items in a media file management application for use with a digital device |
EP1589444A3 (en) * | 2004-04-21 | 2008-03-12 | Samsung Electronics Co., Ltd. | Method, medium, and apparatus for detecting situation change of digital photos and method, medium, and apparatus for situation-based photo clustering in digital photo album |
JP2005309995A (en) * | 2004-04-23 | 2005-11-04 | Olympus Corp | Device and method for managing information, and program |
US20050246324A1 (en) * | 2004-04-30 | 2005-11-03 | Nokia Inc. | System and associated device, method, and computer program product for performing metadata-based searches |
US20050259945A1 (en) * | 2004-05-20 | 2005-11-24 | Anthony Splaver | Method and system for automatic management of digital photography processing |
US20050267750A1 (en) * | 2004-05-27 | 2005-12-01 | Anonymous Media, Llc | Media usage monitoring and measurement system and method |
US20150051967A1 (en) | 2004-05-27 | 2015-02-19 | Anonymous Media Research, Llc | Media usage monitoring and measurment system and method |
US20050289163A1 (en) * | 2004-06-03 | 2005-12-29 | Eric Gordon | Occasion for media objects |
US20050286428A1 (en) * | 2004-06-28 | 2005-12-29 | Nokia Corporation | Timeline management of network communicated information |
US20060004699A1 (en) * | 2004-06-30 | 2006-01-05 | Nokia Corporation | Method and system for managing metadata |
US7827498B2 (en) | 2004-08-03 | 2010-11-02 | Visan Industries | Method and system for dynamic interactive display of digital images |
JP2006065368A (en) * | 2004-08-24 | 2006-03-09 | Sony Corp | Image display device and method, and computer program |
US20060080316A1 (en) * | 2004-10-08 | 2006-04-13 | Meridio Ltd | Multiple indexing of an electronic document to selectively permit access to the content and metadata thereof |
US8510331B1 (en) | 2004-10-28 | 2013-08-13 | Storediq, Inc. | System and method for a desktop agent for use in managing file systems |
US7844582B1 (en) | 2004-10-28 | 2010-11-30 | Stored IQ | System and method for involving users in object management |
US7580952B2 (en) * | 2005-02-28 | 2009-08-25 | Microsoft Corporation | Automatic digital image grouping using criteria based on image metadata and spatial information |
US7849090B2 (en) | 2005-03-30 | 2010-12-07 | Primal Fusion Inc. | System, method and computer program for faceted classification synthesis |
US7533091B2 (en) * | 2005-04-06 | 2009-05-12 | Microsoft Corporation | Methods, systems, and computer-readable media for generating a suggested list of media items based upon a seed |
US20080097972A1 (en) * | 2005-04-18 | 2008-04-24 | Collage Analytics Llc, | System and method for efficiently tracking and dating content in very large dynamic document spaces |
US7313755B2 (en) * | 2005-04-20 | 2007-12-25 | Microsoft Corporation | Media timeline sorting |
US20060259468A1 (en) * | 2005-05-10 | 2006-11-16 | Michael Brooks | Methods for electronic records management |
US20060259511A1 (en) * | 2005-05-13 | 2006-11-16 | Yahoo! Inc. | Media object organization across information management services |
US20070006100A1 (en) * | 2005-06-29 | 2007-01-04 | Virpi Roto | Mobile communication terminal |
KR100664959B1 (en) * | 2005-07-07 | 2007-01-04 | 삼성전자주식회사 | Apparatus and method for image clustering |
KR101345284B1 (en) * | 2005-07-20 | 2013-12-27 | 한국과학기술원 | Method and apparatus for encoding/playing multimedia contents |
KR101304480B1 (en) * | 2005-07-20 | 2013-09-05 | 한국과학기술원 | Method and apparatus for encoding multimedia contents and method and system for applying encoded multimedia contents |
KR101406843B1 (en) * | 2006-03-17 | 2014-06-13 | 한국과학기술원 | Method and apparatus for encoding multimedia contents and method and system for applying encoded multimedia contents |
KR20070011092A (en) * | 2005-07-20 | 2007-01-24 | 삼성전자주식회사 | Method and apparatus for encoding multimedia contents and method and system for applying encoded multimedia contents |
WO2007010187A1 (en) * | 2005-07-22 | 2007-01-25 | British Telecommunications Public Limited Company | Data handling system |
WO2007014203A2 (en) * | 2005-07-22 | 2007-02-01 | Zdk Interactive, Inc. | Database mining for customer targeting |
WO2007030757A2 (en) * | 2005-09-07 | 2007-03-15 | Visan Industries | Systems and methods for organizing media based on associated metadata |
KR100735553B1 (en) * | 2005-10-10 | 2007-07-04 | 삼성전자주식회사 | Method and apparatus for clustering photographs |
KR20070075993A (en) * | 2006-01-17 | 2007-07-24 | 삼성전자주식회사 | Method for saving input to category of photo in mobile communication terminal |
US7644373B2 (en) * | 2006-01-23 | 2010-01-05 | Microsoft Corporation | User interface for viewing clusters of images |
US7945142B2 (en) * | 2006-06-15 | 2011-05-17 | Microsoft Corporation | Audio/visual editing tool |
US7636728B2 (en) * | 2006-06-22 | 2009-12-22 | Microsoft Corporation | Media difference files for compressed catalog files |
US7792868B2 (en) | 2006-11-10 | 2010-09-07 | Microsoft Corporation | Data object linking and browsing tool |
US8301669B2 (en) * | 2007-01-31 | 2012-10-30 | Hewlett-Packard Development Company, L.P. | Concurrent presentation of video segments enabling rapid video file comprehension |
JP4257548B2 (en) * | 2007-01-31 | 2009-04-22 | ブラザー工業株式会社 | Printing device |
US20080189591A1 (en) * | 2007-01-31 | 2008-08-07 | Lection David B | Method and system for generating a media presentation |
US20080244373A1 (en) * | 2007-03-26 | 2008-10-02 | Morris Robert P | Methods, systems, and computer program products for automatically creating a media presentation entity using media objects from a plurality of devices |
JP4564512B2 (en) * | 2007-04-16 | 2010-10-20 | 富士通株式会社 | Display device, display program, and display method |
EP2165326A2 (en) * | 2007-05-31 | 2010-03-24 | Visan Industries | Systems and methods for rendering media |
US9361941B2 (en) * | 2007-08-02 | 2016-06-07 | Scenera Technologies, Llc | Method and systems for arranging a media object in a media timeline |
US8407605B2 (en) | 2009-04-03 | 2013-03-26 | Social Communications Company | Application sharing |
US8397168B2 (en) | 2008-04-05 | 2013-03-12 | Social Communications Company | Interfacing with a spatial virtual communication environment |
US7769806B2 (en) | 2007-10-24 | 2010-08-03 | Social Communications Company | Automated real-time data stream switching in a shared virtual area communication environment |
US9465892B2 (en) * | 2007-12-03 | 2016-10-11 | Yahoo! Inc. | Associating metadata with media objects using time |
US8150098B2 (en) * | 2007-12-20 | 2012-04-03 | Eastman Kodak Company | Grouping images by location |
US20100016074A1 (en) * | 2008-03-20 | 2010-01-21 | Gdi Game Domain International Plc | Apparatus and methods for game conversion |
EP2279472A4 (en) | 2008-04-05 | 2013-11-20 | Social Communications Co | Shared virtual area communication environment based apparatus and methods |
CN106845645B (en) * | 2008-05-01 | 2020-08-04 | 启创互联公司 | Method and system for generating semantic network and for media composition |
US8638858B2 (en) | 2008-07-08 | 2014-01-28 | Intellectual Ventures Fund 83 Llc | Method, apparatus and system for converging images encoded using different standards |
US9818157B2 (en) | 2008-10-07 | 2017-11-14 | State Farm Mutual Automobile Insurance Company | Method for using electronic metadata to verify insurance claims |
US20110184955A1 (en) * | 2008-10-31 | 2011-07-28 | April Sleyoen Mitchell | Organizing data |
US9319357B2 (en) | 2009-01-15 | 2016-04-19 | Social Communications Company | Context based virtual area creation |
US9853922B2 (en) | 2012-02-24 | 2017-12-26 | Sococo, Inc. | Virtual area communications |
TWI393378B (en) * | 2009-04-07 | 2013-04-11 | Inst Information Industry | Hotspot analysis systems and methods, and computer program products thereof |
CN102460433A (en) * | 2009-06-15 | 2012-05-16 | 皇家飞利浦电子股份有限公司 | Method and apparatus for selecting representative image |
US8762889B2 (en) * | 2009-09-23 | 2014-06-24 | Vidan Industries | Method and system for dynamically placing graphic elements into layouts |
US8600967B2 (en) * | 2010-02-03 | 2013-12-03 | Apple Inc. | Automatic organization of browsing histories |
US8655893B2 (en) * | 2010-07-16 | 2014-02-18 | Shutterfly, Inc. | Organizing images captured by multiple image capture devices |
US20120213404A1 (en) | 2011-02-18 | 2012-08-23 | Google Inc. | Automatic event recognition and cross-user photo clustering |
WO2013181026A1 (en) | 2012-06-02 | 2013-12-05 | Social Communications Company | Interfacing with a spatial virtual communications environment |
US9336302B1 (en) | 2012-07-20 | 2016-05-10 | Zuci Realty Llc | Insight and algorithmic clustering for automated synthesis |
WO2014031126A1 (en) | 2012-08-24 | 2014-02-27 | Empire Technology Development Llc | Virtual reality applications |
WO2014035367A1 (en) * | 2012-08-27 | 2014-03-06 | Empire Technology Development Llc | Generating augmented reality exemplars |
US10691743B2 (en) * | 2014-08-05 | 2020-06-23 | Sri International | Multi-dimensional realization of visual content of an image collection |
US9311310B2 (en) * | 2012-10-26 | 2016-04-12 | Google Inc. | System and method for grouping related photographs |
US9298830B2 (en) * | 2013-05-01 | 2016-03-29 | Timothy Alan Barrett | Method, system and apparatus for facilitating discovery of items sharing common attributes |
JP2014229178A (en) * | 2013-05-24 | 2014-12-08 | 株式会社東芝 | Electronic apparatus, display control method, and program |
US20170212906A1 (en) * | 2014-07-30 | 2017-07-27 | Hewlett- Packard Development Company, L.P. | Interacting with user interfacr elements representing files |
US9749718B1 (en) * | 2016-07-20 | 2017-08-29 | Cisco Technology, Inc. | Adaptive telemetry based on in-network cross domain intelligence |
US11205103B2 (en) | 2016-12-09 | 2021-12-21 | The Research Foundation for the State University | Semisupervised autoencoder for sentiment analysis |
US11709925B1 (en) * | 2018-09-27 | 2023-07-25 | Amazon Technologies, Inc. | Visual token passwords |
Citations (11)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5146552A (en) * | 1990-02-28 | 1992-09-08 | International Business Machines Corporation | Method for associating annotation with electronically published material |
US5734893A (en) * | 1995-09-28 | 1998-03-31 | Ibm Corporation | Progressive content-based retrieval of image and video with adaptive and iterative refinement |
US5852823A (en) * | 1996-10-16 | 1998-12-22 | Microsoft | Image classification and retrieval system using a query-by-example paradigm |
US5875446A (en) * | 1997-02-24 | 1999-02-23 | International Business Machines Corporation | System and method for hierarchically grouping and ranking a set of objects in a query context based on one or more relationships |
US5893126A (en) * | 1994-09-30 | 1999-04-06 | Intel Corporation | Method and apparatus for annotating a computer document incorporating sound |
US5895464A (en) * | 1997-04-30 | 1999-04-20 | Eastman Kodak Company | Computer program product and a method for using natural language for the description, search and retrieval of multi-media objects |
US6085185A (en) * | 1996-07-05 | 2000-07-04 | Hitachi, Ltd. | Retrieval method and system of multimedia database |
US6111586A (en) * | 1996-03-15 | 2000-08-29 | Fujitsu Limited | Electronic photo album editing apparatus |
US6374260B1 (en) * | 1996-05-24 | 2002-04-16 | Magnifi, Inc. | Method and apparatus for uploading, indexing, analyzing, and searching media content |
US20020044690A1 (en) * | 2000-10-18 | 2002-04-18 | Burgess Ken L. | Method for matching geographic information with recorded images |
US6636648B2 (en) * | 1999-07-02 | 2003-10-21 | Eastman Kodak Company | Albuming method with automatic page layout |
Family Cites Families (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
AU1569797A (en) | 1995-12-14 | 1997-07-03 | Motorola, Inc. | Electronic book and method of annotation therefor |
US6687876B1 (en) | 1998-12-30 | 2004-02-03 | Fuji Xerox Co., Ltd. | Method and system for maintaining freeform ink annotations on changing views |
-
2001
- 2001-12-19 US US10/021,255 patent/US7076503B2/en not_active Expired - Fee Related
-
2005
- 2005-11-21 US US11/282,610 patent/US20060074973A1/en not_active Abandoned
Patent Citations (11)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5146552A (en) * | 1990-02-28 | 1992-09-08 | International Business Machines Corporation | Method for associating annotation with electronically published material |
US5893126A (en) * | 1994-09-30 | 1999-04-06 | Intel Corporation | Method and apparatus for annotating a computer document incorporating sound |
US5734893A (en) * | 1995-09-28 | 1998-03-31 | Ibm Corporation | Progressive content-based retrieval of image and video with adaptive and iterative refinement |
US6111586A (en) * | 1996-03-15 | 2000-08-29 | Fujitsu Limited | Electronic photo album editing apparatus |
US6374260B1 (en) * | 1996-05-24 | 2002-04-16 | Magnifi, Inc. | Method and apparatus for uploading, indexing, analyzing, and searching media content |
US6085185A (en) * | 1996-07-05 | 2000-07-04 | Hitachi, Ltd. | Retrieval method and system of multimedia database |
US5852823A (en) * | 1996-10-16 | 1998-12-22 | Microsoft | Image classification and retrieval system using a query-by-example paradigm |
US5875446A (en) * | 1997-02-24 | 1999-02-23 | International Business Machines Corporation | System and method for hierarchically grouping and ranking a set of objects in a query context based on one or more relationships |
US5895464A (en) * | 1997-04-30 | 1999-04-20 | Eastman Kodak Company | Computer program product and a method for using natural language for the description, search and retrieval of multi-media objects |
US6636648B2 (en) * | 1999-07-02 | 2003-10-21 | Eastman Kodak Company | Albuming method with automatic page layout |
US20020044690A1 (en) * | 2000-10-18 | 2002-04-18 | Burgess Ken L. | Method for matching geographic information with recorded images |
Cited By (26)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20050278311A1 (en) * | 2004-06-07 | 2005-12-15 | Trent Moore | System and method for generating advertisements utilizing a database of stock imagery |
US7831599B2 (en) * | 2005-03-04 | 2010-11-09 | Eastman Kodak Company | Addition of new images to an image database by clustering according to date/time and image content and representative image comparison |
US20060200475A1 (en) * | 2005-03-04 | 2006-09-07 | Eastman Kodak Company | Additive clustering of images lacking individualized date-time information |
US20070143300A1 (en) * | 2005-12-20 | 2007-06-21 | Ask Jeeves, Inc. | System and method for monitoring evolution over time of temporal content |
US20070260586A1 (en) * | 2006-05-03 | 2007-11-08 | Antonio Savona | Systems and methods for selecting and organizing information using temporal clustering |
US20070270985A1 (en) * | 2006-05-16 | 2007-11-22 | Canon Kabushiki Kaisha | Method for navigating large image sets using sort orders |
US8090712B2 (en) * | 2006-05-16 | 2012-01-03 | Canon Kabushiki Kaisha | Method for navigating large image sets using sort orders |
US20080052321A1 (en) * | 2006-08-23 | 2008-02-28 | Canon Kabushiki Kaisha | Information display apparatus and method |
US7904463B2 (en) * | 2006-08-23 | 2011-03-08 | Canon Kabushiki Kaisha | Information display apparatus and method |
US7970721B2 (en) | 2007-06-15 | 2011-06-28 | Microsoft Corporation | Learning and reasoning from web projections |
WO2008157027A1 (en) * | 2007-06-15 | 2008-12-24 | Microsoft Corporation | Learning and reasoning from web projections |
US20080313119A1 (en) * | 2007-06-15 | 2008-12-18 | Microsoft Corporation | Learning and reasoning from web projections |
US20090070346A1 (en) * | 2007-09-06 | 2009-03-12 | Antonio Savona | Systems and methods for clustering information |
US20100257252A1 (en) * | 2009-04-01 | 2010-10-07 | Microsoft Corporation | Augmented Reality Cloud Computing |
US20120106859A1 (en) * | 2009-06-24 | 2012-05-03 | Philip Cheatle | Image Album Creation |
US9049419B2 (en) * | 2009-06-24 | 2015-06-02 | Hewlett-Packard Development Company, L.P. | Image album creation |
US20120201465A1 (en) * | 2011-02-04 | 2012-08-09 | Olympus Corporation | Image processing apparatus |
US8923629B2 (en) | 2011-04-27 | 2014-12-30 | Hewlett-Packard Development Company, L.P. | System and method for determining co-occurrence groups of images |
CN103366387A (en) * | 2012-02-28 | 2013-10-23 | 索尼公司 | Selecting between clustering techniques for displaying images |
US20130222696A1 (en) * | 2012-02-28 | 2013-08-29 | Sony Corporation | Selecting between clustering techniques for displaying images |
US8972490B1 (en) * | 2012-06-29 | 2015-03-03 | Emc Corporation | User discovery in socially-aware data storage systems |
US20150088890A1 (en) * | 2013-09-23 | 2015-03-26 | Spotify Ab | System and method for efficiently providing media and associated metadata |
US9529888B2 (en) * | 2013-09-23 | 2016-12-27 | Spotify Ab | System and method for efficiently providing media and associated metadata |
US20160065443A1 (en) * | 2014-08-26 | 2016-03-03 | Sugarcrm Inc. | Retroreflective object tagging |
US10169373B2 (en) * | 2014-08-26 | 2019-01-01 | Sugarcrm Inc. | Retroreflective object tagging |
US10482109B2 (en) * | 2015-09-29 | 2019-11-19 | Canon Kabushiki Kaisha | Object indexing method, object searching method, and object indexing system |
Also Published As
Publication number | Publication date |
---|---|
US7076503B2 (en) | 2006-07-11 |
US20030009469A1 (en) | 2003-01-09 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US7076503B2 (en) | Managing media objects in a database | |
US9524349B2 (en) | Identifying particular images from a collection | |
Platt | AutoAlbum: Clustering digital photographs using probabilistic model merging | |
Jaffe et al. | Generating summaries and visualization for large collections of geo-referenced photographs | |
US8150170B2 (en) | Statistical approach to large-scale image annotation | |
US7286723B2 (en) | System and method for organizing images | |
US8156123B2 (en) | Method and apparatus for processing metadata | |
US6721729B2 (en) | Method and apparatus for electronic file search and collection | |
US7660445B2 (en) | Method for selecting an emphasis image from an image collection based upon content recognition | |
US7831599B2 (en) | Addition of new images to an image database by clustering according to date/time and image content and representative image comparison | |
US7788267B2 (en) | Image metadata action tagging | |
US20060153460A1 (en) | Method and apparatus for clustering digital photos based on situation and system and method for albuming using the same | |
US9009163B2 (en) | Lazy evaluation of semantic indexing | |
US20050280719A1 (en) | Method, medium, and apparatus for detecting situation change of digital photo and method, medium, and apparatus for situation-based photo clustering in digital photo album | |
US8117210B2 (en) | Sampling image records from a collection based on a change metric | |
EP2405371A1 (en) | Method for grouping events detected in an image collection | |
US20040098362A1 (en) | Automated propagation of document metadata | |
US20070136680A1 (en) | System and method for selecting pictures for presentation with text content | |
JP2006216026A (en) | Efficient method for temporal event clustering of digital photograph | |
JP2002007413A (en) | Image retrieving device | |
EP1973046A1 (en) | Indexing presentation slides | |
Kim et al. | User‐Friendly Personal Photo Browsing for Mobile Devices | |
Tukaram et al. | Retrieving Images On World Wide Web | |
Veeramani | Multimedia Databases | |
AOKI et al. | MPEG-7 Content-Based Image Retrieval with Spatial and Temporal Information |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STCB | Information on status: application discontinuation |
Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION |
|
AS | Assignment |
Owner name: MICROSOFT TECHNOLOGY LICENSING, LLC, WASHINGTON Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:MICROSOFT CORPORATION;REEL/FRAME:034766/0001 Effective date: 20141014 |