USRE43260E1 - Method for clustering and querying media items - Google Patents

Method for clustering and querying media items Download PDF

Info

Publication number
USRE43260E1
USRE43260E1 US12/150,100 US15010008A USRE43260E US RE43260 E1 USRE43260 E1 US RE43260E1 US 15010008 A US15010008 A US 15010008A US RE43260 E USRE43260 E US RE43260E
Authority
US
United States
Prior art keywords
media items
cluster
computer program
clusters
descriptive information
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Expired - Fee Related, expires
Application number
US12/150,100
Inventor
Joonas Paalasmaa
Jukka-Pekka Salmenkaita
Antti Sorvari
Tapio Tallgren
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Nokia Oyj
Original Assignee
Nokia Oyj
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Nokia Oyj filed Critical Nokia Oyj
Priority to US12/150,100 priority Critical patent/USRE43260E1/en
Application granted granted Critical
Publication of USRE43260E1 publication Critical patent/USRE43260E1/en
Expired - Fee Related legal-status Critical Current
Adjusted expiration legal-status Critical

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/50Information retrieval; Database structures therefor; File system structures therefor of still image data
    • G06F16/55Clustering; Classification
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/50Information retrieval; Database structures therefor; File system structures therefor of still image data
    • G06F16/58Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10TECHNICAL SUBJECTS COVERED BY FORMER USPC
    • Y10STECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10S707/00Data processing: database and file management or data structures
    • Y10S707/99931Database or file accessing
    • Y10S707/99933Query processing, i.e. searching
    • Y10S707/99936Pattern matching access
    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10TECHNICAL SUBJECTS COVERED BY FORMER USPC
    • Y10STECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10S707/00Data processing: database and file management or data structures
    • Y10S707/99941Database schema or data structure
    • Y10S707/99944Object-oriented database structure
    • Y10S707/99945Object-oriented database structure processing
    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10TECHNICAL SUBJECTS COVERED BY FORMER USPC
    • Y10STECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10S707/00Data processing: database and file management or data structures
    • Y10S707/99941Database schema or data structure
    • Y10S707/99948Application of database or data structure, e.g. distributed, multimedia, or image

Definitions

  • the present invention relates to managing media items in data processing terminals. More particularly, the present invention is directed to a method, a device and a computer program product for arranging, viewing and querying media items organized in hierarchical multidimensional clusters in mobile terminals.
  • the publication presents a method and an apparatus for organizing media objects in a database using contextual information for a media object and known media objects, categories, indexes and searches, to arrive at an inference for cataloging the media object in the database.
  • the method and the apparatus are provided for clustering media objects by forming groups of unlabeled data and applying a distance metric to said group.
  • Media objects are automatically organized into various collections by clustering images that are taken near each other in time.
  • a user interface may include one image per collection, where the image is shown to the user. If the user is searching for an image, the user views the images respectively representing collections of images and selects a collection that appears to relate to the desired image. Once a collection is selected, the images corresponding to the collection are shown to the user.
  • the media query problems are usually solved by folder-based approach in local storage (memory card or similar), but this has all the same limitations as the folder-based approach in the desktop environment.
  • the media query problem in a mobile terminal is solved by an access to a remote media collection via a mobile net connection, wherein the user interface logic (use of categories, keywords, etc.) is handled in the server-side.
  • This approach has the benefit of being potentially able to incorporate very advanced metadata-assisted queries, providing the appropriate logic has been implemented in the server-side.
  • this approach is not plausible if the network connection is not available for some reason.
  • the current invention is a client-side approach and the implementation can be carried out in the mobile device.
  • the current invention presents a method and a device and a computer program product for managing media items in mobile terminals. Particularly the current invention focuses on arranging, viewing and querying media items organized in hierarchical multidimensional clusters in mobile terminals, which overcome user interface constraints for metadata-assisted media query in mobile terminals.
  • the invention presents a method for multidimensional clustering and for querying the media items from said clusters and for automatically selecting the depth of cluster hierarchy.
  • the present invention also provides a user interface with a query mechanism to be used with clusters.
  • the media items are provided with descriptive information, a dimension, wherein the media items that have one descriptive information in common are clustered together.
  • the descriptive information is configured as metadata which can be inserted to media item file manually by the user or automatically.
  • suitable descriptive information is location and time, whereupon the cluster contains media items acquired in a certain place at a certain time.
  • the cluster comprising the collection of media items is shown to the user.
  • the user interface according to the invention is arranged so that one cluster is shown as a single item among other individual items in the user interface.
  • another view is opened and the items of that cluster are shown to the user.
  • the benefit of the clustering is that a list of media items being shown to the user is shorter than in the prior art solution (where all the items are shown in one list), which mitigates the limited display capabilities of mobile terminals.
  • the clustering also helps for collecting media items being somehow linked depending on the descriptive information, logically to the same view. It also offers enough information for the user to quickly see the content of the cluster.
  • Cluster naming facilitates organizing the clusters and the media items to the media collections.
  • a media manager according to the invention is available anytime and anywhere, when implemented in a mobile terminal.
  • the specific user interface takes into account the limitations of display capabilities of a mobile terminal and reduces them.
  • the media manager also enables the end-users to construct complex queries only with a limited “point-and-click”, which further creates a chance for automatic adaptation of media query based on the user's previous query behavior and thus reducing the end-users' query formation effort in subsequent query formation situations.
  • FIG. 1 visualizes a cluster area and the changing location of the user
  • FIG. 2 illustrates the example hierarchy of the media items in the display of a mobile terminal
  • FIG. 3 illustrates one example of the electronic device according to the invention.
  • FIG. 4 presents the method according to the invention as a simplified flowchart.
  • the current invention applies methods of data mining and clustering to automatically assist end-users of mobile terminals to generate complex media queries with little effort.
  • the invention is very preferable and advantageous when considering mobile terminals with personal media management software capability and the severe limits of the available user interface technology in those terminals.
  • the invention enables utilization of complex categorization schemes, including deep multidimensional metadata hierarchies to select desired parts of media collection in a mobile device.
  • the method according to the invention is presented as a very simplified flowchart in FIG. 4 .
  • the method according to the invention can be used with different types of media items, but images are used in the following example.
  • an image When an image is taken, it is provided with metadata comprising descriptive information of the image. Then other images or clusters are searched for. Searching focuses on images or clusters taken less than X meters away from the place the current image was taken at and taken on the same day, or the searching can be done by comparing other descriptive information of the items. If that kind of an image or cluster is found, a cluster containing the former images and the new one is created.
  • clusters can also be formed by using only cell ID data by forming a cluster of images taken on the same day in the same cell. If the user has identified (e.g. using landmarks management application) that a group of cell IDs corresponds to one named location (e.g. Summer cottage), then all images taken during the same day in the identified group of cells can form a cluster. Examples of other available location-related information that can be used are location area code (GSM), country code (GSM) and service area identification (WCDMA).
  • GSM location area code
  • GSM country code
  • WCDMA service area identification
  • Images that are temporally inside a relatively tight cluster but do not belong to it can also be added to the cluster.
  • a man is working on a building project at a summer cottage and takes a few pictures there.
  • the shop decides to drive to the nearby shop to buy groceries.
  • the shop he snaps a picture of a funny misspelled sign.
  • the picture snapped at the shop can be added to the summer cottage cluster, because it strongly relates to summer cottage pictures of that day.
  • FIG. 1 visualizes the situation.
  • the points marked with letters A 1 -A 6 indicate snapped images
  • the curve B between the points A 1 -A 6 indicates the location of the user
  • the rectangle with the dotted line defines the cluster area C.
  • One possible way of defining whether a picture can be added to a cluster is to check whether the picture fulfills the following conditions:
  • the picture must be temporally inside a cluster.
  • n refers to some fixed adaptable limit value.
  • the GPS device can be integrated to the device of the invention.
  • the location data can be acquired e.g. at the time of taking the image or periodically. If the location data is not available, the location can be tracked with e.g. cell ID.
  • the automatic tracking of the location can also be done, instead of GPS, by using some other positioning system e.g. different GPS-systems (A-GPS, D-GPS), angle of arrival (AOA), enhanced observed time difference (E-OTD), time difference of arrival (T-DOA), time of arrival (TOA), or the user can define the location coordinates manually.
  • the manually defined coordinates are stored in the location database.
  • the database includes information about the places (“summer cottage”) and coordinates corresponding to them. Location of the terminal and tracking should be done all the time. If the tracking were done only every time a picture is taken, there would be too few tracked places and that would not be sufficient for the calculations.
  • first descriptive information is “hobby” and the other descriptive information is fishing, skiing, golfing, etc and/or a time.
  • the queries can then be made according to the entry, e.g., images of fishing in January 2003.
  • first descriptive information is “people” and then the other descriptive information can be wife, co-workers, child, etc.
  • Labeling can be automatic by using cluster descriptive information, or manual.
  • cluster descriptive information or manual.
  • One practice is to compose a label of information about the place where the images in the cluster were taken at, the time, when they were taken, and how many images there are in the cluster. If the coordinate information is not available, the closeness can be determined by tracking the number of cell ID changes by using higher-level network information, such as location area codes. By assuming a certain upper limit for the speed in which the terminal can move, time information can also be used to determine closeness. Images taken within a short time period are also taken relatively close to each other.
  • the radius information can be utilized in forming clusters in naming clusters. Images inside the Landmark radius are considered to be taken in the same place. Even if images are not taken inside any Landmark, the Landmark name can still be used in naming e.g. “close to Summer cottage” where “Summer cottage” is a landmark name.
  • the name of the cluster can be at least partially based on a name queried from a remote server or terminal database that can provide the user with understandable names for locations (based on cluster coordinates/cell ID/location area code etc.).
  • a cluster name can contain more than one location names (e.g. Finland, Helsinki, Ruoholahti).
  • Naming facilitates organizing the clusters and the images to media collections.
  • the use of different kinds of descriptive information enables different users to see the image information in a way that best suits them.
  • this cluster is preferably shown as a single item among the individual media items in a user interface.
  • the user interface shows an array formed by individual media items and clusters.
  • a view e.g. a list view, comprising one or several clusters can also include individual images that do not belong to any cluster.
  • the cluster can be easily differentiated from the individual images because of its visually different appearance. For example, the appearance can be formed by selecting one or more images of the cluster to be displayed beside the cluster's label and this way by representing the cluster visually. As an example, the selected image could be the one that was first snapped, because then the appearance of the cluster does not change even when new images are snapped and added to the cluster.
  • FIG. 2 illustrates the hierarchy of media items in the display of a mobile terminal.
  • the main menu is named “IMAGES” and it displays the array of clusters and the media items in parallel in one view 1.
  • the cluster is named oiler the descriptive information that is shared by the media items in it.
  • the name is a place where the images were taken (Summer cottage).
  • Other information of the cluster can also be shown in the header of the cluster, such as the date (22 May, 2003) and the final number (4 images) of images. Choosing and opening the cluster displays a next view 2 containing the images inside the cluster.
  • Clusters become events if they are renamed. If “Summer cottage” is renamed as “Flying a kite at summer cottage”, the cluster gets a real meaning and thus it is considered as an event. In some cases event information can also be obtained automatically e.g. by using calendar information.
  • clustering parameters can be selected accordingly or adapted based on the amount of media items that are present.
  • the most applicable sub-clustering options can be communicated to the end-user by e.g. visual cues already before the end-user selects that cluster for further examination.
  • the stepped clustering divides the clustering into two parts.
  • the clusters are preferably time and location-combinations, and the list of them is organized based on time.
  • sub-clusters can be formed.
  • the sub-clusters can be based, for example, on physical presence of people (based on e.g. named Bluetooth-device ID's), on attributes of media items (e.g. “indoors” or “outdoors” based on white-balance settings), on explicit metadata keywords/categories/tags assigned to the media items or on visual similarity of the media items, etc.
  • the clustering method is presented. There is descriptive information of time and location shown in the tables below. The hierarchy of time information is shown in table A and the hierarchy of location information is shown in table B.
  • the user When querying the images, the user at first selects the time information, e.g. February 2000. After this the location information can be selected.
  • the only locations shown in the selection list are the ones fulfilling the February 2000 criteria. In other words, the list, containing only those locations where the user has taken the pictures in February 2000, is shown. If the amount of the information in clusters is different from the information in the query (e.g. months in query and weeks or days in clusters), both images and clusters are shown in the list.
  • the first stage clustering works reasonably well for “recent media items”, e.g. only the latest week or month. However, if the end-users focus is not on recent media items, the first stage clustering can be based on e.g. location arranged in alphabetical (or hierarchical, if location hierarchy is available) order and first stage clustering approach is used for sub-clusters.
  • Methods for generating complex media queries for clusters are described. Methods can also be applied in the data-mining technique.
  • the following methods are for 1) identifying descriptive information in a categorization scheme that divides the collection into sub-spaces (clusters) of suitable size and number, and for 2) on-line analysis of user behavior to automatically identify patterns in query formation that can be applied in further queries.
  • the treelike structure behind it is easy to see.
  • the following methods utilize the treelike structure in queries.
  • This scheme is primarily based on calculating such nodes in hierarchical categorization tree that divides the media item space into a suitable number of clusters. This scheme can reduce the number of navigational steps compared to whether the end-user starts from root node or accesses all the leaf nodes in list form.
  • function v(i) is defined for user-perceived annoyance for having to click i times to get a photo from the list.
  • V(T 1 , . . . , T m ) V(T 1 )+ . . . +V(T m )) where V indicates user annoyance and T 1 , . . . , T m are trees.
  • the list of trees (clusters) is what is presented to the user. Naturally the number of options is wanted to be limited to some reasonable number N (for example 4 to 8).
  • the user annoyance V can be reduced by providing shortcuts to commonly used parts of the tree. This is done by partitioning the initial tree T (which can be assumed to have a single root) to N subtrees T 1 , . . . , T N . In other words trees T 1 , . . . , T N are the subtrees of tree T. This partitions all items in the tree, whereupon V(T 1 , . . . , TN) is minimal. It is assumed that subtrees T 1 , . . . , T N have no common nodes.
  • the algorithm according to the invention calculates for each node the benefit of choosing that node for a root of a new tree. This is done by defining m subtrees.
  • T i For this function (e.g. for node n in tree T i ), the maximum value is chosen, after which T i is split into two parts, T i below n (including n) and T i without said part. Due to this kind of optimization (splitting T i up), only the values for the nodes above n and below n are needed to be re-calculated.
  • the calculation is modified depending on past end-user query formation, which has been analyzed for prioritizing the most likely selections by the end-user.
  • the media items are weighted based on whether they are either known or learned to be likely targets of the media item query. For example, high weight (>1) indicates media items that have been previously viewed often, shared or been associated with transactions, and low weight ( ⁇ 1) indicates media items that are obsolete or not related to current context.
  • This scheme is primarily based on analyzing how media items are distributed to the different dimensions of the applied categorization scheme. With this scheme the dimensions that most effectively divide the media item space into suitable sub-spaces can be identified.
  • the preferable implementation utilizes the methods described above in all dimensions before analyzing the distribution. Criteria for the best dimension can be e.g. 1) how evenly the media items are divided into the calculated sub-trees or 2) what is the average number of navigation steps required to reach media items.
  • the calculation is modified depending on past end-user query formation, which has been analyzed for accounting for personal preferences in query information (for one person it is intuitive to search first for person, then location and for some other person vice versa).
  • media items can be weighted based on whether they are either known or learned to be likely targets of the media item query. For example, high weight (>1) indicates media items that have been previously viewed often, shared, or been associated with transactions, and low weight ( ⁇ 1) indicates media items that are obsolete or not related to current context.
  • high weight >1 indicates media items that have been previously viewed often, shared, or been associated with transactions
  • low weight ⁇ 1 indicates media items that are obsolete or not related to current context.
  • the scheme can be modified based on the analysis of how different queries have been previously applied in different contexts.
  • the end-user scrolls the list up and down to browse categories within one dimension, moves the right/left button to switch between the dimensions (not choosing any), selects (press down) to drill into subcategories within the wanted dimension and selects (soft key) the current category to be part of the query.
  • the device should utilize a hierarchical multidimensional categorization scheme and have navigational means of 6 keys in minimum or similar (e.g. 5-way button, one soft key) to demonstrate the basics of both “X” and “Y” aspects of query formation (X representing how to select automatically/assisted dimension, i.e. “location”/“person”/“event”/and Y representing how to select automatically/assisted the depth within on hierarchical dimension, i.e. “Finland”/“Helsinki”/“Center”/).
  • FIG. 3 shows an example of the electronic device MS according to the invention.
  • the media item manager MM according to the invention can be implemented as a part of a data processing unit CPU in an electronic device MS.
  • the media manager MM can be within server-side of so called media album servers, and can be reached through a network by the electronic device MS.
  • it is more useful to store the full metadata available in a personal device for example for privacy reasons, whereupon the client-side implementation of media item manager MM is preferable.
  • the electronic device can comprise some other applications APP as well.
  • the electronic device MS stores a media collection in the memory MEM.
  • the media collection is acquired, for example, through some known data transfer connection. However, there preferably is a digital camera attached to or integrated in said electronic device MS wherein the images taken with said camera are directly stored into the memory MEM.
  • the media collection is queried and viewed through a user interface UI.
  • the electronic device MS is preferably a terminal with mobile communication and photographing capabilities, e.g. a camera phone.

Abstract

The present invention relates to managing media items in data processing terminals. More particularly, the present invention is directed to a method, a device and a computer program product for arranging, viewing and querying media items organized in hierarchical multidimensional clusters in mobile terminals. Media items are arranged by clustering with multiple dimensions, wherein they are queried by defining the first entry for one dimension, wherein the next entry is based on the other dimension from the media items fulfilling the first entry.

Description

FIELD OF THE INVENTION
The present invention relates to managing media items in data processing terminals. More particularly, the present invention is directed to a method, a device and a computer program product for arranging, viewing and querying media items organized in hierarchical multidimensional clusters in mobile terminals.
BACKGROUND OF THE INVENTION
Software applications that manage media collections have become widely adopted as the amount of digital media, including images, has grown. State-of-the-art programs utilize metadata, or information about the media items managed, to help categorizing media collection. Prior art has concentrated on solutions that typically work on personal computers with associated display and other user interface capabilities. Development of mobile communication and computing technology, however, has made it possible to have similar media collections also in mobile personal communication devices with more constrained user interface capabilities.
There are software applications, for example Adobe Album®, that are developed for managing media collections that are stored in personal computers. One example of the prior art techniques is presented in international publication WO 02/057959A2 “Digital media management apparatus and methods” by Adobe Systems. The publication presents a method and an apparatus for managing, finding and displaying objects, such as digital images. The objects are associated with descriptive textual and numeric data (“metadata”) and stored in a relational database from which they can be selected, sorted and found. These objects can be searched for and displayed according to the degree to which their metadata matches the search criteria. Objects that are in the different match groups can be differentiated from one another in the display area by visual cues, such as being displayed in front of different background colors or patterns.
One example of a method for managing media objects is presented in publication US2003/0009469A1 “Managing media objects in a database” by Microsoft Corporation. The publication presents a method and an apparatus for organizing media objects in a database using contextual information for a media object and known media objects, categories, indexes and searches, to arrive at an inference for cataloging the media object in the database. The method and the apparatus are provided for clustering media objects by forming groups of unlabeled data and applying a distance metric to said group. Media objects are automatically organized into various collections by clustering images that are taken near each other in time. A user interface may include one image per collection, where the image is shown to the user. If the user is searching for an image, the user views the images respectively representing collections of images and selects a collection that appears to relate to the desired image. Once a collection is selected, the images corresponding to the collection are shown to the user.
It can be seen that the above-described methods suit personal computers well, but have usability and operational problems if transferred into mobile environment. The existing methods are not that feasible in all mobile terminal categories due to being dependent on user's capability to view a display of considerable size and to select media items, categories etc. by point-and-click methods, such as a mouse. However, it would be highly preferable for the end-user to have corresponding functionality in a personal mobile terminal, thus providing users with access to their media collections even when the personal computers are not accessible.
In mobile terminals the media query problems are usually solved by folder-based approach in local storage (memory card or similar), but this has all the same limitations as the folder-based approach in the desktop environment. In the prior art methods the media query problem in a mobile terminal is solved by an access to a remote media collection via a mobile net connection, wherein the user interface logic (use of categories, keywords, etc.) is handled in the server-side. This approach has the benefit of being potentially able to incorporate very advanced metadata-assisted queries, providing the appropriate logic has been implemented in the server-side. However, this approach is not plausible if the network connection is not available for some reason.
For the above-mentioned reasons it is necessary to develop a new method for managing large amounts of media items. The method should be reasonably easy to use even in small displays and it should provide practical access only to limited selection mechanisms. The current invention is a client-side approach and the implementation can be carried out in the mobile device.
SUMMARY OF THE INVENTION
The current invention presents a method and a device and a computer program product for managing media items in mobile terminals. Particularly the current invention focuses on arranging, viewing and querying media items organized in hierarchical multidimensional clusters in mobile terminals, which overcome user interface constraints for metadata-assisted media query in mobile terminals. The invention presents a method for multidimensional clustering and for querying the media items from said clusters and for automatically selecting the depth of cluster hierarchy. The present invention also provides a user interface with a query mechanism to be used with clusters.
Due to the invention the media items are provided with descriptive information, a dimension, wherein the media items that have one descriptive information in common are clustered together. The descriptive information is configured as metadata which can be inserted to media item file manually by the user or automatically. One example of suitable descriptive information is location and time, whereupon the cluster contains media items acquired in a certain place at a certain time.
The cluster comprising the collection of media items is shown to the user. The user interface according to the invention is arranged so that one cluster is shown as a single item among other individual items in the user interface. When the user selects the cluster, another view is opened and the items of that cluster are shown to the user.
The benefit of the clustering is that a list of media items being shown to the user is shorter than in the prior art solution (where all the items are shown in one list), which mitigates the limited display capabilities of mobile terminals. The clustering also helps for collecting media items being somehow linked depending on the descriptive information, logically to the same view. It also offers enough information for the user to quickly see the content of the cluster. Cluster naming facilitates organizing the clusters and the media items to the media collections.
A media manager according to the invention is available anytime and anywhere, when implemented in a mobile terminal. The specific user interface takes into account the limitations of display capabilities of a mobile terminal and reduces them. The media manager also enables the end-users to construct complex queries only with a limited “point-and-click”, which further creates a chance for automatic adaptation of media query based on the user's previous query behavior and thus reducing the end-users' query formation effort in subsequent query formation situations.
The preferred embodiments of the invention are set forth in the drawings, in the detailed description which follows, and in the appended claims. Further objects and advantages of the invention are also considered in the description. The invention itself is defined with particularity in the claims.
BRIEF DESCRIPTION OF THE DRAWINGS
FIG. 1 visualizes a cluster area and the changing location of the user,
FIG. 2 illustrates the example hierarchy of the media items in the display of a mobile terminal,
FIG. 3 illustrates one example of the electronic device according to the invention, and
FIG. 4 presents the method according to the invention as a simplified flowchart.
DETAILED DESCRIPTION OF THE INVENTION
The current invention applies methods of data mining and clustering to automatically assist end-users of mobile terminals to generate complex media queries with little effort. The invention is very preferable and advantageous when considering mobile terminals with personal media management software capability and the severe limits of the available user interface technology in those terminals. In practice the invention enables utilization of complex categorization schemes, including deep multidimensional metadata hierarchies to select desired parts of media collection in a mobile device. The method according to the invention is presented as a very simplified flowchart in FIG. 4. The method according to the invention can be used with different types of media items, but images are used in the following example.
Forming Groups of Media Items
It is possible to divide images into groups by clustering them in a time-space coordinate system. However, applying multidimensional clustering where time and space coordinates are considered simultaneously may create confusing results. According to the invention, a stepwise clustering is applied where the images are clustered by date and by location into final groups. By using this solution, the user better understands the logic behind grouping and complexity can be avoided.
The following is an example of a use of the method. The variables can change due to the situation, wherein they should not be considered as limitations.
When an image is taken, it is provided with metadata comprising descriptive information of the image. Then other images or clusters are searched for. Searching focuses on images or clusters taken less than X meters away from the place the current image was taken at and taken on the same day, or the searching can be done by comparing other descriptive information of the items. If that kind of an image or cluster is found, a cluster containing the former images and the new one is created.
If there is no precise location information available, clusters can also be formed by using only cell ID data by forming a cluster of images taken on the same day in the same cell. If the user has identified (e.g. using landmarks management application) that a group of cell IDs corresponds to one named location (e.g. Summer cottage), then all images taken during the same day in the identified group of cells can form a cluster. Examples of other available location-related information that can be used are location area code (GSM), country code (GSM) and service area identification (WCDMA).
Images that are temporally inside a relatively tight cluster but do not belong to it can also be added to the cluster. In the example situation a man is working on a building project at a summer cottage and takes a few pictures there. In the middle of the day he decides to drive to the nearby shop to buy groceries. At the shop he snaps a picture of a funny misspelled sign. The picture snapped at the shop can be added to the summer cottage cluster, because it strongly relates to summer cottage pictures of that day. FIG. 1 visualizes the situation. The points marked with letters A1-A6 indicate snapped images, the curve B between the points A1-A6 indicates the location of the user, and the rectangle with the dotted line defines the cluster area C.
Pictures that are temporally inside a cluster, but do not belong to it, shall not just be added to the cluster. For instance, in a situation where some pictures are taken at home in the morning, some at work during the day, and then in the evening more pictures are taken at home, it is obvious that pictures taken at home fonts a cluster, but pictures snapped at work should not be added to it. Pictures that were taken temporally inside a cluster can be added to it, if the time period of the user being away from the cluster area is not too long. It should also be noticed that the distances between the locations where the pictures were taken and the centroid of a cluster should not be too long.
One possible way of defining whether a picture can be added to a cluster is to check whether the picture fulfills the following conditions:
1. The picture must be temporally inside a cluster.
2. t 1 t 2 dist ( t ) t n ,
where dist(t) is the distance between the user and the center of the cluster at time t. t1 is the time the user left the cluster area C and t2 is the time the user re-entered it (see FIG. 1). “n” refers to some fixed adaptable limit value.
Location of the user can be tracked several ways, for example by GPS device. The GPS device can be integrated to the device of the invention. The location data can be acquired e.g. at the time of taking the image or periodically. If the location data is not available, the location can be tracked with e.g. cell ID. The automatic tracking of the location can also be done, instead of GPS, by using some other positioning system e.g. different GPS-systems (A-GPS, D-GPS), angle of arrival (AOA), enhanced observed time difference (E-OTD), time difference of arrival (T-DOA), time of arrival (TOA), or the user can define the location coordinates manually. The manually defined coordinates are stored in the location database. The database includes information about the places (“summer cottage”) and coordinates corresponding to them. Location of the terminal and tracking should be done all the time. If the tracking were done only every time a picture is taken, there would be too few tracked places and that would not be sufficient for the calculations.
There can also be other descriptive information instead of location and time in the metadata of the media item. One suitable example is a situation where the first descriptive information is “hobby” and the other descriptive information is fishing, skiing, golfing, etc and/or a time. The queries can then be made according to the entry, e.g., images of fishing in January 2003. Yet another example for first descriptive information is “people” and then the other descriptive information can be wife, co-workers, child, etc. By understanding these examples, it becomes obvious that the descriptive information can concern almost anything.
Naming of Clusters
For identifying clusters, they are labeled with some informative name. Labeling can be automatic by using cluster descriptive information, or manual. One practice is to compose a label of information about the place where the images in the cluster were taken at, the time, when they were taken, and how many images there are in the cluster. If the coordinate information is not available, the closeness can be determined by tracking the number of cell ID changes by using higher-level network information, such as location area codes. By assuming a certain upper limit for the speed in which the terminal can move, time information can also be used to determine closeness. Images taken within a short time period are also taken relatively close to each other.
If coordinate-based position is available and the user has created Landmarks (named coordinate locations) with radius information, the radius information can be utilized in forming clusters in naming clusters. Images inside the Landmark radius are considered to be taken in the same place. Even if images are not taken inside any Landmark, the Landmark name can still be used in naming e.g. “close to Summer cottage” where “Summer cottage” is a landmark name. When naming the cluster, the name of the cluster can be at least partially based on a name queried from a remote server or terminal database that can provide the user with understandable names for locations (based on cluster coordinates/cell ID/location area code etc.). A cluster name can contain more than one location names (e.g. Finland, Helsinki, Ruoholahti).
If most of the images are taken e.g. in Finland and the user takes few images in Spain, it would be preferable to display the country name (Spain) instead of other more detailed location information. On the other hand, if the name of the place where the image was taken is unknown it is also possible to label clusters for example by Group(1), Group (2), etc.
The same naming principles can also be applied to individual images. Naming facilitates organizing the clusters and the images to media collections. The use of different kinds of descriptive information enables different users to see the image information in a way that best suits them.
User Interface
As described earlier, it is preferable to bundle images relating closely to each other—taken on the same day at the substantially same place—up into a cluster. According to the invention, this cluster is preferably shown as a single item among the individual media items in a user interface. On the other words, the user interface shows an array formed by individual media items and clusters. A view, e.g. a list view, comprising one or several clusters can also include individual images that do not belong to any cluster. The cluster can be easily differentiated from the individual images because of its visually different appearance. For example, the appearance can be formed by selecting one or more images of the cluster to be displayed beside the cluster's label and this way by representing the cluster visually. As an example, the selected image could be the one that was first snapped, because then the appearance of the cluster does not change even when new images are snapped and added to the cluster.
As an example, FIG. 2 illustrates the hierarchy of media items in the display of a mobile terminal. In this example the main menu is named “IMAGES” and it displays the array of clusters and the media items in parallel in one view 1. Instead of displaying four images snapped at the summer cottage on the 22nd of May 2003, only one of the four images is displayed as a cluster. The cluster is named oiler the descriptive information that is shared by the media items in it. In this example, the name is a place where the images were taken (Summer cottage). Other information of the cluster can also be shown in the header of the cluster, such as the date (22 May, 2003) and the final number (4 images) of images. Choosing and opening the cluster displays a next view 2 containing the images inside the cluster.
Every now and then a cluster can represent an event. Clusters become events if they are renamed. If “Summer cottage” is renamed as “Flying a kite at summer cottage”, the cluster gets a real meaning and thus it is considered as an event. In some cases event information can also be obtained automatically e.g. by using calendar information.
To keep the number of media items or clusters reasonably small, large clusters would be preferred. For this purpose, clustering parameters can be selected accordingly or adapted based on the amount of media items that are present. When large clusters are formed, it is essential to provide the means for accessing the sub-clusters. This can be achieved by applying the clustering process in a step-wise manner. Moreover, the most applicable sub-clustering options can be communicated to the end-user by e.g. visual cues already before the end-user selects that cluster for further examination.
The stepped clustering divides the clustering into two parts. At the first stage of the clustering, the clusters are preferably time and location-combinations, and the list of them is organized based on time. At the second stage of clustering, sub-clusters can be formed. The sub-clusters can be based, for example, on physical presence of people (based on e.g. named Bluetooth-device ID's), on attributes of media items (e.g. “indoors” or “outdoors” based on white-balance settings), on explicit metadata keywords/categories/tags assigned to the media items or on visual similarity of the media items, etc.
One example of the clustering method is presented. There is descriptive information of time and location shown in the tables below. The hierarchy of time information is shown in table A and the hierarchy of location information is shown in table B.
TABLE A
Year 2000
January
February
March
. . .
Year 2001
January
February
March
. . .
TABLE B
Finland
Helsinki
Tampere
Jyväskylä
Sweden
Stockholm
Estonia
Tallinn
When querying the images, the user at first selects the time information, e.g. February 2000. After this the location information can be selected. According to the invention, the only locations shown in the selection list are the ones fulfilling the February 2000 criteria. In other words, the list, containing only those locations where the user has taken the pictures in February 2000, is shown. If the amount of the information in clusters is different from the information in the query (e.g. months in query and weeks or days in clusters), both images and clusters are shown in the list.
When managing large media collections, the first stage clustering works reasonably well for “recent media items”, e.g. only the latest week or month. However, if the end-users focus is not on recent media items, the first stage clustering can be based on e.g. location arranged in alphabetical (or hierarchical, if location hierarchy is available) order and first stage clustering approach is used for sub-clusters.
Next, methods for generating complex media queries for clusters are described. Methods can also be applied in the data-mining technique. The following methods are for 1) identifying descriptive information in a categorization scheme that divides the collection into sub-spaces (clusters) of suitable size and number, and for 2) on-line analysis of user behavior to automatically identify patterns in query formation that can be applied in further queries. When considering an above-mentioned organization of media items, the treelike structure behind it is easy to see. The following methods utilize the treelike structure in queries.
The following schemes can be applied in a situation e.g. where the user has taken several hundreds of images in Finland and tens in several different cities. Few images are taken in Stockholm and Tallinn. When the user selects the location information, the available item could be Helsinki, Tampere, Jyväskylä, Sweden and Estonia or “other”. Additional criteria—such as most often used, etc.—can be used as well.
Automatic/Assisted Selection of Hierarchical Depth within a Dimension of Categorization Scheme
This scheme is primarily based on calculating such nodes in hierarchical categorization tree that divides the media item space into a suitable number of clusters. This scheme can reduce the number of navigational steps compared to whether the end-user starts from root node or accesses all the leaf nodes in list form.
First, function v(i) is defined for user-perceived annoyance for having to click i times to get a photo from the list. For example, v(i) can be v(i)=i or v(i) can be v(i)=pow(i, 1.5).
Next, V(T) is defined for a tree T as
V(T)=sum(v(len(n))*items(n):n in T)
where len(n) is the depth of node n in tree T.
Similarly for a list of trees:
V(T1, . . . , Tm)=V(T1)+ . . . +V(Tm))
where V indicates user annoyance and T1, . . . , Tm are trees.
The list of trees (clusters) is what is presented to the user. Naturally the number of options is wanted to be limited to some reasonable number N (for example 4 to 8).
The user annoyance V can be reduced by providing shortcuts to commonly used parts of the tree. This is done by partitioning the initial tree T (which can be assumed to have a single root) to N subtrees T1, . . . , TN. In other words trees T1, . . . , TN are the subtrees of tree T. This partitions all items in the tree, whereupon V(T1, . . . , TN) is minimal. It is assumed that subtrees T1, . . . , TN have no common nodes.
The algorithm according to the invention calculates for each node the benefit of choosing that node for a root of a new tree. This is done by defining m subtrees. The benefit of choosing a node as a root is calculated for each node n in subtrees T1, . . . , Tm:
function=sum(v(len(k)+l)*items(k))−sum(v(len(k)*items(k)))
wherein “k” is in “Ti” and “n” is in “Ti” and “len(n)=I” in Ti.
For this function (e.g. for node n in tree Ti), the maximum value is chosen, after which Ti is split into two parts, Ti below n (including n) and Ti without said part. Due to this kind of optimization (splitting Ti up), only the values for the nodes above n and below n are needed to be re-calculated.
The calculation is modified depending on past end-user query formation, which has been analyzed for prioritizing the most likely selections by the end-user. The media items are weighted based on whether they are either known or learned to be likely targets of the media item query. For example, high weight (>1) indicates media items that have been previously viewed often, shared or been associated with transactions, and low weight (<1) indicates media items that are obsolete or not related to current context.
Automatic/Assisted Selection of Dimension within Multidimensional Categorization Scheme
This scheme is primarily based on analyzing how media items are distributed to the different dimensions of the applied categorization scheme. With this scheme the dimensions that most effectively divide the media item space into suitable sub-spaces can be identified. The preferable implementation utilizes the methods described above in all dimensions before analyzing the distribution. Criteria for the best dimension can be e.g. 1) how evenly the media items are divided into the calculated sub-trees or 2) what is the average number of navigation steps required to reach media items.
The calculation is modified depending on past end-user query formation, which has been analyzed for accounting for personal preferences in query information (for one person it is intuitive to search first for person, then location and for some other person vice versa).
Also in this case media items can be weighted based on whether they are either known or learned to be likely targets of the media item query. For example, high weight (>1) indicates media items that have been previously viewed often, shared, or been associated with transactions, and low weight (<1) indicates media items that are obsolete or not related to current context. The scheme can be modified based on the analysis of how different queries have been previously applied in different contexts.
When using the schemes described above, the end-user scrolls the list up and down to browse categories within one dimension, moves the right/left button to switch between the dimensions (not choosing any), selects (press down) to drill into subcategories within the wanted dimension and selects (soft key) the current category to be part of the query. In order to allow this the device should utilize a hierarchical multidimensional categorization scheme and have navigational means of 6 keys in minimum or similar (e.g. 5-way button, one soft key) to demonstrate the basics of both “X” and “Y” aspects of query formation (X representing how to select automatically/assisted dimension, i.e. “location”/“person”/“event”/and Y representing how to select automatically/assisted the depth within on hierarchical dimension, i.e. “Finland”/“Helsinki”/“Center”/).
Implementation
FIG. 3 shows an example of the electronic device MS according to the invention. The media item manager MM according to the invention can be implemented as a part of a data processing unit CPU in an electronic device MS. The media manager MM can be within server-side of so called media album servers, and can be reached through a network by the electronic device MS. However, sometimes it is more useful to store the full metadata available in a personal device, for example for privacy reasons, whereupon the client-side implementation of media item manager MM is preferable. It is obvious that the electronic device can comprise some other applications APP as well.
The electronic device MS stores a media collection in the memory MEM. The media collection is acquired, for example, through some known data transfer connection. However, there preferably is a digital camera attached to or integrated in said electronic device MS wherein the images taken with said camera are directly stored into the memory MEM. The media collection is queried and viewed through a user interface UI. The electronic device MS is preferably a terminal with mobile communication and photographing capabilities, e.g. a camera phone.
The foregoing detailed description is provided for clearness of understanding only, and limitation should not necessarily be read therefrom into the claims herein.

Claims (30)

The invention claimed is:
1. A method, comprising:
providing individual media items with metadata comprising at least first and second descriptive information;
forming a first cluster of individual media items that have one descriptive information in common;
forming a second cluster of individual media items that have two descriptive information in common;
automatically sub-clustering together media items within a cluster in question when said media items within said cluster in question have further descriptive information in common;
providing a cluster hierarchy comprising at least the first and second clusters and any sub-clusters; and
presenting each cluster and any sub-clusters as an individual media item to a user interface.
2. The method according to claim 1, further comprising comparing a first individual media item to a plurality of individual media items or to at least said first and second clusters for determining whether to cluster said first individual media item with at least one of said plurality of individual media items or at least one of said first and second clusters.
3. The method according to claim 1, further comprising naming the cluster in question according to descriptive information the individual media items of the cluster in question have in common.
4. The method according to claim 3, wherein the cluster in question is named and updated manually, wherein the name is also updated to the corresponding storage system.
5. The method according to claim 1, further comprising displaying the cluster in question among the individual media items, but differentiated from the individual media items visually.
6. The method according to claim 1, further comprising managing media items and at least said first and second clusters, wherein managing comprises at least arranging, querying and viewing the media items.
7. The method according to claim 6, wherein querying the media items comprises defining a first entry for one descriptive information wherein a next entry is based on at least one subsequent descriptive information of media items fulfilling the first entry.
8. The method according to claim 7, wherein querying the media items is adapted automatically based on a user's previous query behaviour.
9. The method according to claim 6, wherein viewing the media items comprises showing an array of media items and at least said first and second clusters, wherein the media items inside the cluster in question are viewed after selecting the cluster in question.
10. The method according to claim 1, wherein the method is a client-side method.
11. The method according to claim 1, wherein said first descriptive information is the location of a terminal containing the media items.
12. The method according to the claim 11, where the location of the terminal containing the media items is automatically acquired from a positioning system or manually defined by the user.
13. The method according to claim 1, wherein said second descriptive information is the time of acquiring the media item.
14. The method according to claim 1, wherein the media item is an image.
15. A computer program product for managing media items, wherein the computer program product comprises a readable memory, a computer program stored in said readable memory, wherein the computer program comprises instructions executable on a process for
providing individual media items with metadata comprising at least first and second descriptive information;
forming a first cluster of individual media items that have one descriptive information in common;
forming a second cluster of individual media items that have two descriptive information in common;
automatically sub-clustering together media items within a cluster in question when said media items within said cluster in question have further descriptive information in common;
providing a cluster hierarchy comprising at least the first and second clusters and any sub-clusters; and
presenting each cluster and any sub-clusters as an individual media item to a user interface.
16. The method according to claim 1, wherein providing the individual media items comprises searching.
17. The computer program product according to claim 15, wherein the computer program further comprises instructions, that when executed by a data processing unit, cause an apparatus to perform comparing a first individual media item to a plurality of individual media items or to at least said first and second clusters for determining whether to cluster said first individual media item with at least one of said plurality of individual media items or at least one of said first and second clusters.
18. The computer program product according to claim 15, wherein the computer program further comprises instructions, that when executed by a data processing unit, cause an apparatus to perform naming the cluster in question according to descriptive information the individual media items of the cluster in question have in common.
19. The computer program product according to claim 18, wherein the cluster in question is named and updated manually, wherein the name is also updated to the corresponding storage system.
20. The computer program product according to claim 15, wherein the computer program further comprises instructions, that when executed by a data processing unit, cause an apparatus to perform displaying the cluster in question among the individual media items, but differentiated from the individual media items visually.
21. The computer program product according to claim 15, wherein the computer program further comprises instructions, that when executed by a data processing unit, cause an apparatus to perform managing media items and at least said first and second clusters, wherein managing comprises at least arranging, querying and viewing the media items.
22. The computer program product according to claim 21, wherein querying the media items comprises defining a first entry for one descriptive information wherein a next entry is based on at least one subsequent descriptive information of media items fulfilling the first entry.
23. The computer program product according to claim 22, wherein querying the media items is adapted automatically based on a user's previous query behavior.
24. The computer program product according to claim 21, wherein viewing the media items comprises showing an array of media items and at least said first and second clusters, wherein the media items inside the cluster in question are viewed after selecting the cluster in question.
25. The computer program product according to claim 15, wherein the computer program is a client-side computer program.
26. The computer program product according to claim 15, wherein said first descriptive information is the location of a terminal containing the media items.
27. The computer program product according to the claim 26, where the location of the terminal containing the media items is automatically acquired from a positioning system or manually defined by the user.
28. The computer program product according to claim 15, wherein said second descriptive information is the time of acquiring the media item.
29. The computer program product according to claim 15, wherein the media item is an image.
30. The computer program product according to claim 15, wherein providing the individual media items comprises searching.
US12/150,100 2003-10-02 2008-04-23 Method for clustering and querying media items Expired - Fee Related USRE43260E1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US12/150,100 USRE43260E1 (en) 2003-10-02 2008-04-23 Method for clustering and querying media items

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US10/678,591 US7313574B2 (en) 2003-10-02 2003-10-02 Method for clustering and querying media items
US12/150,100 USRE43260E1 (en) 2003-10-02 2008-04-23 Method for clustering and querying media items

Related Parent Applications (1)

Application Number Title Priority Date Filing Date
US10/678,591 Reissue US7313574B2 (en) 2003-10-02 2003-10-02 Method for clustering and querying media items

Publications (1)

Publication Number Publication Date
USRE43260E1 true USRE43260E1 (en) 2012-03-20

Family

ID=34393970

Family Applications (2)

Application Number Title Priority Date Filing Date
US10/678,591 Ceased US7313574B2 (en) 2003-10-02 2003-10-02 Method for clustering and querying media items
US12/150,100 Expired - Fee Related USRE43260E1 (en) 2003-10-02 2008-04-23 Method for clustering and querying media items

Family Applications Before (1)

Application Number Title Priority Date Filing Date
US10/678,591 Ceased US7313574B2 (en) 2003-10-02 2003-10-02 Method for clustering and querying media items

Country Status (6)

Country Link
US (2) US7313574B2 (en)
EP (1) EP1668540A1 (en)
JP (2) JP2007507775A (en)
KR (1) KR100803769B1 (en)
TW (1) TWI360058B (en)
WO (1) WO2005031601A1 (en)

Cited By (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10296166B2 (en) 2010-01-06 2019-05-21 Apple Inc. Device, method, and graphical user interface for navigating and displaying content in context
US10324973B2 (en) 2016-06-12 2019-06-18 Apple Inc. Knowledge graph metadata network based on notable moments
US10362219B2 (en) 2016-09-23 2019-07-23 Apple Inc. Avatar creation and editing
US10572132B2 (en) 2015-06-05 2020-02-25 Apple Inc. Formatting content for a reduced-size user interface
US10803135B2 (en) 2018-09-11 2020-10-13 Apple Inc. Techniques for disambiguating clustered occurrence identifiers
US10846343B2 (en) 2018-09-11 2020-11-24 Apple Inc. Techniques for disambiguating clustered location identifiers
US10891013B2 (en) 2016-06-12 2021-01-12 Apple Inc. User interfaces for retrieving contextually relevant media content
US11086935B2 (en) 2018-05-07 2021-08-10 Apple Inc. Smart updates from historical database changes
US11243996B2 (en) 2018-05-07 2022-02-08 Apple Inc. Digital asset search user interface
US11334209B2 (en) 2016-06-12 2022-05-17 Apple Inc. User interfaces for retrieving contextually relevant media content
US11782575B2 (en) 2018-05-07 2023-10-10 Apple Inc. User interfaces for sharing contextually relevant media content

Families Citing this family (48)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
FR2864279B1 (en) * 2003-12-18 2006-05-26 Eastman Kodak Co METHOD FOR ADDING CHARACTERIZATION DATA DURING IMAGE ENTRY
US7657060B2 (en) * 2004-03-31 2010-02-02 Microsoft Corporation Stylization of video
US20050246324A1 (en) * 2004-04-30 2005-11-03 Nokia Inc. System and associated device, method, and computer program product for performing metadata-based searches
JP2006065368A (en) * 2004-08-24 2006-03-09 Sony Corp Image display device and method, and computer program
US7716194B2 (en) * 2005-01-12 2010-05-11 Microsoft Corporation File management system employing time line based representation of data
US20060156237A1 (en) * 2005-01-12 2006-07-13 Microsoft Corporation Time line based user interface for visualization of data
US7788592B2 (en) * 2005-01-12 2010-08-31 Microsoft Corporation Architecture and engine for time line based visualization of data
US7479970B2 (en) * 2005-01-12 2009-01-20 Microsoft Corporation Systems and methods that facilitate process monitoring, navigation, and parameter-based magnification
US7831599B2 (en) * 2005-03-04 2010-11-09 Eastman Kodak Company Addition of new images to an image database by clustering according to date/time and image content and representative image comparison
US20060294096A1 (en) * 2005-06-22 2006-12-28 Eastman Kodak Company Additive clustering of images into events using capture date-time information
US8326775B2 (en) * 2005-10-26 2012-12-04 Cortica Ltd. Signature generation for multimedia deep-content-classification by a large-scale matching system and method thereof
US7668867B2 (en) 2006-03-17 2010-02-23 Microsoft Corporation Array-based discovery of media items
US8285595B2 (en) * 2006-03-29 2012-10-09 Napo Enterprises, Llc System and method for refining media recommendations
US9003056B2 (en) * 2006-07-11 2015-04-07 Napo Enterprises, Llc Maintaining a minimum level of real time media recommendations in the absence of online friends
US8327266B2 (en) 2006-07-11 2012-12-04 Napo Enterprises, Llc Graphical user interface system for allowing management of a media item playlist based on a preference scoring system
JP2008083787A (en) * 2006-09-26 2008-04-10 Sony Corp Table display method, information setting method, information processor, program for table display, and program for information setting
US20080085055A1 (en) * 2006-10-06 2008-04-10 Cerosaletti Cathleen D Differential cluster ranking for image record access
JP2008134966A (en) * 2006-11-29 2008-06-12 Sony Corp Data management server, data management system, data management method and program
JP2008134866A (en) * 2006-11-29 2008-06-12 Sony Corp Content browsing method, content browsing device and content browsing program
US20080201000A1 (en) * 2007-02-20 2008-08-21 Nokia Corporation Contextual grouping of media items
US7941764B2 (en) 2007-04-04 2011-05-10 Abo Enterprises, Llc System and method for assigning user preference settings for a category, and in particular a media category
US20080250067A1 (en) * 2007-04-06 2008-10-09 Concert Technology Corporation System and method for selectively identifying media items for play based on a recommender playlist
US8839141B2 (en) 2007-06-01 2014-09-16 Napo Enterprises, Llc Method and system for visually indicating a replay status of media items on a media device
US20080307316A1 (en) * 2007-06-07 2008-12-11 Concert Technology Corporation System and method for assigning user preference settings to fields in a category, particularly a media category
US20090138457A1 (en) * 2007-11-26 2009-05-28 Concert Technology Corporation Grouping and weighting media categories with time periods
US8224856B2 (en) 2007-11-26 2012-07-17 Abo Enterprises, Llc Intelligent default weighting process for criteria utilized to score media content items
US20090164512A1 (en) * 2007-12-19 2009-06-25 Netta Aizenbud-Reshef Method and Computer Program Product for Managing Media Items
TWI358647B (en) * 2007-12-28 2012-02-21 Ind Tech Res Inst Data classification system and method for building
US7860866B2 (en) * 2008-03-26 2010-12-28 Microsoft Corporation Heuristic event clustering of media using metadata
US8676001B2 (en) 2008-05-12 2014-03-18 Google Inc. Automatic discovery of popular landmarks
JP4636141B2 (en) * 2008-08-28 2011-02-23 ソニー株式会社 Information processing apparatus and method, and program
US8396287B2 (en) 2009-05-15 2013-03-12 Google Inc. Landmarks from digital photo collections
JP5268787B2 (en) * 2009-06-04 2013-08-21 キヤノン株式会社 Information processing apparatus, control method therefor, and program
TWI494841B (en) * 2009-06-19 2015-08-01 Htc Corp Image data browsing methods and systems, and computer program products thereof
US20110125743A1 (en) * 2009-11-23 2011-05-26 Nokia Corporation Method and apparatus for providing a contextual model based upon user context data
US20110169982A1 (en) * 2010-01-13 2011-07-14 Canon Kabushiki Kaisha Image management apparatus, method of controlling the same, and storage medium storing program therefor
US9026591B2 (en) 2011-02-28 2015-05-05 Avaya Inc. System and method for advanced communication thread analysis
JP5708278B2 (en) * 2011-06-08 2015-04-30 ソニー株式会社 Information processing apparatus and information processing method
KR20130089400A (en) * 2012-02-02 2013-08-12 삼성전자주식회사 Method and apparatus for managing application in portable electronic device
US9690457B2 (en) 2012-08-24 2017-06-27 Empire Technology Development Llc Virtual reality applications
US9607436B2 (en) 2012-08-27 2017-03-28 Empire Technology Development Llc Generating augmented reality exemplars
US9892203B2 (en) * 2012-10-29 2018-02-13 Dropbox, Inc. Organizing network-stored content items into shared groups
US9135569B2 (en) * 2013-02-01 2015-09-15 Telenav, Inc. Navigation system with anomaly detection mechanism and method of operation thereof
US9542619B2 (en) * 2013-03-11 2017-01-10 Yahoo! Inc. Automatic image piling
US20140280241A1 (en) * 2013-03-15 2014-09-18 MediaGraph, LLC Methods and Systems to Organize Media Items According to Similarity
US20160357822A1 (en) * 2015-06-08 2016-12-08 Apple Inc. Using locations to define moments
US9863184B1 (en) * 2016-11-22 2018-01-09 Julian Voss-Andreae Window security screens
KR102158049B1 (en) 2018-11-05 2020-09-21 서강대학교산학협력단 Data clustering apparatus and method based on range query using cf tree

Citations (25)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH05128166A (en) 1991-11-01 1993-05-25 Konica Corp Electronic image filing device
US5598557A (en) * 1992-09-22 1997-01-28 Caere Corporation Apparatus and method for retrieving and grouping images representing text files based on the relevance of key words extracted from a selected file to the text files
US5828067A (en) * 1993-10-20 1998-10-27 Cambridge Imaging Limited Imaging method and apparatus
US6144375A (en) * 1998-08-14 2000-11-07 Praja Inc. Multi-perspective viewer for content-based interactivity
US6240424B1 (en) * 1998-04-22 2001-05-29 Nbc Usa, Inc. Method and system for similarity-based image classification
US6411724B1 (en) * 1999-07-02 2002-06-25 Koninklijke Philips Electronics N.V. Using meta-descriptors to represent multimedia information
US20020087538A1 (en) 1998-06-22 2002-07-04 U.S.Philips Corporation Image retrieval system
WO2002057959A2 (en) 2001-01-16 2002-07-25 Adobe Systems Incorporated Digital media management apparatus and methods
US6437797B1 (en) * 1997-02-18 2002-08-20 Fuji Photo Film Co., Ltd. Image reproducing method and image data managing method
US6446083B1 (en) * 2000-05-12 2002-09-03 Vastvideo, Inc. System and method for classifying media items
US6480840B2 (en) * 1998-06-29 2002-11-12 Eastman Kodak Company Method and computer program product for subjective image content similarity-based retrieval
US20020188602A1 (en) * 2001-05-07 2002-12-12 Eastman Kodak Company Method for associating semantic information with multiple images in an image database environment
US20030009469A1 (en) * 2001-03-09 2003-01-09 Microsoft Corporation Managing media objects in a database
US20030084065A1 (en) 2001-10-31 2003-05-01 Qian Lin Method and system for accessing a collection of images in a database
US6606411B1 (en) * 1998-09-30 2003-08-12 Eastman Kodak Company Method for automatically classifying images into events
JP2003242004A (en) 2002-02-14 2003-08-29 Fuji Photo Film Co Ltd Image display device and image management program
JP2003271617A (en) 2002-03-15 2003-09-26 Konica Corp Image classification method, image classification system, program and data structure of image file
US6629097B1 (en) * 1999-04-28 2003-09-30 Douglas K. Keith Displaying implicit associations among items in loosely-structured data sets
US20030195883A1 (en) * 2002-04-15 2003-10-16 International Business Machines Corporation System and method for measuring image similarity based on semantic meaning
US6650779B2 (en) * 1999-03-26 2003-11-18 Georgia Tech Research Corp. Method and apparatus for analyzing an image to detect and identify patterns
US6661842B1 (en) * 2000-09-22 2003-12-09 General Dynamics Decision Systems, Inc. Methods and apparatus for error-resilient video coding
US6766363B1 (en) * 2000-02-28 2004-07-20 Barpoint.Com, Inc. System and method of linking items in audio, visual, and printed media to related information stored on an electronic network using a mobile device
US6829561B2 (en) * 2002-03-16 2004-12-07 International Business Machines Corporation Method for determining a quality for a data clustering and data processing system
US6907436B2 (en) * 2000-10-27 2005-06-14 Arizona Board Of Regents, Acting For And On Behalf Of Arizona State University Method for classifying data using clustering and classification algorithm supervised
US6910049B2 (en) * 2001-06-15 2005-06-21 Sony Corporation System and process of managing media content

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5890149A (en) 1996-06-20 1999-03-30 Wisdomware, Inc. Organization training, coaching and indexing system
US6233575B1 (en) 1997-06-24 2001-05-15 International Business Machines Corporation Multilevel taxonomy based on features derived from training documents classification using fisher values as discrimination values
JP2002342357A (en) * 2001-05-18 2002-11-29 Kenichiro Ishijima Personal video database generator, image processor, personal database generating method, program and storage medium

Patent Citations (25)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH05128166A (en) 1991-11-01 1993-05-25 Konica Corp Electronic image filing device
US5598557A (en) * 1992-09-22 1997-01-28 Caere Corporation Apparatus and method for retrieving and grouping images representing text files based on the relevance of key words extracted from a selected file to the text files
US5828067A (en) * 1993-10-20 1998-10-27 Cambridge Imaging Limited Imaging method and apparatus
US6437797B1 (en) * 1997-02-18 2002-08-20 Fuji Photo Film Co., Ltd. Image reproducing method and image data managing method
US6240424B1 (en) * 1998-04-22 2001-05-29 Nbc Usa, Inc. Method and system for similarity-based image classification
US20020087538A1 (en) 1998-06-22 2002-07-04 U.S.Philips Corporation Image retrieval system
US6480840B2 (en) * 1998-06-29 2002-11-12 Eastman Kodak Company Method and computer program product for subjective image content similarity-based retrieval
US6144375A (en) * 1998-08-14 2000-11-07 Praja Inc. Multi-perspective viewer for content-based interactivity
US6606411B1 (en) * 1998-09-30 2003-08-12 Eastman Kodak Company Method for automatically classifying images into events
US6650779B2 (en) * 1999-03-26 2003-11-18 Georgia Tech Research Corp. Method and apparatus for analyzing an image to detect and identify patterns
US6629097B1 (en) * 1999-04-28 2003-09-30 Douglas K. Keith Displaying implicit associations among items in loosely-structured data sets
US6411724B1 (en) * 1999-07-02 2002-06-25 Koninklijke Philips Electronics N.V. Using meta-descriptors to represent multimedia information
US6766363B1 (en) * 2000-02-28 2004-07-20 Barpoint.Com, Inc. System and method of linking items in audio, visual, and printed media to related information stored on an electronic network using a mobile device
US6446083B1 (en) * 2000-05-12 2002-09-03 Vastvideo, Inc. System and method for classifying media items
US6661842B1 (en) * 2000-09-22 2003-12-09 General Dynamics Decision Systems, Inc. Methods and apparatus for error-resilient video coding
US6907436B2 (en) * 2000-10-27 2005-06-14 Arizona Board Of Regents, Acting For And On Behalf Of Arizona State University Method for classifying data using clustering and classification algorithm supervised
WO2002057959A2 (en) 2001-01-16 2002-07-25 Adobe Systems Incorporated Digital media management apparatus and methods
US20030009469A1 (en) * 2001-03-09 2003-01-09 Microsoft Corporation Managing media objects in a database
US20020188602A1 (en) * 2001-05-07 2002-12-12 Eastman Kodak Company Method for associating semantic information with multiple images in an image database environment
US6910049B2 (en) * 2001-06-15 2005-06-21 Sony Corporation System and process of managing media content
US20030084065A1 (en) 2001-10-31 2003-05-01 Qian Lin Method and system for accessing a collection of images in a database
JP2003242004A (en) 2002-02-14 2003-08-29 Fuji Photo Film Co Ltd Image display device and image management program
JP2003271617A (en) 2002-03-15 2003-09-26 Konica Corp Image classification method, image classification system, program and data structure of image file
US6829561B2 (en) * 2002-03-16 2004-12-07 International Business Machines Corporation Method for determining a quality for a data clustering and data processing system
US20030195883A1 (en) * 2002-04-15 2003-10-16 International Business Machines Corporation System and method for measuring image similarity based on semantic meaning

Non-Patent Citations (6)

* Cited by examiner, † Cited by third party
Title
English abstract for JP 2003242004, published Aug. 29, 2003.
English abstract for JP 2003271617, published Sep. 26, 2003.
English abstract for JP 5128166, published May 25, 1993.
Loui, et al., Automated Event Clustering and Quality Screening of Consumer Pictures for Digital Albuming, IEEE Transactions of Multimedia, vol. 5, No. 3, Sep. 2003.
Shen, et al., Personal Digital Historian: Story Sharing Around the Table, Interactions, Mar.+Apr. 2003.
Stent et al., Using Event Segmentation to Improve Indexing of Consumer Photographs, SIGIR'01, Sep. 2001, 59-65.

Cited By (17)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11592959B2 (en) 2010-01-06 2023-02-28 Apple Inc. Device, method, and graphical user interface for navigating and displaying content in context
US10732790B2 (en) 2010-01-06 2020-08-04 Apple Inc. Device, method, and graphical user interface for navigating and displaying content in context
US10296166B2 (en) 2010-01-06 2019-05-21 Apple Inc. Device, method, and graphical user interface for navigating and displaying content in context
US11099712B2 (en) 2010-01-06 2021-08-24 Apple Inc. Device, method, and graphical user interface for navigating and displaying content in context
US10572132B2 (en) 2015-06-05 2020-02-25 Apple Inc. Formatting content for a reduced-size user interface
US11681408B2 (en) 2016-06-12 2023-06-20 Apple Inc. User interfaces for retrieving contextually relevant media content
US10891013B2 (en) 2016-06-12 2021-01-12 Apple Inc. User interfaces for retrieving contextually relevant media content
US11334209B2 (en) 2016-06-12 2022-05-17 Apple Inc. User interfaces for retrieving contextually relevant media content
US10324973B2 (en) 2016-06-12 2019-06-18 Apple Inc. Knowledge graph metadata network based on notable moments
US11941223B2 (en) 2016-06-12 2024-03-26 Apple Inc. User interfaces for retrieving contextually relevant media content
US10362219B2 (en) 2016-09-23 2019-07-23 Apple Inc. Avatar creation and editing
US11243996B2 (en) 2018-05-07 2022-02-08 Apple Inc. Digital asset search user interface
US11086935B2 (en) 2018-05-07 2021-08-10 Apple Inc. Smart updates from historical database changes
US11782575B2 (en) 2018-05-07 2023-10-10 Apple Inc. User interfaces for sharing contextually relevant media content
US10846343B2 (en) 2018-09-11 2020-11-24 Apple Inc. Techniques for disambiguating clustered location identifiers
US10803135B2 (en) 2018-09-11 2020-10-13 Apple Inc. Techniques for disambiguating clustered occurrence identifiers
US11775590B2 (en) 2018-09-11 2023-10-03 Apple Inc. Techniques for disambiguating clustered location identifiers

Also Published As

Publication number Publication date
KR20060076311A (en) 2006-07-04
US7313574B2 (en) 2007-12-25
TW200515214A (en) 2005-05-01
JP2007507775A (en) 2007-03-29
TWI360058B (en) 2012-03-11
EP1668540A1 (en) 2006-06-14
JP2010191997A (en) 2010-09-02
KR100803769B1 (en) 2008-02-15
WO2005031601A1 (en) 2005-04-07
US20050076056A1 (en) 2005-04-07

Similar Documents

Publication Publication Date Title
USRE43260E1 (en) Method for clustering and querying media items
US6738759B1 (en) System and method for performing similarity searching using pointer optimization
US8515998B1 (en) Framework for managing document objects stored on a network
US20180046821A1 (en) Providing unique views of data based on changes or rules
US8600942B2 (en) Systems and methods for tables of contents
US5717925A (en) Information catalog system with object-dependent functionality
JP3648051B2 (en) Related information retrieval apparatus and program recording medium
US6408312B1 (en) Method and system for supporting multiple, historical, and future designs in a relational database
US20090210391A1 (en) Method and system for automated search for, and retrieval and distribution of, information
US20090164441A1 (en) Method and apparatus for searching using an active ontology
US20040158567A1 (en) Constraint driven schema association
US20030093417A1 (en) Method and apparatus for document information management
US7302643B1 (en) System and method for scheduled events to subscribe to live information topics
CN105653691A (en) Information resource management method and management apparatus
Sarwar et al. Ontology based image retrieval framework using qualitative semantic image descriptions
US20180307744A1 (en) Named entity-based category tagging of documents
US7861154B2 (en) Integration of annotations to dynamic data sets
US20040210468A1 (en) System and method for providing a territory management tool
WO2002027508A1 (en) Method and software for graphical representation of qualitative search results
US20090006323A1 (en) System and Method for Analyzing Intelligence Information
US20200226160A1 (en) Database for unstructured data
CN113761102A (en) Data processing method, device, server, system and storage medium
CN113127574A (en) Service data display method, system, equipment and medium based on knowledge graph
Liu et al. Structured image retrieval
Majeed et al. SIREA: Image retrieval using ontology of qualitative semantic image descriptions

Legal Events

Date Code Title Description
FEPP Fee payment procedure

Free format text: PAYOR NUMBER ASSIGNED (ORIGINAL EVENT CODE: ASPN); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

FPAY Fee payment

Year of fee payment: 8

FEPP Fee payment procedure

Free format text: MAINTENANCE FEE REMINDER MAILED (ORIGINAL EVENT CODE: REM.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

LAPS Lapse for failure to pay maintenance fees

Free format text: PATENT EXPIRED FOR FAILURE TO PAY MAINTENANCE FEES (ORIGINAL EVENT CODE: EXP.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY