WO2010071617A1

WO2010071617A1 - Method and apparatus for performing image processing

Info

Publication number: WO2010071617A1
Application number: PCT/US2008/013720
Authority: WO
Inventors: Ying Luo; Ju Guo; Dong-Qing Zhang; Rajan Joshi
Original assignee: Thomson Licensing
Priority date: 2008-12-15
Filing date: 2008-12-15
Publication date: 2010-06-24

Abstract

A method (300) for performing image processing utilizes mobile platforms to provide functions including processing, retrieving, annotating and/or mining of media information. According to an exemplary embodiment, the method (300) includes extracting first and second features of an image (320); searching a database according to one of the first and second extracted features and producing candidates in the database that match all or part of the one of the first and second extracted features (340); selecting one or more of the candidates according to the other of the first and second extracted features (350); and enabling display of the selected one or more candidates (350).

Description

METHOD AND APPARATUS FOR PERFORMING IMAGE PROCESSING

BACKGROUND OF THE INVENTION Field of the Invention The present invention generally relates to the field of image processing, and more particularly, to a method and apparatus for performing image processing that utilizes mobile platforms to provide functions including processing, retrieving, annotating and/or mining of media information.

Background Information

The wide availability of imaging devices on mobile platforms has taken computer vision applications into a new era. For example, one application that is facilitated by mobile imaging is the identification of different plants. In this application, assume a person hiking in the mountains, either as a hiker or botanist, has found some plants that he or she doesn't recognize. With current technology, the person can use a digital camera to take a picture of the plants, and later attempt to search the internet to identify the plants in question. This approach, however, tends to be inefficient and time-consuming for users, and certainly is not adequate to facilitate large scale applications in an efficient manner.

Accordingly, there is a need for a method and apparatus for performing image processing that is capable of addressing the aforementioned problems. The present invention described herein addresses these and/or other related issues.

SUMMARY OF THE INVENTION

In accordance with an aspect of the present invention, a method is disclosed. According to an exemplary embodiment, the method comprises extracting first and second features of an image; searching a database according to one of the first and second extracted features and producing candidates in the database that match all or part of the one of the first and second extracted features; selecting one or more of the candidates according to the other of the first and second extracted features; and enabling display of the selected one or more candidates.

In accordance with another aspect of the present invention, an apparatus is disclosed. According to an exemplary embodiment, the apparatus comprises a controller for receiving an image and extracting first and second features from the image, and a search element coupled to the controller for searching a database according to one of the first and second features. The controller instructs the search element to: search the database according to one of the first and second features, produce candidates from the database which match all or part of the first feature, and select one or more of the candidates according to the second feature.

BRIEF DESCRIPTION OF THE DRAWINGS The above-mentioned and other features and advantages of this invention, and the manner of attaining them, will become more apparent and the invention will be better understood by reference to the following description of embodiments of the invention taken in conjunction with the accompanying drawings, wherein: FIG. 1 is a diagram illustrating the framework of a general system according to an exemplary embodiment of the present invention;

FIG. 2 is a block diagram of a system according to an exemplary embodiment of the present invention; and

FIG. 3 is a flowchart illustrating steps according to an exemplary embodiment of the present invention.

The exemplifications set out herein illustrate preferred embodiments of the invention, and such exemplifications are not to be construed as limiting the scope of the invention in any manner.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Referring now to the drawings, and more particularly to FIG. 1 , the general framework of a system 100 according to an exemplary embodiment of the present invention is shown. System 100 of FIG. 1 comprises exemplary images 101-105, an image capturing device 110, a mobile processing block 120, the internet 130, a server 140 and central processing block 150, and mobile devices 160, 170, 180 and 190. For purposes of example and explanation, system 100 is shown and described herein as a system for processing images related to plants. However, it will be intuitive to those skilled in the art that the inventive principles of the present invention may also be applied to other types of systems that process different types of images.

In FIG. 1 , a user uses imaging capturing device 110 on a mobile platform to take pictures and/or videos of different aspects of plants (e.g., leaves, flowers, fruit, tree shape etc.), and thereby capture exemplary images 101-105. In FIG. 1 , this mobile platform comprises imaging capturing device 110, which may be embodied as a digital camera, and mobile processing block 120, which may be embodied as any type of suitable computing device. If mobile processing block 120 has sufficient processing power, some processing of the captured image data can be performed on the mobile platform directly. The user can also send the captured image data through wireless links and/or other physical links to a database system for processing and identification. In FIG. 1 , this database system includes the elements of internet 130, server 140 and central processing block 150, and/or mobile devices 160, 170, 180 and 190. Each of these different elements can perform their own further processing on the captured image data. Preferably a dedicated server, such as server 140, always exists to coordinate the necessary further processing of these elements. Based on the further processing of these elements, retrieval results of plant information are returned to the user. If the user's data is verified to be valid, it can be stored on server 140 at user's will to further enhance its contents.

Processing on Mobile Platform According to an exemplary embodiment, there are at least three (3) different forms of processing which may be performed on the mobile platform of FIG. 1 , which comprises imaging capturing device 110 and mobile processing block 120. These forms of processing include: 1. Automatic Processing. According to this form of processing, the mobile platform acts as a vision computing platform. Through a built-in vision processing module, object segmentation/detection/recognition, color/texture feature extraction are performed automatically by the mobile platform.

2. Interactive Processing. It is not always possible to perform fully automatic processing as described above. Therefore, interactive processing is made available. According to this form of processing, the user can input guidance to the vision processing module through a high resolution touch screen, a mouse like device, and custom designed graphical user interface (GUI). In object segmentation, it is typical to have the user input some rough contour of the object shape through the GUI and a computing module will find accurate shape of the objects. This scenario is particularly suitable for plant processing since leaf shapes, flower shapes and fruit shapes are among the most distinguishing features by which to identify differences in plants.

3. User Processing. According to this form of processing, the user can input text data to annotate the already known knowledge about a given plant, which can serve as auxiliary data for information retrieval.

Media and Feature Data

Also according to an exemplary embodiment, there are at least five (5) different types of data associated with this system 100, including:

1. Visual Data. This data includes pictures and/or video clips of a single plant's leaves, flowers, fruit, trunk, barks and roots. This visual data can also be taken from groups of plants, such as a certain bush.

2. Other Sensory Data. One example of this data is weather information including temperature and humidity information. Another example is chemical sensory data that indicates the scent of a plant's flowers. 3. Computed Feature Data. This type of data includes: a. Region Features - indicates the shapes of the leaves, flowers and overall shapes; b. Texture Features - indicates the textures of the leaves, flowers and trunk surfaces including bark; c. Object Features - includes segmented objects which can be directly used for searching; and d. Color Features - indicates the color of the leaves, flowers and other parts of the plants.

4. User Supplied Data. This type of data includes annotations (e.g., text) provided by the users.

5. Metadata. This type of data includes: a. The time of the image capture indicating the date and season; and b. The location of the image capture. This information may include more accurate information such as global positioning satellite (GPS) information, and/or less accurate or rough information such as a closest detectable location (e.g., a connecting tower location for cell phones, etc.).

Content Based Search and Retrieval

According to an exemplary embodiment, the visual data, other sensory data and computed feature data (i.e., items 1-3 above) are used to search and retrieve search results. Moreover, the user supplied data (e.g., annotations) and metadata (i.e., items 4-5 above) can be used to enhance the search results. For example, the fact that some plants are only available in certain locations will help eliminate many search results. Also, the fact that the color of some plant parts may be different at different seasons and/or weather conditions may further be used to eliminate additional non-relevant search results. Database System

As indicated above, the database system of FIG. 1 includes the elements of internet 130, server 140 and central processing block 150, and/or mobile devices 160, 170, 180 and 190. These elements may be used to provide additional image processing (in addition to that of the mobile platform) for purposes including image identification. According to an exemplary embodiment, there are several possible scenarios for implementing the database system of FIG. 1 which may include one or more of the following design considerations/objectives:

1. Leverage the capacity of internet 130 and search engines. Depending on the search engines' capacity, a search can be performed for the following types of information/data: • Text-based information such as user annotations and metadata;

• Feature-based information such as the feature data described above; and

• Raw data, such as the visual data and other sensory data described above, if the search engine processing is powerful enough.

2. Leverage the capacities of the other mobile devices, such as mobile devices 160, 170, 180 and 190. This scenario assumes that a massive connection of mobile devices is available through the form of a peer-to-peer network. The mobile devices with enough computing power can accept input data and perform a (content-based) search on their own data and return search results to the sender, or server 140.

3. Use a dedicated server, such as server 140. A dedicated server software platform, which can be centralized or distributed, preferably always exists, although it is not an absolute requirement. According to an exemplary embodiment, server 140 includes at least the following functionality: • Initial setup. With the corroboration of engineers, scientists and users, an initial plant information database can be set up to start the application;

• Central processing block 150 can allow some of the fully automatic vision processing to happen on the server side. The advantage to this is that it allows low end mobile devices to send data directly to server 140 to search and retrieve plant information.

• Association with internet 130 or other mobile devices 160, 170, 180 and 190. Collect the results from items 1 and 2, reorganize the results and send back to user. • Data validation and data cleaning. With many users sending the data into the system, it is expected that some of the data, such as annotation and segmentation data, will not be correct or complete. Therefore, the data needs to be validated against the data that already exists in database 140.

• Data storage. The input data can be stored at user's will as a new database item. With the massive contribution from different users, this functionality will certainly enhance the contents of database 140.

• Data management and data mining, which may include: o Further classification of plants. Many plant types can be subdivided into many different sub-types. This work is unlikely to be completely done at the initial setup of database 140. A Wikipedia-style mechanism can be used to manage the classification. o Database purge. This is performed to remove any existing erroneous data that the data validation and data cleaning procedure fails to detect, with input of more new data. o Access fee. Charge users a fee for accessing database 140. o Commercial association. Associate commercials with plant retrieval results and/or charge a fee for the use of such results for advertising purposes. o Academic research. For example, botanists may be interested in a plant that is found at a location where it is not known to exist. Referring now to FIG. 2, a block diagram of a system 200 according to an exemplary embodiment of the present invention is shown. System 200 of FIG. 2 represents a more practical and concrete example of how the present invention may be implemented. As shown in FIG. 2, system 200 comprises a mobile device 210, an online processing block 230, a feature matching engine 240 and a database 260. Also in FIG. 2, mobile device 210 comprises an image capture device 212, a search engine 214, a controller 216, a user input/output (IO) terminal 218, an image editor 220 and a feature extractor 222. Further in FIG. 2, feature matching engine 240 comprises a shape matcher 242, a color matcher 244, a texture matcher 246, a local matcher 248 and a decision logic block 250. According to exemplary embodiments, given functionalities of match engine 240 may also be embodied within mobile device 210. For purposes of example and explanation, system 200 is shown and described herein as a system for processing images related to plants. However, it will be intuitive to those skilled in the art that the inventive principles of the present invention may also be applied to other types of systems that process different types of images.

In mobile device 210, image capture device 212 is operative to capture images in response to user inputs via user I/O terminal 218. Search engine

214 is operative to perform searches under the control of controller 216 and in response to user inputs via user I/O terminal 218. Controller 216 is operative to perform various control functions of mobile device 210 as described herein, and includes one or more associated memories. User I/O terminal 218 is operative to receive user inputs and provide visual and/or aural outputs. User

I/O terminal 218 preferably includes at least a keypad (or other user input means) and an associated display device. Image editor 220 and feature extractor 222 perform various image processing functions under the control of controller 216, as will be described later herein. According to the exemplary embodiment shown in FIG. 2, object detection functionality is not included in mobile device 210 due to the variety of plant part appearances and the projected growing number of unknown plant types. However, such functionality could be included in mobile device 210 as a matter of design choice by providing sufficient processing power and memory capacity.

The main functionality of image editor 220 is to perform interactive object segmentation to yield the shape of a target plant part in a captured image. A commercial off-the-shelf image editor, or a customer designed program can perform these functions. Users may for example use a mouse pointer, a stylus, or a finger on a touch screen of user I/O terminal 218 to trace the rough boundary of a given image object. Interactive image segmentation algorithms such as Magic Wand, Intelligent Scissors and/or Grab Cut can then be used to provide a more accurate image object boundary. Because only the rough boundary input is required from the user, this functionality can be realized on mobile device screens with relatively low resolutions, such as QVGA 320x240 or the like.

The segmented object representing a plant part is fed into feature extractor 222. According to an exemplary embodiment, feature extractor 222 extracts at least four (4) different categories of object-based image features, including: shape descriptors, color descriptors, texture descriptors and local descriptors.

Shape descriptors are global descriptors that describe the overall shape properties of segmented plant parts. According to an exemplary embodiment, two (2) MPEG-7 shape descriptors may be employed. One is the region-based shape descriptor angular radial transformation (ART) and the other is the contour-based shape descriptor curvature scale-space (CSS) representation.

Color descriptors are global descriptors that describe the overall color properties of segmented plant parts. Since most plant parts are less likely to have a variety of colors, two color histogram-based color descriptors are employed according to an exemplary embodiment. They are two color descriptors defined by MPEG-7, namely, a scalable color descriptor (SCD) and a dominant color (DC).

According to an exemplary embodiment, the texture descriptor used is a Gabor wavelet-based MPEG-7 homogeneous texture descriptor (HTD). Also according to an exemplary embodiment, the local descriptor used is speeded up robust features (SURF), where Haar wavelet-based features are extracted from the regions centering the prominent feature points, i.e., the key points.

While all of the aforementioned features are extracted for every plant part, different plant parts have different prominent features. For example, while local descriptors are appropriate for all the plant parts, the shape descriptors are more prominent for the leaves, fruits, flowers and roots. Texture descriptors are more prominent for barks, groups of plant parts and flowers. Color descriptors can not be used alone many times since plant parts, such as leaves and fruits, will change color as the seasons change. However, with the availability of the image capture time (e.g., via user annotation), color descriptors can be used as the most prominent features to filter out data for the leaves and fruits. Note, however, that color descriptors may not be applicable to flowers since even the same plant flowers may have different colors at the same time. Also, shape descriptors may not be applicable to barks and groups of plant parts.

According to an exemplary embodiment, all of the descriptors are expressed in terms of numerical feature vectors that are stored into a feature template, which is a compact representation of a given image object. The size of the feature template is variable due to the nature of some descriptors, such as SIFT. According to the exemplary embodiment of FIG. 2, the feature template, together with other sensory features and user annotations when available, are then sent to feature matching engine 240 and database 260 for identification. For lower-end mobile devices without enough computing power, an entire image can be sent to online processing block 230 for online processing.

This also applies to higher-end mobile devices when the above-described local processing of mobile device 210 is not available. According to an exemplary embodiment, the online processing capabilities of online processing block 230 are triggered when a raw image, instead of a feature vector, is sent to feature matching engine 240 and database 260. In this case, a similar processing procedure as described above is applied, except that the user interface is now web-based and feature extraction is performed at online processing block 230.

According to an exemplary embodiment, feature matching engine 240 cascades its matchers 242-248 in different orders to form a telescoped matcher for each plant part. Submitted feature templates are compared with feature templates stored in database 260. Any appropriate matching algorithm can be used in matchers 242-248. Examples of how matchers 242- 248 can be used in different orders to form different telescoped matchers will now be described.

For plant leaves, fruits and roots, color matcher 244 with time information (i.e. provided as a user annotation) is applied first. If the matching score exceeds a certain threshold, shape matcher 242 is then applied. After the application of shape matcher 242, plant parts with matching scores exceeding a certain threshold are fed into texture matcher 246 and local matcher 248 to further reduce the number of matching candidates.

For plant flowers, shape matcher 242 is used first. Then, texture matcher 246 and local matcher 248 can be used to help exclude flowers with similar shapes, but different textures and prominent color patterns, as reflected on gray level distributions. For barks and groups of plant parts, texture matcher 246 and local matcher 248 play major roles, followed by color matcher 244 to exclude obvious different plants. Shape matcher 242 may not be used here.

Each of the above-described telescoped matchers will return a list of candidates to the user of mobile device 210, ranked according to the matching score. The above-described assembling of telescoped matchers is performed by decision logic block 250. According to an exemplary embodiment, decision logic block 250 also performs the following logic functions:

• If a text annotation (i.e., provided from a user of mobile device 210) that indicates the type of plant part is attached to a search, the feature matching engine 240 should arrange the order of matchers according to the indicated type, as described above. If the text annotation specifies a matcher, the feature template will only go through the specified matcher. Otherwise, the feature template will go through all the telescoped matchers. The matching scores are normalized and compared. A ranked candidate list is returned to the user of mobile device 210 based on the normalized matching scores. • If location information of a captured image (i.e., indicating the location of image capture) is available as an annotation, the candidate list is re-sorted according to the distances to the averaged center location and height.

• If location and time information of a captured image are available as an annotation, the location information together with the time information (i.e., indicating the time, day and/or month of image capture) will be sent to an internet weather web server (not shown in FIG. 2) to retrieve the weather conditions at the time and location of image capture, which includes temperature and humidity information.

• The plant names of the candidate list are searched against a commercial table in database 260. A partial match of a keyword phrase will result in the association of a product with the candidate, and a link to the product is returned to the user of mobile device 210 together with the plant information. This link, when executed, enables display of a name of a store selling goods represented by the selected candidate.

After receiving the search results, the user can select one or more candidates matching the submitted data and return the selected candidate(s) back to database 260 for storage and to enhance its contents.

According to an exemplary embodiment, database 260 is a relational database which stores information including plant information. According to this exemplary embodiment, there are at least three (3) different tables in database 260, namely a plant information table, a feature table and a commercial table. The plant information table schema has at least the following fields: plant identification; plant name; type; parent type; possible locations; description. The feature table schema has at least the following fields: feature identification; plant name; feature type; capturing time; capturing location; capturing weather condition; description; link to the feature template as a binary blob file residing in the file system. The commercial table schema has at least the following fields: product identification; plant name; product name; product type; product description; price; order information; vendor information.

Referring to FIG. 3, a flowchart 300 illustrating steps according to an exemplary embodiment of the present invention is shown. For purposes of example and explanation, the exemplary steps of FIG. 3 will be described with reference to the elements of FIG. 2. The steps of FIG. 3 are exemplary only, and are not intended to limit the scope of the present invention in any manner.

At step 310, an image is captured. According to an exemplary embodiment, a user of mobile device 210 causes the image to be captured at step 310 by providing an input via user I/O terminal 218 which causes image capture device 212 to capture the image. At step 320, features are extracted from the captured image. According to an exemplary embodiment, image editor 220 and feature extractor 222 operate under the control of controller 216 to thereby extract features from the captured image. As previously indicated herein, feature extractor 222 extracts at least four (4) different categories of object-based image features, including: shape descriptors, color descriptors, texture descriptors and local descriptors. All of the aforementioned descriptors are expressed in terms of numerical feature vectors that are stored into a feature template, which is a compact representation of the captured image.

At step 330, auxiliary data is received. According to an exemplary embodiment, the user of mobile device 210 inputs the auxiliary data to mobile device 210 via user I/O terminal 218. Such auxiliary data may include information such as the time and/or location at which the image was captured. Such auxiliary data becomes an annotation to the feature template of the captured image. It is noted that step 330 is optional based on the user's discretion.

At step 340, a database is searched. According to an exemplary embodiment, the feature template of the captured image together with any other available sensory features and user annotations (i.e., auxiliary data) are sent to feature matching engine 240 and database 260 for processing and identification. In particular, feature matching engine 240 processes the feature template and any auxiliary data, and database 260 is searched such that the submitted feature template is compared with feature templates stored in database 260.

As mentioned previously, the feature matching engine should arrange search order according to the type of plant part. At step 350, one or more candidates are provided for user selection.

According to an exemplary embodiment, the searching of database 260 performed at step 340 generates one or more candidates which are provided to the user of mobile device 210 for selection via a display device of user I/O terminal 218.

At step 360, auxiliary data may be added to the database. According to an exemplary embodiment, the user can select to provide any auxiliary data associated with the selected candidate(s) back to database 260 for storage and to enhance its contents.

As described herein, the present invention provides a method and apparatus for performing image processing that utilizes mobile platforms to provide functions including processing, retrieving, annotating and/or mining of media information. While this invention has been described as having a preferred design, the present invention can be further modified within the spirit and scope of this disclosure. This application is therefore intended to cover any variations, uses, or adaptations of the invention using its general principles. Further, this application is intended to cover such departures from the present disclosure as come within known or customary practice in the art to which this invention pertains and which fall within the limits of the appended claims.

Claims

PU080190CLAIMS

1. A method (300), comprising steps of: extracting first and second features of an image (320); searching a database according to one of the first and second extracted features and producing candidates in the database that match all or part of the one of the first and second extracted features (340); selecting one or more of the candidates according to the other of the first and second extracted features (350); and enabling display of the selected one or more candidates (350).

2. The method (300) of claim 1 , further comprising a step of receiving the image and auxiliary data from one or more sources.

3. The method (300) of claim 2, further comprising a step of selecting the one of the first and second extracted features according to the auxiliary data.

4. The method (300) of claim 2, wherein the auxiliary data includes location information where the image was captured, each of the selected one or more of the candidates associates with a location in the database, and the selected one or more candidates are arranged according to distances derived from the location information included in the auxiliary data and location information in the database.

5. The method (300) of claim 4, wherein the auxiliary data further includes a month and day the image was captured, each of the selected one or more candidates associates with a period and a location in the database, and each of the selected one or more candidates has an associated period including the month and date. PU080190

6. The method (300) of claim 2, further comprising steps of: enabling a user to select one of the selected one or more candidates; and receiving a selection from the user.

7. The method (300) of claim 6, further comprising a step of enabling addition of all or part of the auxiliary data to the database for one of the candidates selected by the user.

8. The method (300) of claim 6, further comprising a step of enabling display of a name of a store selling goods represented by one of the candidates selected by the user.

9. The method (300) of claim 2, wherein if the auxiliary data includes the first feature, the searching step searches the database according to the first feature and the selecting step is not performed.

10. An apparatus (210), comprising: a controller (216) for receiving an image and extracting first and second features from the image; a search element (214) coupled to the controller (216) for searching a database (260) according to one of the first and second features; and wherein the controller (216) instructs the search element (214) to: search the database (260) according to one of the first and second features, produce candidates from the database (260) which match all or part of the first feature, and select one or more of the candidates according to the second feature.

11. The apparatus (210) of claim 10, wherein the controller (216) receives the image and auxiliary data from one or more sources. PU080190

12. The apparatus (210) of claim 11 , wherein the one of the first and second extracted features is selected according to the auxiliary data.

13. The apparatus (210) of claim 11 , wherein the auxiliary data includes location information where the image was captured, each of the selected one or more of the candidates associates with a location in the database (260), and the selected one or more candidates are arranged according to distances derived from the location information included in the auxiliary data and location information in the database (260).

14. The apparatus (210) of claim 13, wherein the auxiliary data further includes a month and day the image was captured, each of the selected one or more candidates associates with a period and a location in the database (260), and each of the selected one or more candidates has an associated period including the month and date.

15. The apparatus (210) of claim 11 , wherein the controller (216) enables a user to select one of the selected one or more candidates, and receives a selection from the user.

16. The apparatus (210) of claim 15, wherein the controller (216) enables addition of all or part of the auxiliary data to the database (260) for one of the candidates selected by the user.

17. The apparatus (210) of claim 15, wherein the controller (216) further enables display of a name of a store selling goods represented by one of the candidates selected by the user. PU080190

18. The apparatus (210) of claim 11 , wherein if the auxiliary data includes the first feature, the search element (214) searches the database (260) according to the first feature and does not perform the selection.

19. An apparatus (210), comprising: means (216) for receiving an image and extracting first and second features from the image; means (214) for searching a database (260) according to one of the first and second features; and wherein the receiving and extracting means (216) instructs the searching means (214) to: search the database (260) according to one of the first and second features, produce candidates from the database (260) which match all or part of the first feature, and select one or more of the candidates according to the second feature.

20. The apparatus (210) of claim 19, wherein the receiving and extracting means (216) receives the image and auxiliary data from one or more sources.

21. The apparatus (210) of claim 20, wherein the one of the first and second extracted features is selected according to the auxiliary data.

22. The apparatus (210) of claim 20, wherein the auxiliary data includes location information where the image was captured, each of the selected one or more of the candidates associates with a location in the database (260), and the selected one or more candidates are arranged according to distances derived from the location information included in the auxiliary data and location information in the database (260). PU080190

23. The apparatus (210) of claim 20, wherein the auxiliary data further includes a month and day the image was captured, each of the selected one or more candidates associates with a period and a location in the database (260), and each of the selected one or more candidates has an associated period including the month and date.

24. The apparatus (210) of claim 20, wherein the receiving and extracting means (216) enables a user to select one of the selected one or more candidates, and receives a selection from the user.

25. The apparatus (210) of claim 24, wherein the receiving and extracting means (216) enables addition of all or part of the auxiliary data to the database (260) for one of the candidates selected by the user.

26. The apparatus (210) of claim 25, wherein the receiving and extracting means (216) further enables display of a name of a store selling goods represented by one of the candidates selected by the user.

27. The apparatus (210) of claim 20, wherein if the auxiliary data includes the first feature, the searching means (214) searches the database (260) according to the first feature and does not perform the selection.