CN102902821A - Methods for labeling and searching advanced semantics of imagse based on network hot topics and device - Google Patents

Abstract

The invention discloses methods for labeling and searching advanced semantics of images based on network hot topics and a device. The method for labeling the advanced semantics of the images comprises the step of: searching images similar to the entity semantics of images to be labeled and accompanying texts by using an entity semantic work of the images to be labeled based on a search engine of text key words, then extracting topics from the accompanying texts, establishing association relationship among topics, among images, and between images and topics, based on the association relationship, classifying the images with similar topics and visual features into a type, and classifying similar topics corresponding to the images with the similar visual features into a type; and selecting image types which are the most similar to the images to be labeled in the visual features of the images, and taking the corresponding topics as hot topics. According to the invention, by means of the process, the advanced semantic labeling the images is realized, and in addition, the obtained advanced semantics can be used to accurately describe the images to be labeled through denoising.

Description

The senior semantic tagger of the image of much-talked-about topic Network Based, search method and device

Technical field

The present invention relates to image labeling and searching field, be specifically related to the senior semantic tagger of image, search method and the device of much-talked-about topic Network Based.

Background technology

Image is a kind of multi-medium data of complexity, has comprised abundant semantic content.The semanteme of image is divided into three levels, and ground floor is the bottom semantic layer, the bottom visual signatures such as the color of namely utilizing original image data to extract to obtain, texture; The second layer is the Entity Semantics layer, namely utilizes the bottom visual signature that extracts, and carries out certain reasoning from logic, identifies the object type that comprises in the image, centers on the Entity Semantics of the object extraction of image.The 3rd layer is that the abstract semantics layer is senior semanteme, has comprised the senior semantemes such as scene, behavior and emotion, is the more senior reasoning to Entity Semantics.

Along with the development of digitized video technology and Internet technology, the user can obtain a large amount of images easily.Retrieve for the convenience of the user the image that meets demand from a large amount of images, the image labeling technology is arisen at the historic moment.Image labeling refers to add the technology that can describe its semantic keyword for image.The user is that searching key word just can retrieve associated picture from network by text retrieval like this.Along with the development of technology, image labeling develops into automatic image annotation namely by seeking the incidence relation between semantic and the bottom visual signature by artificial mark, with this opening relationships model, realizes the mark to the unknown semantics image.

At present, the automatic image annotation technology mainly refers to the mark of and Entity Semantics semantic to the image bottom, and based on this, the user also can't retrieve image by the mode of inputting senior semantic content.But along with the development of internet, the user often need to retrieve the image relevant with senior semantic content.Such as, the user often wants to retrieve the image relevant with network hot topic.Herein, network hot topic refers to sometime in the section, (burst) event that occurs on the network or the topic of discussion widely.The clicking rate that generally is presented as webpage sharply rises or the inquiry of image, upload, download increases.

Therefore, be badly in need of at present a kind of method that the senior semanteme of image is marked, especially to the mask method of image-related network hot topic.

Summary of the invention

In view of this, the object of the present invention is to provide a kind of senior semantic tagger of image, search method and device thereof of much-talked-about topic Network Based, for realizing that the user retrieves the condition that provides by senior semanteme to image.

The embodiment of the invention provides a kind of image of much-talked-about topic Network Based senior semanteme marking method, and described method comprises:

The senior semanteme marking method of a kind of image of much-talked-about topic Network Based is characterized in that, described method comprises:

Take at least one Entity Semantics word of image to be marked as query word, utilize the search engine based on text key word, the semantic similar image of retrieval and described image to be marked and described semantic similar image follows text from network;

Extract the described theme of following in the text, and based on the described corresponding relation of following text and the corresponding relation of described theme to set up the similar image of described semanteme and described theme;

Visual signature is the similar and similar image of described semanteme that have a similar theme is polymerized to a class, forms the set of image class; Similar similar motif polymerization corresponding to image of the described semanteme that visual signature is similar is a class, forms the theme class set;

Set up the corresponding relation of described image class set and the set of described theme class;

According to the visual signature of described image to be marked, from the set of described image class, search the image class similar to the visual signature of described image to be marked, and extract theme class corresponding to described similar image class as the network hot topic of described image to be marked;

According to described network hot topic described image to be marked is carried out semantic tagger.

Preferably, described method also comprises the step of in advance described image to be marked being carried out the Entity Semantics mark, specifically comprises:

Extract the visual signature of described image to be marked;

According to described visual signature, from limited training set, search the candidate image with described image similarity to be marked;

Extract the Entity Semantics word of described candidate image, and utilize described Entity Semantics word that described image to be marked is carried out the Entity Semantics mark.

Preferably, after the Entity Semantics word of the described candidate image of described extraction, utilize described Entity Semantics word that described image to be marked is carried out before the Entity Semantics mark, described method also comprises:

Poly-according to the candidate image that Entity Semantics is similar of described Entity Semantics word is a class, forms the set of candidate image class;

From described candidate image class set, search the candidate image class the most similar to described Image Visual Feature to be marked as the neighbor image class;

Describedly utilize described Entity Semantics word that described image to be marked is carried out Entity Semantics mark to comprise:

Utilize the Entity Semantics word of described neighbor image class that described image to be marked is carried out the Entity Semantics mark.

Preferably, described poly-according to the candidate image that Entity Semantics is similar of described Entity Semantics word be a class, form the candidate image class and comprise

Set up hypergraph model G (Vs, Ts), and obtain the similarity matrix H of hypergraph model based on this, wherein, described hypergraph model take with the set Vs of the candidate image of described image similarity to be marked as vertex set, take the set Ts of the Entity Semantics word of described candidate image as super limit collection; Element Hij in the described matrix H represents the symbiosis of contacting of each image Vi and corresponding Entity Semantics word Tj and each Entity Semantics word and a plurality of candidate images;

According to described similarity matrix H, utilize spectral clustering, described hypergraph model is carried out cluster, it is a class that the candidate image of sharing the super limit of some is gathered, and forms described candidate image class.

Preferably, described method also comprises:

Utilize formula

Calculate Entity Semantics word in the described neighbor image class and the degree of correlation of described image to be marked; Wherein,

Ii is the neighbor image among the neighbor image class S, and iq is image to be marked; P (ii/iq) equals the visual signature similarity of ii and described iq;

The described Entity Semantics word that utilizes described neighbor image carries out the Entity Semantics mark to described image to be marked and comprises:

According to described degree of correlation order from big to small, the Entity Semantics word of choosing predetermined number from described neighbor image class carries out the Entity Semantics mark to described image to be marked.

Preferably, the described theme of following in the text of described extraction, and based on the described corresponding relation of following text and theme, the corresponding relation of setting up the similar image of described semanteme and described theme comprises:

Utilize the described text of following to set up the LDA model, based on the described theme of described LDA model extraction and set up image-Topic relative matrix Rvt;

Described visual signature is the similar and similar image of described semanteme that have a similar theme is polymerized to a class, forms the set of image class; Similar similar motif polymerization corresponding to image of the described semanteme that visual signature is similar is a class, forms the theme class set and comprises:

Set up the described Topic relative matrix Rt that follows text;

Utilize the visual similarity of image, calculate the visual similarity matrix Rv of the similar image of described semanteme;

Utilize Rt, Rvt, Rv, set up complicated graph model G (Rv, Rt, Rvt);

Described complicated figure G (Rv, Rt, Rvt) is carried out cluster, form described image class set and the set of described theme class.

Preferably, describedly according to described network hot topic described image to be marked is carried out semantic tagger and comprises:

Utilize evolution method of inspection χ ²Front K the word that extraction and the described network hot topic degree of correlation are the highest carries out semantic tagger to described image to be marked.

The present invention also provides a kind of image of much-talked-about topic Network Based senior semantic tagger device, and described device comprises:

The text retrieval unit, be used at least one Entity Semantics word take image to be marked as query word, utilization is based on the search engine of text key word, and the semantic similar image of retrieval and described image to be marked and described semantic similar image follows text from network;

The theme extraction unit is used for extracting the described theme of following text;

The first associative cell is used for based on the described corresponding relation of following text and theme, sets up the corresponding relation of the similar image of described semanteme and described theme;

Cluster cell is used for the similar image of described semanteme visual signature is similar and that have a similar theme and is polymerized to a class, forms the set of image class; Similar similar motif polymerization corresponding to image of the described semanteme that visual signature is similar is a class, forms the theme class set;

The second associative cell is used for setting up the corresponding relation of described image class set and the set of described theme class;

The first content retrieval unit is used for the visual signature according to described image to be marked, searches the image class similar to the visual signature of described image to be marked from described image class set;

The much-talked-about topic extraction unit is used for extracting theme class corresponding to described similar image class as the network hot topic of described image to be marked;

Much-talked-about topic mark unit is used for according to described network hot topic described image to be marked being carried out semantic tagger.

Preferably, described device also comprises Entity Semantics mark unit, is used for described image to be marked is carried out the Entity Semantics mark; Described Entity Semantics mark unit specifically comprises:

The Visual Feature Retrieval Process unit is for the visual signature that extracts described image to be marked;

The second content retrieval unit is used for according to described visual signature, searches the candidate image with described image similarity to be marked from limited training set;

Entity Semantics word extraction unit is used for extracting the Entity Semantics word of described candidate image;

Entity Semantics mark subelement is used for utilizing described Entity Semantics word that described image to be marked is carried out the Entity Semantics mark.

Preferably, described device also comprises the denoising unit, is used for denoising is carried out in described candidate image; Concrete, described denoising unit comprises:

The candidate image cluster cell, being used for poly-according to the candidate image that Entity Semantics is similar of described Entity Semantics word is a class, forms the set of candidate image class;

The 3rd content retrieval unit is used for searching the candidate image class the most similar to described Image Visual Feature to be marked as the neighbor image class from described candidate image class set;

Described Entity Semantics mark subelement, concrete Entity Semantics word for utilizing described neighbor image class carries out the Entity Semantics mark to described image to be marked.

Preferably, described candidate image cluster cell comprises:

The hypergraph model unit, be used for setting up hypergraph model G (Vs, Ts), and obtain the similarity matrix H of hypergraph model based on this, wherein, described hypergraph model take with the set Vs of the candidate image of described image similarity to be marked as vertex set, take the set Ts of the Entity Semantics word of described candidate image as super limit collection; Element Hij in the described matrix H represents the symbiosis of contacting of each image Vi and corresponding Entity Semantics word Tj and each Entity Semantics word and a plurality of candidate images;

The spectral clustering unit is used for according to described similarity matrix H, utilizes spectral clustering, and described hypergraph model is carried out cluster, and it is a class that the candidate image of sharing the super limit of some is gathered, and forms described candidate image class.

Preferably, described device also comprises:

Correlation calculating unit is used for utilizing formula

Calculate Entity Semantics word in the described neighbor image class and the degree of correlation of described image to be marked; Wherein,

Ii is the neighbor image among the neighbor image class S, and iq is image to be marked; P (ii/iq) equals the visual signature similarity of ii and described iq;

Described Entity Semantics mark subelement, concrete being used for according to described degree of correlation order from big to small, the Entity Semantics word of choosing predetermined number from described neighbor image class carries out the Entity Semantics mark to described image to be marked.

Preferably, described theme extraction unit, concrete being used for utilizes the described text of following to set up the LDA model, based on the described theme of described LDA model extraction;

Described the first associative cell, concrete being used for set up image-Topic relative matrix Rvt based on described LDA model;

Described cluster cell comprises:

The Topic relative matrix unit is used for setting up the described Topic relative matrix Rt that follows text;

The visual similarity matrix unit for the visual similarity that utilizes image, is calculated the visual similarity matrix Rv of the similar image of described semanteme;

Complicated graph model unit is used for utilizing Rt, Rvt, Rv, sets up complicated graph model G (Rv, Rt, Rvt);

Complicated figure cluster cell is used for described complicated figure G (Rv, Rt, Rvt) is carried out cluster, forms described image class set and the set of described theme class.

Preferably, described much-talked-about topic mark unit specifically is used for, and utilizes evolution method of inspection χ ²Front K the word that extraction and the described network hot topic degree of correlation are the highest carries out semantic tagger to described image to be marked.

The present invention also provides a kind of image search method of much-talked-about topic Network Based, and wherein, described network hot topic utilizes above-mentioned image labeling method to obtain, and this search method comprises:

Receive the image retrieval text of user's input; Described image retrieval text comprises a network hot topic word at least;

Image on the crawl internet and the markup information of described image thereof;

Whether the markup information of judging described image is complementary with described image retrieval text;

Such as coupling, then with the described image that is complementary and markup information output thereof.

Preferably, when comprising simultaneously network hot topic and Entity Semantics in the described image retrieval text, whether the described markup information of judging described image is complementary with described image retrieval text and comprises:

Judge in the markup information of described image and whether comprise described network hot topic word.

Preferably, described as coupling then comprises the described image that is complementary and markup information output thereof:

Such as coupling, then will export with the content of described image retrieval text-dependent in the described image that is complementary and the markup information thereof.

The present invention also provides a kind of image retrieving apparatus of much-talked-about topic Network Based, and wherein, described network hot topic utilizes above-mentioned image labeling method to obtain, and described device comprises:

The text receiving element is used for receiving the image retrieval text that the user inputs; Described image retrieval text comprises a network hot topic word at least;

Placement unit is for the image on the crawl internet and the markup information of described image thereof;

Judging unit is used for judging whether the markup information of described image is complementary with described image retrieval text;

Output unit is used for exporting the described image that is complementary and markup information thereof and exporting when the markup information of described image is complementary with described image retrieval text.

Preferably, described judging unit, judges in the markup information of described image whether comprise described network hot topic word at concrete being used for when described image retrieval text comprises network hot topic and Entity Semantics simultaneously.

Preferably, described output unit, will be exported with the content of described image retrieval text-dependent in the described image that is complementary and the markup information thereof at concrete being used for when the markup information of described image is complementary with described image retrieval text.

Compare with prior art, the present invention has following beneficial effect:

The invention provides a kind of mask method of the senior semanteme of image of much-talked-about topic Network Based, at first utilize the Entity Semantics word of image to be marked, based on the search engine of text key word, from the mass data of internet, retrieved the image similar to image entities semanteme to be marked and followed text.Because training set is based on the internet, therefore, obtain semantic comprehensively and have a real-time update.Then the present invention extracts theme from follow text, and set up incidence relation between theme and theme, image and image, image and the theme, and based on this, to have the similar image of similar theme and visual signature poly-is a class, and will have similar theme corresponding to the image of similar visual signature poly-is a class.Therefrom select the image class the most similar to Image Visual Feature to be marked, the theme that it is corresponding is as much-talked-about topic.By said process, carried out denoising with following in the text with the incoherent theme of correspondence image and with the not high image of Image Visual Feature similarity to be marked, so that image to be marked can be described accurately in the senior semanteme that obtains.

Description of drawings

Fig. 1 is the embodiment of the invention 1 text based image search method process flow diagram;

Fig. 2 A-2C is that image to be marked reaches the semantic similar image that arrives based on text retrieval in the embodiment of the invention;

Fig. 3 is the denoising process flow diagram of semantic similar image in the embodiment of the invention;

Fig. 4 is the senior semanteme marking method process flow diagram of much-talked-about topic Network Based in the embodiment of the invention 2;

Fig. 5 is the process flow diagram that in advance image to be marked is carried out the Entity Semantics mark in the embodiment of the invention 3;

Fig. 6 is hypergraph model schematic diagram in the embodiment of the invention;

Fig. 7 is that the embodiment of the invention 5 is based on the senior semantic tagger structure drawing of device of the image of much-talked-about topic;

Fig. 8 is the image search method process flow diagram of the embodiment of the invention 6 much-talked-about topics Network Based;

Fig. 9 is the image retrieving apparatus structural drawing of the embodiment of the invention 7 much-talked-about topics Network Based.

Embodiment

In order to make those skilled in the art person understand better the scheme of the embodiment of the invention, below in conjunction with drawings and embodiments the embodiment of the invention is described in further detail.

For realizing the mark to the image network much-talked-about topic, must determine network hot topic corresponding to image.We know, on network, most of image all interts in text message.We claim these text messages that occur with image for following text, and are common such as Word message in the web page text etc.Image with follow text to have to a great extent relevance, therefore, we can think tentatively that these follow the much-talked-about topic that embodies in the text is exactly much-talked-about topic corresponding to image.

We know, the semanteme of piece image can obtain from its similar image.Based on this, the present invention need to retrieve the text of following with the image of image similarity to be marked and these similar images, then follows from these and extracts network hot topic text, and this network hot topic is exactly much-talked-about topic corresponding to image to be marked.

The text of following of retrieval similar image and similar image can carry out in several ways.The embodiment of the invention 1 provides a kind of image retrieval based on text key word (TBIR, Text Based ImageRetrieval) method, and referring to Fig. 1, the method comprises:

S11, take at least one Entity Semantics word of image to be marked as query word, based on the search engine of text key word, the semantic similar image of retrieval and described image to be marked and described semantic similar image follows text from network.

At least one Entity Semantics word among the present invention can be that the user manually inputs according to image to be marked, also can be that system is according to the automatically extraction of Entity Semantics word of the image labeling to be marked of input.In actual applications, image to be marked may have a plurality of Entity Semantics words, and for farthest searching for the image similar to image, semantic to be marked, user or system can only select one of them or part Entity Semantics word to retrieve.But the image that retrieves like this may comprise a large amount of images not high with image similarity to be marked.The image and the Image similarity to be marked that retrieve for raising in a preferred embodiment of the invention, can be retrieved take all Entity Semantics words of image to be marked as query word.

In the present invention, searching system can be Baidu or google, can certainly be other searching system.

Based on the text of following that retrieves, can follow the theme in the text to form much-talked-about topic by extraction.But this much-talked-about topic is correspondence follows text, if based on image and the incidence relation of following text, much-talked-about topic and image direct correlation are got up, can introduce a lot of noises, is mainly reflected in two aspects:

At first, the text of following of image is normally taken from the web page text at image place, and image is the part of web page text, and therefore, a part of theme of web page text is semantic incoherent with image.

Such as telling about in the web page text of handset configuration one, be interspersed with the image of loudspeaker in the mobile phone, but in web page text, having greatly, content has nothing to do with loudspeaker all in the structure of telling about other parts of mobile phone.

Secondly, all texts of following all are to utilize identical query word search to obtain, because the vision polysemy of word, these follow the corresponding image of text and image to be marked visually may have very big-difference.

Such as, suppose that Fig. 2 A is image to be marked, its Entity Semantics word is " apple ", if but take " apple " as query word, can search the image graph 2B similar to Fig. 2, also can search the image with Fig. 2 A diverse " i Phone ".

For these reasons, when determining network hot topic corresponding to image to be marked, need to utilize the visual similarity of image that semantic similar image is carried out denoising, pick out to the most similar image set of image to be marked and follow text set, thereby improve the correctness of the much-talked-about topic of choosing.Its concrete denoising process comprises as shown in Figure 3:

The theme in the text is followed in S21, extraction, sets up the corresponding relation of following between text and its theme, and sets up the corresponding relation of image and theme based on the corresponding relation of following text and theme.

S22, set up the symbiosis between theme and the theme, set up the visual similarity relation between image and the image.

S23, image that visual signature is similar and corresponding similar motif polymerization thereof are a class, form the theme class set, and the image that will have similar theme and have a similar visual signature is polymerized to a class, forms the set of image class.And set up corresponding relation between above-mentioned image class and the theme class based on said process.

S24, search in the set of above-mentioned image class and image to be marked similar image class on visual signature, similar theme class corresponding to image class of described visual signature is the network hot topic of image to be marked.

In the present invention, can in the set of above-mentioned image class, search some to image to be marked similar image class on visual signature.In a preferred embodiment of the invention, in the set of above-mentioned image class, search and image to be marked the most similar image class on visual signature.

Can find out by said process, the present invention need to consider three kinds of incidence relations when denoising: the symbiosis between theme and the theme, the similarity relation between image and the image, the corresponding relation between image and theme.Because image follows text to have certain semantic association with it, therefore, the corresponding relation of image and theme can be similar to following the corresponding relation of text and theme to obtain.

In the present invention, above-mentioned various incidence relation can obtain based on various ways.

Such as, follow corresponding relation and the symbiosis between theme and the theme between text and the theme in the leaching process of theme, to set up.Follow the extraction of theme in the text also to have various ways, such as utilizing existing topic model vector space model, latent semantic analysis model (LSA) etc., preferred LDA (the potential Di Li Cray of Latent Dirichlet Allocation apportion model) carries out the theme extraction among the present invention.

LD A model is a kind of theme probability model that carries out modeling for the discrete type text, a text-theme-three layers of Bayesian model of word, be the probability mixed distribution of some themes with text representation, have the more text semantic descriptive power of approaching to reality data, can process efficiently large-scale corpus.The LDA model is by the theme modeling, with the characteristic vector space of text by the dimension of word change the dimension of theme into, the relative words of synonym and nearly justice are mapped to same subject, realize the modeling of semantic level.

This model has two parameters to need to estimate: one is " text-theme " distribution probability, and another is several " theme-word " distribution probabilities.By these two parameters, we can know the interested theme of text author, and the theme ratio that contains of each text etc.The incidence relation that needs to estimate " text-theme " among the present invention i.e. " text-theme " distribution probability.Existing method for parameter estimation mainly contains variation-EM (expectation maximization, expectation maximization) algorithm, also has the Gibbs sampling.The preferred Gibbs sampling of the present invention is carried out parameter estimation.Utilize GibbsLDA, can obtain the incidence relation of text and theme Wherein, α is the parameter that the corresponding Dirichlet of theme distributes, and T is the number of different themes, n _j ^DjThe number of times that theme j occurs among the text dj, n ^DjIt is the total degree that all themes occur in text dj.

About the symbiosis of theme and theme, can represent by the matrix Rt that sets up symbiosis between the reflection theme, wherein, Topic (Zi, Zj) is the entry of a matrix element, is defined as follows:

$\mathrm{Topic}(\mathrm{zi},\mathrm{zj})=p\left(\mathrm{zi}\right|\mathrm{zj})=\frac{C(\mathrm{zi}\∩\mathrm{zj})}{C(\mathrm{zi}\∩\mathrm{zj})+C(\stackrel{-}{z}i\∩\stackrel{-}{z}j)}*\frac{C(\mathrm{zi}\∩\mathrm{zj})}{C\left(\mathrm{zj}\right)}+\frac{C(\stackrel{-}{z}i\∩\stackrel{-}{z}j)}{C(\mathrm{zi}\∩\mathrm{zj})+C(\stackrel{-}{z}i\∩\stackrel{-}{z}j)}*\frac{C(\stackrel{-}{z}i\∩\stackrel{-}{z}j)}{C\left(\stackrel{-}{z}j\right)}$

Wherein, C (zi ∩ zj) and

Represent respectively theme zi and theme the zj common frequency that occurs and absent number of times of while in following text set, represented the incidence relation between the theme.

Visual similarity relation about image and image can draw by the similarity between the computed image.In conjunction with prior art, the vision similarity of image can obtain according to each Characteristic of Image vector calculation.The concrete similarity incidence matrix H that can set up image and image _VEntry of a matrix element Sim _v(Ii, Ij) is defined as follows:

${\mathrm{Sim}}_v(\mathrm{Ii},\mathrm{Ij})=({\mathrm{\Σ}}_{d=1}^n\min (\mathrm{id},\mathrm{jd})/\max (\mathrm{id},\mathrm{jd}))/n$

Wherein, the n dimensional feature vector that image I i extracts is expressed as Ii[i1, i2, i3...in], the n dimensional feature vector that image I j extracts is expressed as Ij[j1, j2, j3...jn], id and jd represent respectively the number of times that d kind feature occurs in the image of correspondence.

Every width of cloth image follows text that certain theme semantic association is arranged with it, so image obtains with the related incidence relation of following text and theme that can be similar to image of theme.Concrete, but the corresponding relation probability of use p (zj|Ii) of image and theme expression.The probability of image I i mark keyword Zj is as follows:

Wherein, p (zj|Ii) ≈ p (zj|dj) is obtained by the LDA training; P (Ij|Ii) ≈ Sim _V(Ii, Ij)

Based on above-mentioned three kinds of incidence relations, can in several ways image and theme be carried out cluster among the present invention.Because have image and two kinds of isomery summits of theme in above-mentioned three kinds of incidence relations, preferred complicated graph model carries out cluster in the embodiment of the invention.

The complicated figure of definition (Complex graph) G={Rv, Rt, Rvt}, wherein, Rv, Rt represent respectively theme vertex set and image vertex set, limit set Rvt comprises two subsets, is designated as

Wherein, S is the limit weight matrix that isomorphism connects among the Rv, and A represents the limit weight matrix that the isomery between Rv, the Rt connects.N1, N2 represent respectively the number of vertices among Rv, the Rt.

Based on above-mentioned complicated figure definition, complicated figure is carried out cluster can realize difference cluster to theme vertex set and image vertex set, and, utilize the isomorphism between the summit to be connected incidence relation with isomery, can set up two corresponding relations between the vertex set classification, i.e. the corresponding relation of the i class among the image vertex set Rt and the j class among the theme vertex set Rv.

Based on above-mentioned analysis, the embodiment of the invention 2 provides a kind of senior semanteme marking method of much-talked-about topic Network Based, and referring to Fig. 4, idiographic flow is as follows:

S31, input initial query word utilize the search engine of text, obtain semantic similar image set and follow text collection.

S32, utilize the visual similarity of image, the visual similarity matrix Rv between the computing semantic similar image.

S33, utilization follow text collection to set up the LDA model, Topic relative matrix Rt and image-Topic relative matrix Rvt.

S34, input Rv, Rt, Rvt sets up complicated graph model G (Rv, Rt, Rvt).

S35, complexity figure G (Rv, Rt, Rvt) is carried out cluster, finds out the expansion neighbor image set similar to the query image vision, extract its corresponding much-talked-about topic.

Based on prior art such as Wu Fei, Han Yahong, the complicated figure clustering algorithm that proposes in " the Web image clustering method of image-text relevant mining " that Zhuan Yueting, Shao Jian write, the cluster process of complicated figure G can change into the process of the optimum solution of finding the solution following optimization problem.The optimization problem of complicated figure cluster is defined as the optimum solution of finding the solution objective function L:

$\left(\begin{array}{c}\arg {\min }_{C^{\left(1\right)},C^{{\left(2\right)}^L}}\\ L={\min }_{C^{\left(1\right)},C^{\left(2\right)}{\left|\right|S-C^{\left(T\right)}D{\left(C^{\left(2\right)}\right)}^T\left|\right|}^2+{\left|\right|A-C^{\left(1\right)}B{\left(C^{\left(2\right)}\right)}^T\left|\right|}^2}\\ S,t,C^{\left(1\right)}\∈{\left(\mathrm{0,1}\right)}^{N_1*K_1},C^{\left(2\right)}\∈{\left(\mathrm{0,1}\right)}^{N_2*K_2}\end{array}\right)$ (formula 1)

Wherein, Matrix C ⁽¹⁾The Clustering of node that is to say in the expression Rv

Link strength (degree of association) in the expression Rv between i class and the j class.Matrix C ⁽²⁾The Clustering of node that is to say in the expression Rt

Link strength (degree of association) in the expression Rt between i class and the j class.Matrix B represents in the Rv Clustering of node in the node and Rt, that is to say i class in B (i, j) the expression Rv and the strength of association of the j class in the Rt, and showing as is a kind of probability.Matrix D represents the Clustering of Rv inner vertex, and the strength of joint in matrix element D (i, j) the expression Rv between i class and the j class shows as a kind of probability.

The below provides the solution formula of matrix D, B.If D, B are the optimum solutions of formula (1), then have:

$\left(\begin{array}{c}D={\left({\left(C^{\left(1\right)}\right)}^T{C}^{\left(1\right)}\right)}^{-1}{\left(C^{\left(1\right)}\right)}^T{\mathrm{SC}}^{\left(1\right)}{\left({\left(C^{\left(1\right)}\right)}^T{C}^{\left(1\right)}\right)}^{-1}\\ B={\left({\left(C^{\left(1\right)}\right)}^T{C}^{\left(1\right)}\right)}^{-1}{\left(C^{\left(1\right)}\right)}^T{\mathrm{AC}}^{\left(2\right)}{\left({\left(C^{\left(2\right)}\right)}^T{C}^{\left(2\right)}\right)}^{-1}\\ S,t,C^{\left(1\right)}\∈{\left(\mathrm{0,1}\right)}^{N_1*K_1},C^{\left(2\right)}\∈{\left(\mathrm{0,1}\right)}^{N_2*K_2},D\∈R_{+}^{K1*K2},B\∈R_{+}^{K1*K2}\end{array}\right)$ Formula (2)

Based on above-mentioned analysis, the present invention utilizes the method for complicated figure cluster as follows:

Input: complicated figure G=(Rv, Rt, Rvt).Set the cluster number K1 of set Rv, the cluster number K2 of set Rt;

Output: Subject Clustering is C as a result ⁽¹⁾Image clustering is C as a result ⁽²⁾Matrix Pk1*k2, matrix element represent in the theme vertex set incidence relation of each classification after the cluster in each classification after the cluster and image collection.

Concrete calculation process is:

1, given C ⁽¹⁾, C ⁽²⁾Initial value, according to formula (1) (2), calculate successively the initial value of D, B, L, make L _Min=L _Init

2, fixedly D, B and C ⁽²⁾, upgrade line by line C ⁽¹⁾In 1 position so that each time more new capital guarantee that the L that this time renewal obtains is minimum, upgrade L _Min

3, fixedly D, B and C ⁽¹⁾, upgrade line by line C ⁽²⁾In 1 position so that each time more new capital guarantee that the L that this time renewal obtains is minimum, upgrade L _Min

4, calculate successively D, B according to formula (2);

5, constantly repeat 2-4 until convergence;

6, according to formula P (I|T)=P (I|T ') * P (T ' | T) try to achieve the correlation degree matrix of image class and theme class.The strength of joint matrix of P (I|T ') representative image and theme wherein, i.e. B matrix, P (T ' | T) represent the strength of joint matrix of theme and theme, i.e. the D matrix.According to the corresponding D of final L that the 5th step tried to achieve, the B matrix gets final product to get incidence matrix P (I|T).

Utilize complicated figure clustering algorithm respectively image vertex set and theme vertex set to be carried out cluster, and set up one by one mapping relations of classification in the theme vertex set and the class in the image vertex set.In cluster process, three kinds of incidence relations interact, and for the cluster of image, the similar and image total similar theme of vision content aggregates into a class; For the cluster of theme, the related subject of similar image aggregates into a class, has formed much-talked-about topic.For example, when using " prick the gram Buick " image during as the initial query image, obtain pricking the initial query word of gram Buick, through the TBIR retrieval, obtained multiple image and a plurality of text of following that the gram Buick occurred pricking, after complicated figure modeling and cluster, formed and the set of the similar image of query image, simultaneously corresponding formation " facebook, founder; social networks film wait a plurality of relevant with " prick and restrain Buick " mark words.

The much-talked-about topic that extracts based on said method is the image class corresponding theme the most similar to image to be marked, for further improving the accuracy of much-talked-about topic, in a preferred embodiment of the invention, can use evolution method of inspection χ ²Extract and the mark word of the highest front K the word of the topic degree of correlation as this topic, so that image to be marked is marked.Wherein, K greater than 0 less than or equal to the word number in the network hot topic.

X2 is write as CHI sometimes, and its formula is: $\mathrm{CHI}(t,c_i)=\frac{N*{(\mathrm{AD}-\mathrm{BC})}^2}{(A+C)*(B+D)*(A+B)*(C+D)}$

Wherein, t represents a certain word in the word list, and ci is a certain much-talked-about topic, and CHI (t, ci) represents the degree of association of t and ci.N represents all number of documents, the A representative comprises word t and belongs to the number of documents of theme ci, the B representative comprises the number of documents that word t does not still belong to theme ci, the C representative does not comprise the number of documents that word t still belongs to theme ci, and D represents the number of documents that does not comprise word t nor belong to theme ci.

Need to prove, might comprise the descriptor of image entities semanteme in the much-talked-about topic that obtains based on said process.As seen, based on said process, utilize the text of following of similar image on the internet can also expand simultaneously Entity Semantics mark to image to be marked.And the mode of relatively existing limited training set, the present invention is take the internet mass data as training set, so that mark comprehensively semantic and have real-time update.

In embodiments of the invention 3, before obtaining the Entity Semantics word of image to be marked, method of the present invention also comprises the process of in advance image to be marked being carried out the Entity Semantics mark.As shown in Figure 5, this process comprises the steps:

The visual signature of S41, the described image to be marked of extraction.

S42, according to described visual signature, utilize CBIR (CBIR, Content BasedImage Retrieval), from limited training set, search the candidate image with described image similarity to be marked.

The Entity Semantics word of S43, the described candidate image of extraction, and utilize described Entity Semantics word that described image to be marked is carried out the Entity Semantics mark.

Need to prove because the problem of semantic gap, although above-mentioned candidate image visually with image similarity to be marked, semantically may have very big difference.Such as, the image of an apple of input.Based on the retrieval of CBIR, may all export as candidate image being similar to the spherical object of apple or people's head etc. so.And the Entity Semantics word that obtains based on these candidate images is not the true semanteme of image to be marked obviously such as " ball ", " head " etc.Therefore, in a preferred embodiment of the invention, need to carry out denoising to candidate image, to choose the candidate image more approaching with image, semantic to be marked.

We know, the image that candidate image is concentrated has marked the Entity Semantics word that a plurality of quantity do not wait, the text of following of correspondence image can be regarded as in these Entity Semantics words, and following between image corresponding to text and its all is that semantic content is relevant to a certain extent.Therefore, in order to investigate the otherness of the semantic content between the similar candidate image of these vision contents, the semantic content that can be considered as every width of cloth image is similar to and is equal to the semantic content that it follows text, thereby follow semantic difference between the text by investigating these, just can indirectly obtain the semantic content otherness between the candidate image.Follow between the text at these, semantic content is similar follows text, and the quantity of the similar Entity Semantics word that it is owned together is just more.

Based on this, the embodiment of the invention 4 provides a kind of denoising method of candidate image, specifically comprises:

According to the similarity between the Entity Semantics word, image corresponding to text of following that semanteme is similar carries out cluster, forms the set of candidate image class.

Then choose to the Entity Semantics word of image to be marked the most similar image class on visual signature image to be marked is marked.

In the present invention, utilize hypergraph to hint obliquely at the Entity Semantics word and the incidence relation of following text of image.

Hypergraph is a kind of graph model, and its limit is referred to as super limit, can connect two or more summits.That is to say, every limit is exactly a vertex subset.In the present invention, the summit in the hypergraph has just represented image, and super limit is exactly Entity Semantics word (word).Obviously, an image may have a plurality of super frontier junctures joining.

Such as, when image A, B, C, D, E are arranged, if the Entity Semantics word of following text of image A is " apple ", " fruit " and " price ", the Entity Semantics word of following text of image B is " apple ", " fruit ".The Entity Semantics word of following text of image C is " apple ", " fruit " and " illumination ", the Entity Semantics word of following text of image D is " Qiao Busi " and " price ", the Entity Semantics word of following text of image E is " football ", it is image A, B, the text of following of C is shared Entity Semantics word " apple ", " fruit ", image A, the text of following of D is shared Entity Semantics word " price ", image E follow text not with any image follow text to share the Entity Semantics word time, represent above-mentioned relation with hypergraph, it specifically as shown in Figure 6, image A wherein, B, C shares two super limits, image A, D shares a super limit.Image E is an isolated summit.

Concentrate in candidate image, noise figure picture and query image are discrepant at semantic content usually, and noise image also is semantic discrepant each other, and semanteme also is similar between the candidate image similar to the query image semantic content.From the angle of cluster, the isolated point that above-mentioned situation then shows as a plurality of dispersions is often corresponding a plurality of noise images have then formed the larger class of quantity to query image semantic similar image.Therefore, can carry out cluster analysis at hypergraph, thereby the image that candidate image is concentrated separates division according to text similarity.By cluster, realized that a plurality of super limits (Entity Semantics word) are farthest shared on the summit (image) between similar, and the inhomogeneity difference is very large.

Through above-mentioned hypergraph modeling and cluster process, just search being obtained image has gathered into some classes, utilize the visual similarity of image, just can find out the image class the most similar to query image, just can obtain initial Entity Semantics thereby in this image class, extract the Entity Semantics word.

Each image class is to there being a plurality of Entity Semantics words, and each Entity Semantics word is different from the degree of correlation of image to be marked.Be further to improve the accuracy of image entities semantic tagger to be marked, in the preferred embodiment of the present invention, at first calculate the degree of correlation of each Entity Semantics word and image to be marked in the above-mentioned the most similar image class.Then choosing the highest Entity Semantics word of a front K degree of correlation marks image to be marked.

Wherein, the degree of correlation of each Entity Semantics word and image to be marked can be utilized formula in the most similar image class

Calculate.Wherein, S is the most similar image class, ti is the Entity Semantics word in the most similar image class, ii is the image in the most similar image class, and iq is image to be marked, p (ii/iq) ≈ similary (ii, iq), the visual signature similarity of similary (ii, iq) expression ii and iq, this similarity can utilize method of the prior art to calculate.No longer elaborate herein. Its expression is when comprising ti among the ii, and value is 1, otherwise is 0.

To sum up, the invention provides a kind of mask method of the senior semanteme of image based on much-talked-about topic, at first utilize the Entity Semantics word of image to be marked, the text based image retrieval has retrieved the image similar to image entities semanteme to be marked and has followed text from the mass data of internet.Because training set is based on the internet, therefore, obtain semantic comprehensively and have a real-time update.Then the present invention extracts theme from follow text, and set up incidence relation between theme and theme, image and image, image and the theme, and based on this, to have the similar image of similar theme and visual signature poly-is a class, and will have similar theme corresponding to the image of similar visual signature poly-is a class.Therefrom select the image the most similar to Image Visual Feature to be marked, the theme that it is corresponding is as much-talked-about topic.By said process, carried out denoising with following in the text with the incoherent theme of correspondence image and with the not high image of Image Visual Feature similarity to be marked, so that image to be marked can be described accurately in the senior semanteme that obtains.

Corresponding said method, the embodiment of the invention 5 also provides a kind of image based on much-talked-about topic senior semantic tagger device, and referring to Fig. 7, this device comprises:

Receiving element 11 is at least one Entity Semantics word of the image to be marked that receives user's input.

Text retrieval unit 12 is used for take described at least one Entity Semantics word as query word, the text based picture search, and the semantic similar image of retrieval and described image to be marked and described semantic similar image follows text from network.

In actual applications, image to be marked may have a plurality of Entity Semantics words, and for farthest searching for the image similar to image, semantic to be marked, the user can only select one of them Entity Semantics word input to retrieve.But the image that retrieves like this may comprise a large amount of images not high with image similarity to be marked.The image and the Image similarity to be marked that retrieve for raising in a preferred embodiment of the invention, can be retrieved take all Entity Semantics words of image to be marked as query word.

In the present invention, the device of retrieving can be Baidu or google, can certainly be other indexing unit.

Theme extraction unit 13 is used for extracting the described theme of following text.

According to before embodiment of the method analysis partly as can be known, when determining network hot topic corresponding to image to be marked, need to utilize the visual similarity of image to carry out denoising, pick out to the most similar image set of image to be marked and follow text set, thereby improve the correctness of the much-talked-about topic of choosing.Based on this, described device also comprises:

The first associative cell 14 is used for based on the described corresponding relation of following text and theme, sets up the corresponding relation of the similar image of described semanteme and described theme.

Cluster cell 15 is used for the similar image of described semanteme visual signature is similar and that have a similar theme and is polymerized to a class, forms the set of image class; Similar similar motif polymerization corresponding to image of the described semanteme that visual signature is similar is a class, forms the theme class set.

The second associative cell 16 is used for setting up the corresponding relation of described image class set and the set of described theme class.

First content retrieval unit 17 is used for the visual signature according to described image to be marked, searches the image class similar to the visual signature of described image to be marked from described image class set.

Finished the semantic similar image that obtains from network and the denoising of following text thereof by said process.Then, extract on this basis network hot topic:

Much-talked-about topic extraction unit 18 is used for extracting theme class corresponding to described similar image class as the network hot topic of described image to be marked.

Much-talked-about topic mark unit 19 is used for according to described network hot topic described image to be marked being carried out semantic tagger.

Among the present invention, the cluster of theme extraction and image and theme can realize by multiple device.Such as the extraction of theme can utilize existing device based on the topic model vector space model, based on device of latent semantic analysis model (LSA) etc.

The embodiment of the invention provides a kind of device that extracts network hot topic and cluster:

Device based on LDA (the potential Di Li Cray of Latent Dirichlet Allocation apportion model) carries out the theme extraction, and namely the theme extraction unit specifically is used for utilizing the described text of following to set up the LDA model, based on the described theme of described LDA model extraction.

Described the first associative cell, concrete being used for set up image-Topic relative matrix Rvt based on described LDA model.

Described cluster cell comprises:

The Topic relative matrix unit is used for setting up the described Topic relative matrix Rt that follows text.

The visual similarity matrix unit for the visual similarity that utilizes image, is calculated the visual similarity matrix Rv of the similar image of described semanteme.

Complicated graph model unit is used for utilizing Rt, Rvt, Rv, sets up complicated graph model G (Rv, Rt, Rvt).

Complicated figure cluster cell is used for described complicated figure G (Rv, Rt, Rvt) is carried out cluster, forms described image class set and the set of described theme class.

The much-talked-about topic that extracts based on said process is the image class corresponding theme the most similar to image to be marked, for further improving the accuracy of much-talked-about topic, in a preferred embodiment of the invention, described much-talked-about topic mark unit, specifically be used for, utilize evolution method of inspection χ ²Method is extracted with the highest front K the word of the network hot topic degree of correlation and as marking word image to be marked is marked, K greater than 0 less than or equal to the word number in the network hot topic.Wherein, can be referring to the embodiment of the method part about the concrete formula of the evolution method of inspection.

Need to prove, might comprise the descriptor of image entities semanteme in the much-talked-about topic that obtains based on said process.As seen, based on said process, utilize the text of following of similar image on the internet can also expand simultaneously Entity Semantics mark to image to be marked.And the mode of relatively existing limited training set, the present invention is take the internet mass data as training set, so that mark comprehensively semantic and have real-time update.

The Entity Semantics word of image to be marked is by the realization of Entity Semantics mark, and in specific embodiments of the invention, described device also comprises the Entity Semantics mark unit that in advance image to be marked is carried out the Entity Semantics mark.Be used for described image to be marked is carried out the Entity Semantics mark; Described Entity Semantics mark unit specifically comprises:

The Visual Feature Retrieval Process unit is for the visual signature that extracts described image to be marked;

The second content retrieval unit is used for according to described visual signature, searches the candidate image with described image similarity to be marked from limited training set;

Entity Semantics word extraction unit is used for extracting the Entity Semantics word of described candidate image;

Entity Semantics mark subelement is used for utilizing described Entity Semantics word that described image to be marked is carried out the Entity Semantics mark.

Need to prove because the problem of semantic gap, although above-mentioned candidate image visually with image similarity to be marked, semantically may have very big difference.Such as, the image of an apple of input.Based on the retrieval of CBIR, may all export as candidate image being similar to the spherical object of apple or people's head etc. so.And the mark word that obtains based on these candidate images is not the true semanteme of image to be marked obviously such as " ball ", " head " etc.Therefore, in a preferred embodiment of the invention, need to carry out denoising to candidate image, to choose the candidate image more approaching with image, semantic to be marked.

We know, the image that candidate image is concentrated has marked the mark word that a plurality of quantity do not wait, and these mark the text of following that correspondence image can be regarded as in words, and following between image corresponding to text and its all is that semantic content is relevant to a certain extent.Therefore, in order to investigate the otherness of the semantic content between the similar candidate image of these vision contents, the semantic content that can be considered as every width of cloth image is similar to and is equal to the semantic content that it follows text, thereby follow semantic difference between the text by investigating these, just can indirectly obtain the semantic content otherness between the candidate image.Follow between the text at these, semantic content is similar follows text, and the quantity of the similar mark word that it is owned together is just more.

Based on this, device of the present invention also comprises the denoising unit, is used for denoising is carried out in described candidate image; Concrete, described denoising unit comprises:

The candidate image cluster cell, being used for poly-according to the candidate image that Entity Semantics is similar of described Entity Semantics word is a class, forms the set of candidate image class;

The 3rd content retrieval unit is used for searching the candidate image class the most similar to described Image Visual Feature to be marked as neighbor image from described candidate image class set;

Institute's Entity Semantics mark subelement, concrete Entity Semantics word for utilizing described neighbor picture image set carries out the Entity Semantics mark to described image to be marked.

The candidate image cluster can be by multiple device realization, and it is wherein a kind of that the embodiment of the invention provides, and described candidate image cluster cell comprises:

The hypergraph model unit, be used for setting up hypergraph model G (Vs, Ts), and obtain the similarity matrix H of hypergraph model based on this, wherein, described hypergraph model take with the set Vs of the candidate image of described image similarity to be marked as the summit, take the set Ts of the Entity Semantics word of described candidate image as super limit; Element Hij in the described matrix H represents the symbiosis of contacting of each image Vi and corresponding Entity Semantics word Tj and each Entity Semantics word and a plurality of candidate images;

The spectral clustering unit is used for according to described similarity matrix H, utilizes spectral clustering, and described hypergraph model is carried out cluster, and it is a class that the candidate image of sharing the super limit of some is gathered, and forms described candidate image class.

Each image class is to there being a plurality of Entity Semantics words, and each Entity Semantics word is different from the degree of correlation of image to be marked.For further improving the accuracy of image entities semantic tagger to be marked, in the preferred embodiment of the present invention, described device also comprises:

Correlation calculating unit is used for utilizing formula Calculate Entity Semantics word in the described neighbor image and the degree of correlation of described image to be marked; Wherein, ii is the neighbor image among the neighbor image class S, and iq is image to be marked;

P (ii/iq) equals the visual signature similarity of ii and described iq; P (ii/ip) is the visual signature similarity of image ii and described image to be marked.

Described Entity Semantics mark subelement, concrete front K Entity Semantics word for choosing degree of correlation maximum carries out the Entity Semantics mark to described image to be marked.

Corresponding above-mentioned mask method and device, the embodiment of the invention 6 also provides a kind of image search method of much-talked-about topic Network Based, and wherein, the network hot topic of image is based on that method mark in above-described embodiment obtains.Referring to Fig. 8, described search method comprises:

The image retrieval text of S51, reception user input; Described image retrieval text comprises a network hot topic word at least.

In retrieval, the user is the input text relevant with network hot topic only, also can input simultaneously Entity Semantics and the text relevant with network hot topic.The present invention does not limit this.

Image on S52, the crawl internet and the markup information of described image thereof.

Can utilize existing method for capturing network information among the present invention, as utilize spider traverses network resource, obtain image and markup information thereof on the internet.

S53, judge whether the markup information of described image is complementary with described image retrieval text.

Concrete, can judge whether comprised all images retrieval text of inputting in the markup information.Perhaps, also can judge the image retrieval text word that whether has comprised in the markup information greater than predetermined threshold value quantity.

Such as, when user input " Beijing ", " apple ", " price overgrowing " several word, when the system that can arrange comprises 2 identical terms in retrieving markup information, think that namely image and text retrieval information are complementary.

In a specific embodiment of the present invention, when text retrieval information comprises network hot topic word and Entity Semantics word simultaneously, can be arranged on when comprising the network hot topic word in the markup information that retrieves, think that image and text retrieval information are complementary.

S54, such as coupling, then will the described image that is complementary and markup information export.

Most of users only pay close attention to the content relevant with the text retrieval information of oneself input, therefore, in a preferred embodiment of the invention, can only export the image and the markup information relevant with text retrieval information that is complementary.

Corresponding above-mentioned search method, the embodiment of the invention 7 also provides a kind of image retrieving apparatus of much-talked-about topic Network Based, and wherein said network hot topic utilizes above-mentioned image labeling method to obtain.Referring to Fig. 9, described device comprises:

Text receiving element 21 is used for receiving the image retrieval text that the user inputs; Described image retrieval text comprises a network hot topic word at least.

In retrieval, the user is the input text relevant with network hot topic only, also can input simultaneously Entity Semantics and the text relevant with network hot topic.The present invention does not limit this.

Placement unit 22 is for the image on the crawl internet and the markup information of described image thereof.

Can utilize existing method for capturing network information among the present invention, as utilize spider traverses network resource, obtain image and markup information thereof on the internet.

Judging unit 23 is used for judging whether the markup information of described image is complementary with described image retrieval text.

In a specific embodiment of the present invention, when text retrieval information comprised network hot topic word and Entity Semantics word simultaneously, described judging unit can be used for judging in the markup information of described image whether comprise described network hot topic word.

Output unit 24 is used for exporting the described image that is complementary and markup information thereof and exporting when the markup information of described image is complementary with described image retrieval text.

Most of users only pay close attention to the content relevant with the text retrieval information of oneself input, therefore, in a preferred embodiment of the invention, described output unit, concrete being used for when the markup information of described image is complementary with described image retrieval text, will export with the content of described image retrieval text-dependent in the described image that is complementary and the markup information thereof.

Need to prove, said apparatus embodiment is corresponding with embodiment of the method, therefore the device part is no longer described in detail, and relevant portion gets final product referring to embodiment of the method.

Above the embodiment of the invention is described in detail, has used embodiment herein the present invention is set forth, the explanation of above embodiment just is used for helping to understand method and apparatus of the present invention; Simultaneously, for one of ordinary skill in the art, according to thought of the present invention, all will change in specific embodiments and applications, in sum, this description should not be construed as limitation of the present invention.

Claims (20)

1. the senior semanteme marking method of the image of a much-talked-about topic Network Based is characterized in that, described method comprises:

Take at least one Entity Semantics word of image to be marked as query word, utilize the search engine based on text key word, the semantic similar image of retrieval and described image to be marked and described semantic similar image follows text from network;

Extract the described theme of following in the text, and based on the described corresponding relation of following text and the corresponding relation of described theme to set up the similar image of described semanteme and described theme;

Visual signature is the similar and similar image of described semanteme that have a similar theme is polymerized to a class, forms the set of image class; Similar similar motif polymerization corresponding to image of the described semanteme that visual signature is similar is a class, forms the theme class set;

Set up the corresponding relation of described image class set and the set of described theme class;

According to the visual signature of described image to be marked, from the set of described image class, search the image class similar to the visual signature of described image to be marked, and extract theme class corresponding to described similar image class as the network hot topic of described image to be marked;

According to described network hot topic described image to be marked is carried out semantic tagger.

2. method according to claim 1 is characterized in that, described method also comprises the step of in advance described image to be marked being carried out the Entity Semantics mark, specifically comprises:

Extract the visual signature of described image to be marked;

According to described visual signature, from limited training set, search the candidate image with described image similarity to be marked;

Extract the Entity Semantics word of described candidate image, and utilize described Entity Semantics word that described image to be marked is carried out the Entity Semantics mark.

3. method according to claim 2 is characterized in that, after the Entity Semantics word of the described candidate image of described extraction, utilizes described Entity Semantics word that described image to be marked is carried out before the Entity Semantics mark, and described method also comprises:

Poly-according to the candidate image that Entity Semantics is similar of described Entity Semantics word is a class, forms the set of candidate image class;

From described candidate image class set, search the candidate image class the most similar to described Image Visual Feature to be marked as the neighbor image class;

Describedly utilize described Entity Semantics word that described image to be marked is carried out Entity Semantics mark to comprise:

Utilize the Entity Semantics word of described neighbor image class that described image to be marked is carried out the Entity Semantics mark.

4. method according to claim 3 is characterized in that, described poly-according to the candidate image that Entity Semantics is similar of described Entity Semantics word be a class, form the candidate image class and comprise

Set up hypergraph model G (Vs, Ts), and obtain the similarity matrix H of hypergraph model based on this, wherein, described hypergraph model take with the set Vs of the candidate image of described image similarity to be marked as vertex set, take the set Ts of the Entity Semantics word of described candidate image as super limit collection; Element Hij in the described matrix H represents the symbiosis of contacting of each image Vi and corresponding Entity Semantics word Ti and each Entity Semantics word and a plurality of candidate images;

According to described similarity matrix H, utilize spectral clustering, described hypergraph model is carried out cluster, it is a class that the candidate image of sharing the super limit of some is gathered, and forms described candidate image class.

5. according to claim 3 or 4 described methods, it is characterized in that, described method also comprises:

Utilize formula

Calculate Entity Semantics word in the described neighbor image class and the degree of correlation of described image to be marked; Wherein,

Ii is the neighbor image among the neighbor image class S, and iq is image to be marked; P (ii/iq) equals the visual signature similarity of ii and described iq;

The described Entity Semantics word that utilizes described neighbor image carries out the Entity Semantics mark to described image to be marked and comprises:

According to described degree of correlation order from big to small, the Entity Semantics word of choosing predetermined number from described neighbor image class carries out the Entity Semantics mark to described image to be marked.

6. method according to claim 1 is characterized in that, the described theme of following in the text of described extraction, and based on the described corresponding relation of following text and theme, the corresponding relation of setting up the similar image of described semanteme and described theme comprises:

Utilize the described text of following to set up the LDA model, based on the described theme of described LDA model extraction and set up image-Topic relative matrix Rvt;

Described visual signature is the similar and similar image of described semanteme that have a similar theme is polymerized to a class, forms the set of image class; Similar similar motif polymerization corresponding to image of the described semanteme that visual signature is similar is a class, forms the theme class set and comprises:

Set up the described Topic relative matrix Rt that follows text;

Utilize the visual similarity of image, calculate the visual similarity matrix Rv of the similar image of described semanteme;

Utilize Rt, Rvt, Rv, set up complicated graph model G (Rv, Rt, Rvt);

Described complicated figure G (Rv, Rt, Rvt) is carried out cluster, form described image class set and the set of described theme class.

7. each described method is characterized in that according to claim 1-6, describedly according to described network hot topic described image to be marked is carried out semantic tagger and comprises:

Utilize front K the word that evolution method of inspection χ 2 extracts and the described network hot topic degree of correlation is the highest that described image to be marked is carried out semantic tagger.

8. the senior semantic tagger device of the image of a much-talked-about topic Network Based is characterized in that, described device comprises:

The text retrieval unit, be used at least one Entity Semantics word take image to be marked as query word, utilization is based on the search engine of text key word, and the semantic similar image of retrieval and described image to be marked and described semantic similar image follows text from network;

The theme extraction unit is used for extracting the described theme of following text;

The first associative cell is used for based on the described corresponding relation of following text and theme, sets up the corresponding relation of the similar image of described semanteme and described theme;

Cluster cell is used for the similar image of described semanteme visual signature is similar and that have a similar theme and is polymerized to a class, forms the set of image class; Similar similar motif polymerization corresponding to image of the described semanteme that visual signature is similar is a class, forms the theme class set;

The second associative cell is used for setting up the corresponding relation of described image class set and the set of described theme class;

The first content retrieval unit is used for the visual signature according to described image to be marked, searches the image class similar to the visual signature of described image to be marked from described image class set;

The much-talked-about topic extraction unit is used for extracting theme class corresponding to described similar image class as the network hot topic of described image to be marked;

Much-talked-about topic mark unit is used for according to described network hot topic described image to be marked being carried out semantic tagger.

9. device according to claim 8 is characterized in that, described device also comprises Entity Semantics mark unit, is used for described image to be marked is carried out the Entity Semantics mark; Described Entity Semantics mark unit specifically comprises:

The Visual Feature Retrieval Process unit is for the visual signature that extracts described image to be marked;

The second content retrieval unit is used for according to described visual signature, searches the candidate image with described image similarity to be marked from limited training set;

Entity Semantics word extraction unit is used for extracting the Entity Semantics word of described candidate image;

Entity Semantics mark subelement is used for utilizing described Entity Semantics word that described image to be marked is carried out the Entity Semantics mark.

10. device according to claim 9 is characterized in that, described device also comprises the denoising unit, is used for denoising is carried out in described candidate image; Concrete, described denoising unit comprises:

The candidate image cluster cell, being used for poly-according to the candidate image that Entity Semantics is similar of described Entity Semantics word is a class, forms the set of candidate image class;

The 3rd content retrieval unit is used for searching the candidate image class the most similar to described Image Visual Feature to be marked as the neighbor image class from described candidate image class set;

Described Entity Semantics mark subelement, concrete Entity Semantics word for utilizing described neighbor image class carries out the Entity Semantics mark to described image to be marked.

11. device according to claim 10 is characterized in that, described candidate image cluster cell comprises:

The hypergraph model unit, be used for setting up hypergraph model G (Vs, Ts), and obtain the similarity matrix H of hypergraph model based on this, wherein, described hypergraph model take with the set Vs of the candidate image of described image similarity to be marked as vertex set, take the set Ts of the Entity Semantics word of described candidate image as super limit collection; Element Hij in the described matrix H represents the symbiosis of contacting of each image Vi and corresponding Entity Semantics word Tj and each Entity Semantics word and a plurality of candidate images;

The spectral clustering unit is used for according to described similarity matrix H, utilizes spectral clustering, and described hypergraph model is carried out cluster, and it is a class that the candidate image of sharing the super limit of some is gathered, and forms described candidate image class.

12. according to claim 10 or 11 described devices, it is characterized in that, described device also comprises:

Correlation calculating unit is used for utilizing formula Calculate Entity Semantics word in the described neighbor image class and the degree of correlation of described image to be marked; Wherein,

Ii is the neighbor image among the neighbor image class S, and iq is image to be marked; P (ii/iq) equals the visual signature similarity of ii and described iq;

Described Entity Semantics mark subelement, concrete being used for according to described degree of correlation order from big to small, the Entity Semantics word of choosing predetermined number from described neighbor image class carries out the Entity Semantics mark to described image to be marked.

13. device according to claim 8 is characterized in that, described theme extraction unit, and concrete being used for utilizes the described text of following to set up the LDA model, based on the described theme of described LDA model extraction;

Described the first associative cell, concrete being used for set up image-Topic relative matrix Rvt based on described LDA model;

Described cluster cell comprises:

The Topic relative matrix unit is used for setting up the described Topic relative matrix Rt that follows text;

The visual similarity matrix unit for the visual similarity that utilizes image, is calculated the visual similarity matrix Rv of the similar image of described semanteme;

Complicated graph model unit is used for utilizing Rt, Rvt, Rv, sets up complicated graph model G (Rv, Rt, Rvt);

Complicated figure cluster cell is used for described complicated figure G (Rv, Rt, Rvt) is carried out cluster, forms described image class set and the set of described theme class.

14. each described device according to claim 8-13, it is characterized in that, described much-talked-about topic mark unit specifically is used for, and utilizes front K the word that evolution method of inspection χ 2 extracts and the described network hot topic degree of correlation is the highest that described image to be marked is carried out semantic tagger.

15. the image search method of a much-talked-about topic Network Based is characterized in that, described network hot topic utilizes that each method obtains among the claim 1-7, and described method comprises:

Receive the image retrieval text of user's input; Described image retrieval text comprises a network hot topic word at least;

Image on the crawl internet and the markup information of described image thereof;

Whether the markup information of judging described image is complementary with described image retrieval text;

Such as coupling, then with the described image that is complementary and markup information output thereof.

16. method according to claim 15 is characterized in that, when comprising simultaneously network hot topic and Entity Semantics in the described image retrieval text, whether the described markup information of judging described image is complementary with described image retrieval text and comprises:

Judge in the markup information of described image and whether comprise described network hot topic word.

17. according to claim 15 or 16 described methods, it is characterized in that, described as coupling, then will the described image that is complementary and markup information export and comprise:

Such as coupling, then will export with the content of described image retrieval text-dependent in the described image that is complementary and the markup information thereof.

18. the image retrieving apparatus of a much-talked-about topic Network Based is characterized in that, described network hot topic utilizes that each method obtains among the claim 1-7, and described device comprises:

The text receiving element is used for receiving the image retrieval text that the user inputs; Described image retrieval text comprises a network hot topic word at least;

Placement unit is for the image on the crawl internet and the markup information of described image thereof;

Judging unit is used for judging whether the markup information of described image is complementary with described image retrieval text;

Output unit is used for exporting the described image that is complementary and markup information thereof and exporting when the markup information of described image is complementary with described image retrieval text.

19. device according to claim 18, it is characterized in that, described judging unit, judges in the markup information of described image whether comprise described network hot topic word at concrete being used for when described image retrieval text comprises network hot topic and Entity Semantics simultaneously.

20. according to claim 18 or 19 described devices, it is characterized in that, described output unit, will be exported with the content of described image retrieval text-dependent in the described image that is complementary and the markup information thereof at concrete being used for when the markup information of described image is complementary with described image retrieval text.

Images