CN100413316C - Ultra-resolution ratio reconstructing method for video-image - Google Patents

Abstract

The present invention relates to an ultra-resolution ratio reconstructing method of video images, which uses at least an observation frame to repair ultra-resolution ratio images. The method comprises the following steps that firstly, a reference frame as the pre-estimated value of the ultra-resolution images is constructed; secondly, at least a pixel point in an observation frame is projected to the reference frame by a motion estimation method; thirdly, the proper repair function is selected according to the position of the projected pixel point in the reference frame to calculate the grey scale estimated value of the projected pixel point; fourthly, when the difference of the grey scale estimated value of the projected pixel point and the actual grey scale value of the corresponding pixel point in the observation frame is greater than the threshold value determined by the current observation frame, the images in the reference frame are repaired. Furthermore, the method can reduce the edge oscillation effect and enhance the reconstructing image accuracy.

Description

A kind of ultra-resolution ratio reconstructing method for video-image

Technical field

The present invention relates to the video processing technique in the communications field, particularly relate to a kind of improved ultra-resolution ratio reconstructing method for video-image based on the convex set projection algorithm.

Background technology

Because people are endless to the demand of information transfer rate, so-called " broadband " is a relative concept also, and " arrowband " is absolute, and bandwidth and people's demand is compared always narrow.Therefore, how effectively a large amount of information of transmission is the focus of present this Research on Communication Technology of broadband connections, for multimedia communication, wishes that information transmitted is few more good more, and for people's audiovisual demand, wishes that then the information of obtaining is The more the better.

The most direct method that improves spatial resolution be by the transducer manufacturing technology reduce pixel size (such as, increase the pixel count of unit are).Yet along with Pixel Dimensions reduces, available luminous energy also can reduce, and consequent grain noise can make the picture quality serious degradation.The approach that another one improves spatial resolution is to increase chip size, yet it can cause the increase of capacitance.These two kinds too high by hardware method for processing cost.

Super-resolution reconstruction is the key technology that addresses this problem.It utilizes signal processing technology to obtain the image (or sequence) of a panel height resolution from several low-resolution images that observe.Thereby when improving spatial resolution, do not take too much bandwidth, perhaps spend too high hardware cost.

Super-resolution reconstruction generally is made of interpolation, estimation and three links of image repair, and wherein whether the estimation subpixel information that is related to consecutive frame can be used for the reconstruct of reference picture exactly.The estimation of subpixel is forbidden the reconstructed image distortion that causes, can regard that the motion estimation error noise causes as, in existing a lot of reconstructing methods, this motion estimation error all is left in the basket usually, or simple hypothesis motion estimation error between all low-resolution images all is identical, thereby influenced the precision of reconstructed image.

In addition, in convex set projection (POCS) algorithm of current use, adopt image correction process usually based on point spread function (PSF).When using this method to repair, tangible bright rays can appear in the place at the image arris, edge oscillatory occurences that Here it is.Shown in Fig. 8 (a), darkening of a side pixel of edge's dark color, and a side pixel of light color becomes more shallow, and it is owing to all having adopted isotropic reparation function to cause to having a few in the image.

Summary of the invention

The objective of the invention is to propose a kind of ultra-resolution ratio reconstructing method for video-image, this method can improve the precision of reconstructed image.

According to ultra-resolution ratio reconstructing method for video-image of the present invention, it uses at least one observer frame that super-resolution image is repaired, and may further comprise the steps:

Step 1, the structure reference frame is as the evaluation of estimating of super-resolution image;

Step 2 by method for estimating, projects to reference frame with at least one pixel in the observer frame;

Step 3 is used the gray scale estimated value of repairing function calculation projection image vegetarian refreshments;

Step 4, when the difference of the actual grey value of corresponding pixel points in the gray scale estimated value of projection image's vegetarian refreshments and the observer frame greater than to the determined threshold value of current observer frame the time, reference frame is carried out image repair.

Wherein, described reparation function is to select according to the position of projection image's vegetarian refreshments in reference frame.

Above-mentioned steps 3 can may further comprise the steps:

Step 31 judges whether projection image's vegetarian refreshments is in marginal position in reference frame;

Step 32 when projection image's vegetarian refreshments is in marginal position in reference frame, is further judged the residing edge direction of projection image's vegetarian refreshments;

Step 33, and select anisotropic point spread function to calculate the gray scale estimated value of projection image's vegetarian refreshments according to the direction of being judged.

Wherein, use Sobel operator method to judge whether pixel is in marginal position in reference frame in the step 31.

Wherein, use is judged the residing edge direction of projection image's vegetarian refreshments based on the edge direction determination methods of cluster in the step 32.

In the method for the invention, step 3 also can may further comprise the steps:

Step 31 judges whether projection image's vegetarian refreshments is in marginal position in reference frame;

Step 32 when projection image's vegetarian refreshments is not in marginal position in reference frame, uses isotropic point spread function to calculate the gray scale estimated value of projection image's vegetarian refreshments.

Wherein, use Sobel operator method to judge whether pixel is in marginal position in reference frame in the step 31.

In addition, in said method, can use different threshold values for different observer frames.Wherein, can utilize the statistical variance of motion estimation error, determine described threshold value; Perhaps, according to the relation of motion estimation error statistical variance and frame pitch, directly calculate described threshold value.

Compared with prior art, the present invention adopts suitable reparation function to repair by the projection image's vegetarian refreshments for diverse location, can improve the precision of reconstructed image.Further, in the present invention, owing to used edge constraint and, made the edge oscillation effect obviously reduce based on the constraints of motion estimation error.

Description of drawings

Fig. 1 is the flow chart of existing P OCS algorithm specific algorithm flow process.

Fig. 2 is according to existing POCS algorithm, uses the flow chart of the flow process that observer frame repairs reference frame.

Fig. 3 is to be the curve chart of the relation of the frame pitch that comes out of example and motion estimation error distribution variance with MobileAndCalendar.

Fig. 4 is the comparison that utilizes the sum of errors frame pitch relation curve that comes out among motion estimation error variance that fitting formula calculates and Fig. 3.

Fig. 5 is the POCS algorithm of adjusting according to threshold value, uses the flow chart of the flow process that observer frame repairs reference frame.

Fig. 6 is based on the projection reconstruction methods of edge self-adaption, uses the flow chart of the flow process that observer frame repairs reference frame.

Fig. 7 describes in detail according to the projection reconstruction methods based on edge self-adaption of the present invention, judges the flow chart of the process of edge direction.

Fig. 8 (a) is the correction image that existing POCS algorithm obtains, and Fig. 8 (b) is the correction image that POCS algorithm according to the present invention obtains.

Embodiment

For a more clear understanding of the present invention, before explanation the present invention, at first accompanying drawings has convex set projection (POCS) algorithm now.The basic procedure that the POCS algorithm is realized is to set up reference frame, according to the observed image sequence reference frame is revised then, up to obtaining the acceptable reconstruction result.

Fig. 1 illustrates the concrete algorithm flow of existing P OCS algorithm.At step S1, structure reference frame Frame-R is as the pre-estimation image of super-resolution image reconstruction.General way is to take out a frame to carry out grey scale interpolation in video sequence, to meet the requirements of resolution.

After setting up reference frame, project to remaining frame in the observation sequence in the reference frame.At first, read in a observer frame in the observation sequence at step S2, Frame-O#1 for example is as current observer frame Frame-O.Then, at step S3, use current observer frame Frame-O that reference frame Frame-R is carried out image repair, the process of image repair will specify with reference to figure 2 below.Then, in step S4, after finish current observer frame Frame-O, judged whether to finish frames all in the observation sequence, if also have the frame of not carrying out repair in the observation sequence, then read in this frame, Frame-O#2 for example, 3,4... or the like, repeated execution of steps S2-S4 continues reference frame Frame-R is repaired, all frame in finishing observation sequence.

Fig. 2 illustrates according to existing POCS algorithm, the flow process of using observer frame (for example observer frame Frame-O#N) that reference frame Frame-R is repaired.

At step S31, read in a pixel among the observer frame Frame-O#N, for example P1 then, projects to this pixel among the reference frame Frame-R at step S32, as projection image's vegetarian refreshments, for example PP1.Be generally guarantee in the image be projected to correct position in the reference frame a little, must carry out estimation, the alternate position spike (motion vector) of an object in two two field pictures of promptly seeking common ground, the method that is adopted among the present invention is common method for estimating, detailed process is no longer described here.

Then, at step S33, use point spread function (PSF) to calculate the gray scale estimated value GE (PP1) of the vegetarian refreshments PP1 of projection image.In step S34, the absolute value of the difference between the gray scale estimated value GE (PP1) of the calculating vegetarian refreshments PP1 of projection image and the actual grey value GR (P1) of actual pixels point P1.At step S35, judge that whether this absolute difference is greater than the threshold value δ that stipulates ₀If absolute value is greater than defined threshold δ ₀, and iterations do not reach the number of times of regulation, then the corresponding pixel points in the reference frame revised (step S36, concrete modification method will be described in detail later).Otherwise, finish the work of this point, and all pixels among the current observer frame Frame-O have been judged whether to finish, if also have the pixel of not carrying out repair among the current observer frame Frame-O, then read in this pixel, for example P2,3,4... or the like, proceed the reparation of reference frame Frame-R, all pixels in finishing current observer frame Frame-O.

In the POCS algorithm of prior art, threshold value δ ₀For each observer frame in the observation sequence all is identical.Yet usually, along with the increase of frame pitch, although the change of background in each frame is little, the position and the shape of object have evident difference, and the hangover that this species diversity can make motion estimation error distribute is elongated.Therefore suppose simply that motion estimation error is identical between all observation images, can influence the precision of reconstructed image.

Usually, the pass between frame pitch and the motion estimation error distributed constant is that along with the increase of frame pitch, the variance of error can increase.This distribution can be regarded the gaussian shaped profile of broad sense as, and the error of its standard deviation and estimation is proportional, and its mean approximation is 0.Therefore, by at threshold value δ ₀The middle motion estimation error of introducing distributes with the variation of frame pitch, can improve the image reconstruction quality.

δ ₀(k)＝λσ(k) (1)

The standard deviation of the motion estimation error noise between σ (k) expression reference frame and k the low-resolution frames, λ represents confidential interval.Can realize the self adaptation value of σ (k) by the variance of statistical error.

Fig. 3 is to be the frame pitch that comes out of example and the relation of motion estimation error distribution variance with MobileAndCalendar.Wherein added up 6 observer frames altogether, its corresponding data as shown in Table 1:

Form 1

Frame pitch	1	2	3	4	5	6
							Variance	27.6866	35.1121	39.6863	43.2536	45.9148	47.8995

Can see that from form 1 along with the increase of frame pitch, variance increases gradually.Because the amount of calculation for each observer frame statistics motion estimation error variance is bigger, therefore, approximate way is directly to calculate the motion estimation error variance of observer frame according to certain computing formula that data fitting obtains.Provide an example that obtains the polynomial computation formula by data fitting below.

Because the characteristic of different video sequences is different, so its computing formula that is suitable for all can be different with fitting parameter.Here change slowly sequence (for example MobileAndCalendar sequence) for scene and made statistical research.

At first, utilize the suitably order of the selected polynomial fitting of degree, by calculating when order is 4, match is well spent near 0.5, therefore selecting exponent number is the sum of errors frame pitch relation curve that comes out among fitting of a polynomial Fig. 3 of 4, the multinomial of match is as follows to the end:

d(n)＝-0.2×n ⁴+3.1×n ³-16.5×n ²+40.5×n+0.2 (2)

Wherein n represents the distance between reference frame and the consecutive frame, and 0≤| n|≤5.

Fig. 4 has shown the comparison that utilizes the sum of errors frame pitch relation curve that comes out among motion estimation error variance that this fitting formula calculates and Fig. 3, can see that both goodnesses of fit are fine.Therefore, when repairing reference frame, can utilize above-mentioned formula (2) directly to calculate the variance of motion estimation error according to the distance of this observer frame and reference frame for the observer frame that uses other.

For general image sequence,, can find that it presents following relation by the relation between statistics frame spacing and the variance:

d(n)＝a(1-e ^-bn) (3)

A wherein, b is a parameter to be estimated, is called amplitude factor and scale factor, can be determined by concrete statistical sample value.Afterwards, in the reparation of reference frame, the distance of frame and reference frame utilizes above-mentioned formula (3) directly to calculate the variance of motion estimation error according to the observation.

Therefore, the present invention carries out introducing in the employed threshold value in the image repair variance of motion estimation error and this relation of frame pitch at above-mentioned POCS algorithm, for different observer frames, by the frame pitch between itself and the reference frame, determine using this observer frame that reference frame is carried out the threshold value used in the process of image repair, thereby can improve the precision of finishing the reconstructed image after repairing and the convergence of improving repair process.Fig. 5 illustrates the POCS algorithm of adjusting according to threshold value of the present invention, uses observer frame, observer frame Frame-O#N for example, the flow process that reference frame Frame-R is repaired.Contrast Fig. 2 can see, before reading in the pixel of observer frame, has increased according to frame pitch calculated threshold δ ₀Step, its Calculation Method can be according to as mentioned above, by determining the fitting formula form, the match statistical sample obtains the fitting formula parameter, calculates the threshold value of motion estimation error then, and is identical among step afterwards and Fig. 2.

The restorative procedure of image as step S32 among Fig. 2 to shown in the S36, promptly, motion vector according to the estimation gained projects to (step S32) on the reference frame to the pixel of a frame observed image, find out the interior pixel of PSF scope of projected pixel in the reference frame, calculate the gray scale estimated value (step S33) of projected pixel by PSF, compare with the actual grey value then (step S34), if error exceeds allowed band, then the pixel in the reference frame is revised in error is reduced to the scope of permission (step S36).

Specify below among the step S33 and step S36 of Fig. 2, calculate the gray scale estimated value of projection image's vegetarian refreshments and the method that pixel in the reference frame is revised by PSF.Usually, the PSF of observed image is determined by concrete imaging system, generally adopts following common PSF model:

$\mathrm{\σ}(x,y)=e^{\frac{{(x-X_0)}^2+{(y-Y_0)}^2}{2}}---\left(4\right)$

In the formula, X ₀And Y ₀Center point coordinate value for point spread function.It is constant that postulated point spread function PSF is that linearity is moved, make σ (x, y) be σ (x, normalized function value y), promptly

${\mathrm{\&sigma;}}^{\&prime;}(x,y)=\frac{\mathrm{\&sigma;}(x,y)}{\underset{m=X_0-\mathrm{<figure-callout\; id="0"\; label="R\; X"\; filenames="C20061000749200151.png"\; state="\{\{state\}\}">R</figure-callout>}}{\overset{X_{}}{\mathrm{\&Sigma;}}}\underset{n=Y_0-\mathrm{<figure-callout\; id="0"\; label="R\; Y"\; filenames="C20061000749200151.png"\; state="\{\{state\}\}">R</figure-callout>}}{\overset{Y_{}}{\mathrm{\&Sigma;}}}\mathrm{\&sigma;}(m,n)}---\left(5\right)$

Wherein R represents the effective range of PSF, uses f _Ref(then the gray scale estimated value of this projection image's vegetarian refreshments is for x, the y) pixel in the expression reference frame:

$\hat{f}({\mathrm{<figure-callout\; id="0"\; label="X"\; filenames="C20061000749200151.png"\; state="\{\{state\}\}">X</figure-callout>}}_0,Y_0)=\underset{m=X_0-\mathrm{<figure-callout\; id="0"\; label="R\; X"\; filenames="C20061000749200151.png"\; state="\{\{state\}\}">R</figure-callout>}}{\overset{X_{}}{\mathrm{\&Sigma;}}}\underset{n=Y_0-\mathrm{<figure-callout\; id="0"\; label="R\; Y"\; filenames="C20061000749200151.png"\; state="\{\{state\}\}">R</figure-callout>}}{\overset{Y_{}}{\mathrm{\&Sigma;}}}{f}_{\mathrm{ref}}(x,y){\mathrm{\&sigma;}}^{\&prime;}(x,y)---\left(6\right)$

With y (then the error between gray scale estimated value and the observed image gray value is for m, n) expression observed image gray value:

$r=y(m,n)-\hat{f}({\mathrm{<figure-callout\; id="0"\; label="X"\; filenames="C20061000749200151.png"\; state="\{\{state\}\}">X</figure-callout>}}_0,Y_0)---\left(7\right)$

If the absolute value of r is greater than specified threshold value δ ₀, just the grey scale pixel value in the PSF scope in the reference frame is revised according to following formula (8):

f _ref(x，y)＝f _ref(x，y)+pδ ₀σ′(x，y) (8)

In the formula, 0≤p≤1, X ₀-R≤x≤X ₀+ R, Y ₀-R≤y≤Y ₀+ R.

Consider the correction that in repair process, also can be subjected to other constraintss, generally get p＜1, adopt following mode:

$f_{\mathrm{ref}}(x,y)=\left\{\begin{array}{cc}{f}_{\mathrm{ref}}(x,y)+(r+{\mathrm{\&delta;}}_0){\mathrm{\&sigma;}}^{\&prime;}(x,y)& r<-{\mathrm{\&delta;}}_0\\ f_{\mathrm{ref}}(x,y)& -{\mathrm{\&delta;}}_0\&le;r\&le;{\mathrm{\&delta;}}_0\\ f_{\mathrm{ref}}(x,y)+(r-{\mathrm{\&delta;}}_0){\mathrm{\&sigma;}}^{\&prime;}(x,y)& r>{\mathrm{\&delta;}}_0\end{array}\right.---\left(9\right)$

With f _Ref(x, value y) is adapted in the error range.

Isotropic reparation function is unreasonable to reconstruct, because image is a kind of non-stationary process, especially for marginal point, it is very obvious with on every side some difference, isotropic reparation function to have a few and all carried out identical processing, can not well keep marginal point level and smooth along edge direction, stride the sharp-pointed characteristic in edge.And the common employing of POCS algorithm at present when using this method to repair, tangible bright rays can occur, edge oscillatory occurences that Here it is based on the image correction process of isotropism PSF at image arris place.Shown in Fig. 8 (a), darkening of a side pixel of edge's dark color, and a side pixel of light color becomes more shallow, it is owing to all having adopted isotropic reparation function to cause to having a few in the image, and the point that wherein adopts the subpoint both sides of edges carries out gray scale to it and estimates that calculating can introduce unnecessary gamma error.Therefore when repairing, consider the character of each point in the image, and then different points is adopted different reparation functions.

Owing to use isotropic point spread function to carry out the projection reparation in traditional method, caused the image border vibration; In order to protect marginal information, the present invention proposes projection reconstruction methods based on edge self-adaption, promptly the pixel for non-flat skating area and edge adopts anisotropic reparation function.Realize anisotropic repair process, idiographic flow as shown in Figure 6.

Among Fig. 6 when handling current observer frame, the step S30 of front is identical in S32 and Fig. 5, at step S331, judge the vegetarian refreshments PP1 of the projection image edge pixel point whether read in pixel P1, concrete determination methods can use Sobel operator method of the prior art to carry out.When the judged result explanation vegetarian refreshments PP1 of projection image of step S331 is edge pixel point, the direction under step S332 then judges this marginal point.Among the present invention, adopt and a kind ofly judge the edge direction of central point, select suitable reparation function (below will describe the process of judging edge direction and selecting the reparation function) at step S333 then based on the edge direction determination methods of cluster.If the judgement of the S331 explanation vegetarian refreshments PP1 of projection image is not the edge pixel point, then use isotropic PSF that it is revised.

Describe the process of judging edge direction and selecting to repair function in detail below with reference to Fig. 7.The coverage of postulated point spread function is that radius is 3 * 3 zone, as shown in Figure 7 be 9 points at center with E, judge the edge direction that E is affiliated, realize accurate judgement, can be according to the thought of cluster, point in 3 * 3 zones is carried out the classification of different directions, wishes that distance is enough little in the class, interior between distance enough greatly.Class distance in the class statistical variance can be weighed in utilizing, this sentences 4 directions is example, be horizontal direction (DEF), oblique 45 ° of directions (CEG), vertical direction (BEH), oblique 135 ° of directions (AEI) (only for example, it is also conceivable that other direction such as CEH, BEH, AEH, DEI etc. herein) for improving precision with 4 directions.Class variances in four classification directions around computer center's point at first as follows,

var ₁＝{[C-mean(D，E，F)] ²+[E-mean(D，E，F)] ²+[F-mean(D，E，F)] ²}/3，

var ₂＝{[C-mean(C，E，G)] ²+[E-mean(C，E，G)] ²+[G-mean(C，E，G)] ²}/3，

var ₃＝{[B-mean(B，E，H)] ²+[E-mean(B，E，H)] ²+[H-mean(B，E，H)] ²}/3，

var ₄＝{[A-mean(A，E，I)] ²+[E-mean(A，E，I)] ²+[I-mean(A，E，I)] ²}/3；

Find out reckling var wherein then _Min=min{var ₁, var ₂, var ₃, var ₄, minimum implication is three deviation minimums between the picture element gray value, they should belong to and are that same class edge direction, the direction of variance subscript correspondence are the direction under the central point; The last direction that obtains according to judgement is selected to repair function and is carried out the gray scale correction:

Can see, be the marginal point of horizontal direction for edge direction, only selects the neighbor pixel of its horizontal direction that it is carried out gray scale estimation calculating, the i.e. pixel of y=Y0; Similarly, for vertical, the pixel of oblique 45 ° and oblique 135 ° edge direction all only adopts the pixel of its edge direction that it is carried out gray scale and estimates to calculate, be i.e. x=X ₀, x-X ₀=y-Y ₀, x-X ₀=-(y-Y ₀).Like this,, thereby can reduce the gamma error of introducing, and eliminate the vibration of unnecessary edge because pixel and projection image's vegetarian refreshments gray value on the same edge are approaching.

The comparison of Fig. 8 identical reparation image graph 8 (a) that to be the reparation image graph 8 (b) that obtains according to improvement according to the present invention POCS algorithm obtain with common POCS algorithm.As can be seen from Figure 8, adopt method of the present invention, can recover certain details.Owing to used edge constraint and based on the constraints of motion estimation error, after carrying out 3 iteration, the result of method gained of the present invention does not compare with using these constraintss, and the edge oscillation effect obviously reduces, the numeral 26 among the gained result and 31 clear than among Fig. 8 (a).

Claims (9)

1. ultra-resolution ratio reconstructing method for video-image, it uses at least one observer frame that super-resolution image is repaired, and may further comprise the steps:

Step 1, the structure reference frame is as the evaluation of estimating of super-resolution image;

Step 2 by method for estimating, projects to reference frame with at least one pixel in the observer frame;

Step 3 is used the gray scale estimated value of repairing function calculation projection image vegetarian refreshments;

Step 4, when the difference of the actual grey value of corresponding pixel points in the gray scale estimated value of projection image's vegetarian refreshments and the observer frame greater than to the determined threshold value of current observer frame the time, reference frame is carried out image repair.

2. ultra-resolution ratio reconstructing method for video-image as claimed in claim 1 wherein, utilizes the statistical variance of motion estimation error, determines described threshold value.

3. ultra-resolution ratio reconstructing method for video-image as claimed in claim 1 wherein, according to the relation of estimation statistical variance and frame pitch, directly calculates described threshold value.

4. ultra-resolution ratio reconstructing method for video-image as claimed in claim 1, wherein, described reparation function is to select according to the position of projection image's vegetarian refreshments in reference frame.

5. ultra-resolution ratio reconstructing method for video-image as claimed in claim 1, wherein, step 3 may further comprise the steps:

Step 31 judges whether projection image's vegetarian refreshments is in marginal position in reference frame;

Step 32 when projection image's vegetarian refreshments is in marginal position in reference frame, is further judged the residing edge direction of projection image's vegetarian refreshments;

Step 33, and select anisotropic point spread function to calculate the gray scale estimated value of projection image's vegetarian refreshments according to the direction of being judged.

6. ultra-resolution ratio reconstructing method for video-image as claimed in claim 5 wherein, uses Sobel operator method to judge whether pixel is in marginal position in reference frame in the step 31.

7. as claim 5 or 6 described ultra-resolution ratio reconstructing method for video-image, wherein, use in the step 32 and judge the residing edge direction of projection image's vegetarian refreshments based on the edge direction determination methods of cluster.

8. ultra-resolution ratio reconstructing method for video-image as claimed in claim 1, wherein, step 3 may further comprise the steps:

Step 31 judges whether projection image's vegetarian refreshments is in marginal position in reference frame;

Step 32 when projection image's vegetarian refreshments is not in marginal position in reference frame, uses isotropic point spread function to calculate the gray scale estimated value of projection image's vegetarian refreshments.

9. ultra-resolution ratio reconstructing method for video-image as claimed in claim 8 wherein, uses Sobel operator method to judge whether pixel is in marginal position in reference frame in the step 31.

Images