US20010016005A1

US20010016005A1 - Brightness-variation compensation method and coding/decoding apparatus for moving pictures

Info

Publication number: US20010016005A1
Application number: US09/832,451
Authority: US
Inventors: Kazuto Kamikura; Hiroshi Watanabe; Hirohisa Jozawa
Original assignee: Nippon Telegraph and Telephone Corp
Current assignee: Nippon Telegraph and Telephone Corp
Priority date: 1996-09-03
Filing date: 2001-04-11
Publication date: 2001-08-23
Anticipated expiration: 2017-08-29
Also published as: US20020196849A1; US6934331B2; US6266370B1; US6456658B2

Abstract

A moving image brightness variation compensation method for encoding digital moving images for transmission and storage, and for image processing when editing moving images, the moving image brightness variation compensation method comprising a step of compensating for overall brightness variations by correcting a luminance value x of each pixel according to the formula D_C·x+ D_B, wherein D_Bis a parameter indicating a gain change and D_Cis a parameter indicating a contrast change, the parameters representing overall luminance changes between a reference image plane and an image plane being processed.

Description

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a method of compensating for the overall brightness variation between image frames required for image processing such as in video editing and coding for efficient transmission and storage of digital video images.

2. Background Art

As a conventional coding technique for efficiently transmitting and storing video image data, a technique known as motion compensation interframe prediction coding has been widely used. With this technique, the luminance values (or color differences) of the image frames being processed are not directly transmitted and stored; instead, the technique measures how much objects inside an image have moved between a video image frame which has already been coded (referred to as a reference image frame) and the image frame being processed, and luminance difference values at positions which are shifted by the distance of movement are transmitted and stored.

Movement compensation interframe prediction coding enables transmission and storage of reduced quantities of data because the difference values are generally less than the luminance values themselves.

In this case, when measuring movement, it is usually assumed that the image frame being processed and the reference image frame have the same lighting conditions. That is, movement is detected under the assumption that corresponding objects in both frames have the same luminance values. However, the luminance values for corresponding objects can differ considerably between frames in cases such as when there is flickering, when a strobe light has flashed, or when the entire image is made gradually brighter (hereinafter referred to as “fade-in”) or darker (hereinafter referred to as “fade-out”) by adjusting the diaphragm of the camera or by using video editing equipment. Consequently, the following problems are associated with motion compensation interframe prediction coding:

(1) The detection of movement is not properly performed.

(2) The coding efficiency is not sufficiently improved.

As a conventional method principally for resolving the above-mentioned problem (1), there is a technique of measuring luminance changes between image frames, and performing movement measurements while correcting for the measured luminance changes.

Luminance changes can be largely separated into types wherein the degree of change changes according to the spatial position in the image frame, such as those which are caused by noise, and types wherein the degree of change is the same over the entire image frame, such as those which are caused by fade-ins and fade-outs.

Since it is important to measure the luminance changes as accurately as possible when attempting to resolve problem (1), the measurements are usually made in small areas within the frame in order to handle both types of luminance changes. However, although this resolves problem (1), the size of the interframe luminance difference values can still result in problem (2).

Thus, there is also a technique of reducing the interframe luminance difference values by transmitting and storing the luminance changes of the small areas as coded data, and correcting for these luminance changes at the decoder side. However, since this technique involves sending coded data expressing the luminance change for each small area, the quantities of data are ultimately increased to such a degree that the overall data quantity is not much different. In other words, the coding efficiency is not much improved.

On the other hand, there is a technique of measuring a single luminance change for the image frame as a whole, and transmitting and storing that luminance change as coded data for the purposes of improving the coding efficiency when the same luminance change has occurred over the entire image frame. With this technique, the quantity of coded data is extremely small because it is sufficient to send only a single piece of coded data expressing the luminance change for each image frame. Additionally, this technique contributes largely to the improvement of coding efficiency in cases of fade-ins and fade-outs because the interframe luminance change values can be made somewhat smaller simply by correcting for the luminance change.

As a specific parameter for expressing the overall luminance change, this technique uses only a parameter expressing the gain difference. That is, taking a parameter D _Brepresenting the gain difference, it is assumed that the luminance value x of each pixel will change to x′ as shown in Equation 1:

x′=x+D _B (1)

In this case, the luminance change x is always constant:

Δx=x′−x=D _B (2)

FIG. 3 shows the changes in the luminance values in an actual case wherein a still image containing various brightness values was taken by a camera and the diaphragm of the camera was gradually closed so as to darken the overall image, i.e. a fade-out. When considering the changes in the luminance values from

frame

1 to frame 11, or frame 11 to frame 21, the luminosity changes are approximately constant regardless of the luminance values themselves. Therefore, in this case, the assumption of Equation (1) holds. That is, the constant amount of change corresponds to D_Bin Equation (1).

On the other hand, FIG. 4 shows the changes in the luminance values in a case wherein the same still image was taken without changing the diaphragm of the camera, then a fade-out was caused using digital editing equipment. When considering the changes in the luminance values from

frame

41 to frame 51, or frame 51 to frame 61, the luminance changes change depending on the luminance values. Therefore, the assumption of Equation (1) does not hold in this case, and the precision is insufficient to express overall luminance changes using only the parameter D_B.

When the images are divided into small areas of 8×8 pixels, the variations in the luminances of these small areas are generally not very large. For example, when the luminance value in the image frame overall changes from 0-255, the variations in the small areas are almost always in the range of 20-40. In this range, the luminance change can be considered to be approximately constant without regard to the luminance values themselves. That is, the assumption of Equation (1) is still appropriate if the luminance change is found only in small areas.

However, the range of luminance values broadens when the image frame is taken as a whole. That is, there is a considerable possibility that there will be luminance changes covering the entire range of luminance values 0-255. Therefore, the approximation of Equation (1) is insufficient for determining one type of luminance change over the entire image frame.

SUMMARY OF THE INVENTION

The object of the present invention is to offer a brightness variation compensation method and coding/decoding apparatus for moving images, capable of more efficiently coding moving images with overall brightness variations.

Therefore, the present invention offers a moving image brightness variation compensation method for encoding digital moving images for transmission and storage, and for image processing when editing moving images, the moving image brightness variation compensation method comprising a step of compensating for overall brightness variations by correcting a luminance value x of each pixel according to the formula D _C·x+ D_B, wherein D_Bis a parameter indicating a gain change and D_Cis a parameter indicating a contrast change, the parameters representing overall luminance changes between a reference image plane and an image plane being processed.

According to this method, coding procedures can be made more efficient when coding moving images having overall brightness variations, due to the use of a parameter indicating the contrast change in addition to a parameter indicating the gain change, when measuring or compensating for overall brightness variations.

Additionally, the present invention offers a moving image coding apparatus, comprising parameter determining means for determining a parameter D _Bindicating a gain change and a parameter D_Cindicating a contrast change, the parameters representing overall luminance changes between a reference image plane and an image plane being processed; and brightness variation compensation means for correcting a luminance value x of each pixel in the image plane being processed using the parameters D_Band D_Caccording to the formula D_C·x+ D_B.

According to this apparatus, coding procedures can be made more efficient when coding moving images having overall brightness variations, due to the use of a parameter indicating the contrast change in addition to a parameter indicating the gain change, when measuring or compensating for overall brightness variations.

Additionally, the present invention offers a computer-readable recording medium recording a moving image coding program, the moving image coding program containing instructions for a computer to execute a parameter determining step of determining a parameter D _Bindicating a gain change and a parameter D_Cindicating a contrast change, the parameters representing overall luminance changes between a reference image plane and an image plane being processed; and a brightness variation compensation step of correcting a luminance value x of each pixel in the image plane being processed using the parameters D_Band D_Caccording to the formula D_C·x+ D_B.

By reading the program recorded onto this recording medium and executing the program with a computer, coding procedures can be made more efficient when coding moving images having overall brightness variations, due to the use of a parameter indicating the contrast change in addition to a parameter indicating the gain change, when measuring or compensating for overall brightness variations.

Additionally, the present invention offers a computer-readable recording medium recording coded data of moving images, wherein the coded data includes a plurality of areas recording image plane coded data encoded by units of image planes, the areas recording image plane coded data comprising a parameter area recording parameters D _Bindicating gain changes and D_Cindicating contrast changes, the parameters representing overall luminance changes between reference image planes and image planes being processed; and a coded area recording data in units of image planes relating to differences in pixel values for corresponding pixels between the reference image planes and image planes wherein luminance values x of the pixels in the reference image planes have been corrected by using the parameters D_Band D_Caccording to the formula D_C·x+ D_B.

Since the coded data on this recording medium includes parameters indicating the contrast change in addition to parameters indicating the gain change when measuring or compensating for overall brightness variations, and includes data coded by using these parameters, coding procedures can be made more efficient when coding moving images having overall brightness variations.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing a moving image coding apparatus using the moving image luminance change compensation method of the present invention. [0029]
FIG. 2 is a block diagram showing a moving image decoding apparatus using the moving image luminance change compensation method of the present invention. [0030]
FIG. 3 is a diagram for explaining overall luminance changes of images. [0031]
FIG. 4 is a diagram for explaining overall luminance changes of images. [0032]
FIG. 5 is a diagram for explaining the manner of division of an image frame. [0033]
FIG. 6 is a diagram for explaining the structure of coded data in the present invention. [0034]
FIG. 7 is a diagram showing the structure of coded data for a case wherein an area is provided to record the parameters for each block. [0035]
FIG. 8 is a diagram showing the structure of coded data for a case wherein an area is provided to record only the parameter D[0036] _Bwhich represents the gain change for each image frame.
FIG. 9 is a diagram for explaining the video object (VO), video object layer (VOL) and the video object plane (VOP) in MPEG-[0037] 4.

DESCRIPTION OF PREFERRED EMBODIMENTS OF THE INVENTION

Hereinbelow, a brightness variation compensation method and moving image coding/decoding apparatus according to the present invention shall be explained. Brightness variation compensation of moving images and coding/decoding of moving images using brightness variation compensation are performed by units of image planes. Here, “image plane” is a comprehensive expression referring to an “image frame” corresponding to a single frame of a moving image or a “video object plane” (hereinafter abbreviated to “VOP”) which is an image corresponding to the variation over time of the contents of a moving image. Below, an explanation will be made using image frames as examples of image planes. VOPs shall be explained separately later. [0038]
First, the brightness variation compensation method for moving images according to the present invention shall be explained. [0039]
The brightness variation compensation method of the present invention compensates for overall brightness variations by correcting the luminance value x of each pixel by D[0040] _C· x+ D_B, wherein D_Brepresents the gain change and D_Crepresents the contrast change.
In order to resolve the above problems, the moving image brightness variation compensation method of the present invention makes the assumption that the luminance value x of each pixel changes to x′ as shown in Equation (3) due to the parameter D[0041] _Brepresenting the gain change and the parameter D_Crepresenting the contrast change.
x′=D _C ·x+D _B (3)
In this case, the luminance change _x depends on the luminance value x of the pixel. [0042]
Δx=x′−x=(D _C−1)·x+D _B (4)
As described above, when determining a single type of brightness variation over the entire image frame, the use of the Equation (3) which is capable of expressing the brightness variation as a function of the luminance values themselves is highly significant in that the coding efficiency is improved when brightness variations of the image overall such as fade-ins and fade-outs occur. [0043]
According to an embodiment of the present invention, the parameter D[0044] _Brepresenting the gain change and the parameter D_Crepresenting the contrast change are determined under the condition of minimizing the sum of the squared differences between corresponding pixels in the image frame being processed and the reference image frame.
With the luminance change measuring method for moving images according to the present invention, the two parameters D[0045] _Band D_Care measured for each block of K×L (wherein K and L are natural numbers) pixels in the image frame being processed. At this time, as a criterion for calculating these parameters, the sums of the squared differences between corresponding pixels in the image frame being processed and the reference image frame are minimized. Next, the mean value or the most frequent value of each of the two parameters D_Band D_Ccalculated for the blocks of K×L pixels is taken as the parameter for the image frame being processed overall.
Hereinbelow, the equations for determining the parameters for each block of K×L pixels are derived. The two parameters for the i-th block are represented by D[0046] _B(i) and D_C(i), while the pixel values in each block of K×L pixels in the image frame being processed are represented by y_k,land the pixel values in each block of K×L pixels in the reference image frame are represented by x_k,l. As described above, the criterion for calculating the parameters is to minimize the sum of the squared differences J between corresponding pixels in the image frame being processed and the reference image frame. That is: $\begin{matrix} J = \sum_{I = 1}^{L} \sum_{k = 1}^{K} {y_{k, l} - (D_{C} (i) \cdot x_{k, l} + D_{B} (i))}^{2} & (5) \end{matrix}$
Taking the partial derivatives of J with respect to D[0047] _C(i) and D_B(i), then setting them equal to zero results in the following: $\begin{matrix} \frac{\partial J}{\partial D_{C} (i)} = \sum_{I = 1}^{L} \sum_{k = 1}^{K} (2 D_{C} (i) x_{k, l}^{2} - 2 x_{k . l} y_{k, l} + 2 x_{k, l} D_{B} (i)) = 0 ∴ \sum_{I = 1}^{L} \sum_{k = 1}^{K} (D_{C} (i) x_{k, l}^{2} - x_{k, l} y_{k, l} + x_{k, l} D_{B} (i)) = 0 & (6) \\ \frac{\partial J}{\partial D_{B} (i)} = \sum_{I = 1}^{L} \sum_{k = 1}^{K} (2 D_{C} (i) x_{k, l} - 2 y_{k, l} + 2 D_{B} (i)) = 0 ∴ \sum_{I = 1}^{L} \sum_{k = 1}^{K} (D_{C} (i) x_{k, l} - y_{k, l} + D_{B} (i)) = 0 & (7) \end{matrix}$
Equations (6) and (7) further give the following: [0048] $\begin{matrix} D_{C} (i) = \frac{K \cdot L \cdot Z - X \cdot Y}{K \cdot L \cdot T - X^{2}} & (8) \\ D_{B} (i) = \frac{T \cdot Y - X \cdot Y}{K \cdot L \cdot T - X^{2}} & (9) \end{matrix}$
wherein: [0049] $\begin{matrix} X = \sum_{l = 1}^{L} \sum_{k = 1}^{K} x_{k, l} Y = \sum_{l = 1}^{L} \sum_{k = 1}^{K} y_{k, l} T = \sum_{l = 1}^{L} \sum_{k = 1}^{K} x_{k, l}^{2} Z = \sum_{l = 1}^{L} \sum_{k = 1}^{K} (x_{k, l} \cdot y_{k, l}) & (10) \end{matrix}$
As mentioned above, the mean values or most frequent values of the two parameters D[0050] _C(i) and D_B(i) calculated for each block of K×L pixels by means of Equations (8) and (9) are taken as the parameters D_Cand D_Bfor the entire image being processed.
The moving image coding apparatus according to the present invention has parameter determining means for determining the parameter D[0051] _Brepresenting the gain change and the parameter D_Crepresenting the contrast change as parameters which represent the overall luminance change between the image frame being processed and the reference image frame, and brightness variation compensation means for correcting the luminance value x of each pixel in the reference image frame using the parameters D_Band D_C, according to the formula D_C·x+ D_B.
According to an embodiment of the present invention, the parameter determining means determines the parameters D[0052] _Band D_Cby minimizing the sum of the squared differences between corresponding pixels in the image frame being processed and the reference image frame.
In one embodiment of the present invention, the parameter determining means calculates the parameters D[0053] _Band D_Cfor each of n (wherein n is an integer greater than or equal to 2) blocks composed of K×L (wherein K and L are natural numbers) pixels in the image frame being processed, then determines the pair of parameters D_Band D_Chaving the highest frequency to be the parameters D_Band D_Cfor the entire image frame being processed.
In another embodiment of the present invention, the parameter determining means calculates the parameters D[0054] _Band D_Cfor each of n (wherein n is an integer greater than or equal to 2) blocks composed of K×L (wherein K and L are natural numbers) pixels in the image frame being processed, then determines the mean values of the parameters D_Band D_Cto be the parameters D_Band D_Cfor the entire image frame being processed
A moving image decoding apparatus according to the present invention has brightness variation compensation means for correcting the luminance values x of each pixel in the reconstructed image frame after decoding has been completed, using the parameters D[0055] _Band D_Cdetermined by the moving image coding apparatus, according to the formula D_C·x+ D_B.
Next, embodiments of the present invention shall be explained with reference to the drawings. [0056]
FIG. 1 is a block diagram showing a moving image coding apparatus using the moving image brightness variation compensation method of the present invention. [0057]
The moving image coding apparatus comprises an [0058] input terminal 1, frame memories 2 and 3, a motion detection section 4, a brightness variation detection section 5, a frequency calculation section 6, a brightness variation compensation section 7, an output terminal 8, a frame memory 9, a motion detection/compensation section 10, a subtractor 11, a discrete cosine transform section 12, a quantization section 13, an output terminal 14, a dequantization section 15, an inverse discrete cosine transform section 16, an adder 17, and an output terminal 18.
An N-th frame original image is inputted from the [0059] input terminal 1 into frame memory 2, frame memory 3, motion detection section 4 and brightness variation detection section 5. The frame memory 3 stores the N-th frame original image, and sends the (N−1)-th frame original image which had been stored to the motion detection section 4 and the brightness variation detection section 5.
The [0060] motion detection section 4 determines the movement of the N-th frame original image with respect to the (N−1)-th frame original image for each block of 16×16 pixels as shown in FIG. 5, by means of a block matching method. Assuming the number of blocks in each frame to be n, the blocks are labelled as B₁, B₂, . . . , B_n. Additionally, the motion vector calculated for each block B_iis expressed by V_i. The motion vector V_iis sent to the brightness variation detection section 5. At the brightness variation detection section 5, the two parameters D_C(i) and D_B(i) are calculated for each block B_i, using the luminance values of the 16×16 pixels in each block B_iof the N-th frame original image and the luminance values of the 16×16 pixels at positions shifted by the vector V_iin the (N−1)-th frame original image. The resulting two parameters D_C(i) and D_B(i) are sent to the frequency calculation section 6. At the frequency calculation section 6, the frequencies of D_C(i) and D_B(i) are determined over a single frame, the pair of parameters with the highest frequency are taken to be the parameters D_C(i) and D_B(i) representing the overall brightness variation between the N-th frame original image and the (N−1)-th frame original image, and these values are sent to the brightness variation compensation section 7 and sent outside through the output terminal 8.
On the other hand, the N-th frame original image inputted from the [0061] input terminal 1 is delayed by the frame memory 2 for one frame, and then inputted into the subtractor 11. The (N−1)-th frame reconstructed image is stored in the frame memory 9, and the brightness variation compensation section 7 compensates for the brightness variations of the luminance values x in the images by means of Equation (3), and inputs the values x′ which have been compensated for brightness variation into the motion detection/compensation section 10.
The [0062] subtractor 11 takes the difference between the data and sends the difference to the discrete cosine transform section 12. The discrete cosine transform section 12 performs a discrete cosine transform of data for one block composed of 16×16 pixels in units of 8×8 pixels, then the resulting discrete cosine transform coefficient is quantized by the quantization section 13. The quantized discrete cosine transform coefficient is sent outside through the output terminal 14 and also dequantized by the dequantization section 15, then inverse discrete cosine transformed by the inverse discrete cosine transform section 16. The adder 17 adds data sent from the motion detection/compensation section 10 to that data, then stores the result in the frame memory 9 as a N-th frame reconstructed image. The N-th frame reconstructed image stored in the frame memory 9 is used as a reference image when coding the image of the next frame (i.e. the (N+ 1)-th frame).
Alternatively, the average values of the parameters D[0063] _C(i) and D_B(i) (wherein i=1-n) can also be taken as the parameters D_Cand D_B.
The data coded by the moving image coding apparatus is decoded by the moving image decoding apparatus shown in FIG. 2, using the moving image brightness variation compensation method of the present invention. [0064]
The moving image decoding apparatus comprises input terminals [0065] 21-23, a dequantization section 24, an inverse discrete cosine transform section 25, an adder 26, an output terminal 27, a frame memory 28, a brightness variation compensation section 29, and a motion compensation section 30.
The quantized discrete cosine transform coefficient outputted by the [0066] output terminal 14 of FIG. 1 is inputted into the input terminal 21. A movement vector for each block outputted from the output terminal 18 of FIG. 1 is inputted into the input terminal 22. The two parameters D_Cand D_Brepresenting the brightness variation for each frame outputted from the output terminal 8 is inputted into the input terminal 23.
In this case, it is assumed that coded data of a block B[0067] _i(16×16 pixels) of the N-th frame image is inputted. That is, the quantized discrete cosine transform coefficient of block B_iis inputted into the input terminal 21. The movement vector V_iof block B_iis inputted into the input terminal 22. The two parameters D_Cand D_Bwhich are values identical throughout the entire N-th frame are inputted into the input terminal 23.
The quantized discrete cosine transform coefficient inputted from the [0068] input terminal 21 is dequantized by the dequantization section 24, then inverse discrete cosine transformed by the inverse discrete cosine transform section 25 and sent to the adder 26.
The [0069] frame memory 28 contains (N−1)-th frame reconstructed image data which has already been decoded. The brightness variation compensation section 29 performs brightness variation compensation of each luminance value x in the image by means of Equation (3), then inputs the brightness variation compensated values x′ into the motion compensation section 30. The motion compensation section 30 compensates for a shift by the movement vector V_i, and inputs the result into the adder 26.
The [0070] adder 26 obtains a N-th frame reconstructed image by adding the data from the inverse discrete cosine transform section 25 with the data from the motion compensation section 30, outputs the result to the output terminal 27 and also stores the result in the frame memory 28 for use as a reference image when decoding the image of the next frame (i.e. the (N+ 1)-th frame).
Next, the data structure when recording coded data generated by the moving image coding apparatus shown in FIG. 1 onto a computer-readable recording medium shall be explained with reference to FIG. 6. [0071]
In this case, the coded data is coded by units of image frames, and is composed of a plurality of coded image frames. While the structures of the coded data of the image frames in FIG. 6 are identical, the coded data of the N-th frame which is the coded data of the N-th image frame shall be explained. [0072]
As shown in FIG. 6, the coded data of the N-th frame is composed of a [0073] parameter area 70 which is an area for recording the parameter D_C·81 and the parameter D_B·82, and a coded area 71 recording coded data of pixel value difference data for each corresponding pixel in the N-th image frame and the image frame corrected by using the parameters D_C·81 and D_B·82 in accordance with the formula D_C·x+ D_B. Furthermore, the coded area 71 is composed of block areas 72 and 73 containing coded data for each block of 16×16 pixels shown in FIG. 5. When there are B₁-B_nblocks of 16×16 pixels as shown in FIG. 5, the coded area 71 is composed of n block areas as shown in FIG. 6.
Each block area is composed of a movement vector V[0074] _ifor each block, and four areas C_i-1-C_i-4for storing coded data of each of four sub-blocks obtained by dividing each block into 8×8 pixels. Taking the first block area 72 in FIG. 6 as an example, reference numeral 91 corresponds to the movement vector and the reference numerals 92-95 correspond to data coded for each sub-block.
Here, the relationship between the coded data of FIG. 6 and the outputs from the moving image coding apparatus of FIG. 1 is such that the parameters D[0075] _C-81 and D_B-82 correspond to the output values from the output terminal 8 of the frequency calculation section 6 of FIG. 1. Additionally, the movement vector of each block area (e.g. reference numeral 91) corresponds to the output from the output terminal 18 of the motion detection/compensation section 10 of FIG. 1, and the coded data of each sub-block (e.g. reference numerals 92-95) correspond to the output values from the output terminal 14 of the quantization section 13.
Additionally, in the case of the structure of coded data shown in FIG. 6, the output activity of the moving image coding apparatus shown in FIG. 1 is as follows. [0076]
First, the parameters D[0077] _C·81 and D_B·82 indicating the brightness change in the image frame overall are outputted.
Next, the movement vector V[0078] ₁·91 of the block B₁shown in FIG. 5 and the coded data C_1-1·92-C_1-4·95 for each sub-block SB_1-1-SB_1-4are outputted.
Then, the movement vector V[0079] ₁and the coded data C_1-1-C_1-4are sequentially outputted for the block B₁, and the same procedure is repeated for each block until the final block B_n.
This operation is performed for each image frame, and the coded data are recorded. [0080]
When decoding the coded data shown in FIG. 6, the parameters D[0081] _C·81 and D_B·82 of the parameter areas of FIG. 6 are inputted into the input terminal 23 of the moving image decoding apparatus, then the movement vector V_ifor each block area is inputted into the input terminal 22 and the coded data C_1-1-C_1-4of each sub-block is inputted into the input terminal 21, in order to decode for each image frame.
FIG. 7 is a diagram showing a coded data structure for recording the movement vector [0082] 55, coded data 56-59 of each sub-block and the parameters D_C·53 and D_B·54 of each block by the block areas indicated by the reference numerals 51 and 52. In the case of fade-in and fade-out, there are no large differences in the coding efficiency in sub-block units either when the parameters D_Cand D_Bare determined by the block or when the parameters D_Cand D_Bare determined uniquely for the entire image frame. For example, when the size of the image frame is 352×240 pixels (330 blocks of 16×16 pixels) and the parameters D_Cand D_Bare each eight bits long, then the coded data shown in FIG. 6 can be made less than the coded data shown in FIG. 7 by 5264 bits (= {8+ 8}×{330−1}) per image frame.
FIG. 8 is a diagram showing the structure of the coded data when only a single parameter D[0083] _C·61 representing the gain change for each image frame is recorded, and when the structures of the block areas as indicated by reference numerals 62 and 63 are identical to the case for FIG. 6. In this case, assuming the parameters D_Cand D_Bare eight bits long, the coded data of FIG. 8 is eight bits shorter per image frame than the coded data of FIG. 6. However, in the case of fade-ins and fade-outs, when the brightness variation changes according to the luminance values themselves as shown in FIG. 4, the approximation of the brightness variation is insufficient, even if the luminance values x of the pixels in the reference image plane are corrected by the parameter D_C. For this reason, the discrete cosine transformed difference values are relatively large, as a result of which the amount of coded data for each sub-block (e.g. reference numerals 64-67) increases, and the amount of coded data for a single image frame overall increases.
As explained above, the coding efficiency can be increased by configuring the coded data as shown in FIG. 6. [0084]
While examples of outputting D[0085] _Cand D_Bby units of image frames have been described in the above embodiments, the present invention is not limited thereby, and the units may correspond to the image frames; for example, the units may be video object planes (VOPs) in the standardized MPEG-4 (Moving Picture Experts Group—Phase 4) coding format for color video images. That is, the parameters D_Cand D_Bcan be determined between a reference VOP and a VOP being processed, and brightness variation compensation and coding/decoding can be performed using these parameters. In this case, the structure of the coded data will be such that the parameters D_Cand D_Bare recorded as pairs with the coded data of the VOPs which have been brightness variation compensated using these parameters, in the structure of the MPEG-4 coded data. That is, in the case of an MPEG-4 data structure, when standard data used as a header is included at the top of the frame, for example when all of the discrete cosine transforms in sub-block SB_1-1of FIG. 5 result in zero, then the data of FIG. 6 will be different from the coded data for each frame as shown in FIG. 6 in that the coded data C_1-1will be encoded with data signifying that “the data of that sub-block are all zero”. Although there will be differences such as this, there will be no difference with respect to FIG. 6 in that the parameters D_Cand D_Bof the parameter area 70 and the coded data areas (corresponding to the coded area 71 of FIG. 6) of the VOPs brightness variation transformed by using these parameters form pairs.
The MPEG standard coded data structure is described in detail in T. Ebrahimi, “MPEG-[0086] 4 video verification model: A video encoding/decoding algorithm based on content representation” Signal Processing: Image Communication, vol. 9, no. 4, pp. 367-384, May 1997.
Here, the layered structure of MPEG-[0087] 4 shall be explained briefly with respect to the drawings. That is, a video image is composed of a collection of one or more single shots referred to as video objects (hereinafter abbreviated to VO). FIG. 9 shows two shots referred to as VO₁and VO₂.
Each VO is composed of a collection of one or more collections of video image contents referred to as video object layers (hereinafter abbreviated to VOL). For example, in FIG. 9, VO[0088] ₁is composed of the two VOLs which are a snowman VOL₁₁and a house VOL₁₂. VO₂is composed of the three VOLs which are a square VOL₂₁, a triangle VOL₂₂and an X VOL₂₃.
Additionally, each VOL is composed of a collection of images referred to as video object planes wherein the contents change over time. For example VOL[0089] ₁₁is composed of a collection of the three VOPs VOP₁-VOP₃.
Each of these VOPs can be encoded using the above-mentioned parameters D[0090] _Cand D_B.
While a discrete cosine transform is performed when coding and decoding in the examples of the above embodiments, the present invention is not limited thereby, and it is possible to use other orthogonal transforms such as Hadamard transforms or wavelet transforms. [0091]
Additionally, while each block has 16×16 pixels and each sub-block has 8×8 pixels in the above-described embodiments as shown in FIG. 5, the present invention is not necessarily limited thereby. [0092]
Additionally, the coding or decoding of moving images can be performed by recording a program for executing the operations of the moving image coding apparatus of FIG. 1 or the moving image decoding apparatus of FIG. 2 onto a computer-readable recording medium, then reading the program recorded onto this recording medium by a computer, and executing the program. [0093]

Claims

What is claimed is:

1. A moving image brightness variation compensation method for encoding digital moving images for transmission and storage, and for image processing when editing moving images, said moving image brightness variation compensation method comprising:

correcting a luminance value x of each pixel according to the formula D_C·x+ D_B, wherein D_Bis a parameter indicating a gain change and D_Cis a parameter indicating a contrast change, the parameters representing overall luminance changes between a reference image plane and an image plane being processed, and being determined by using differences in luminance values between corresponding pixels in the reference image plane and the image plane being processed.

2. A moving image brightness variation compensation method in accordance with

claim 1

, wherein the parameter D_Bindicating a gain change and the parameter D_Cindicating a contrast change are determined by minimizing the sum of squared differences in luminance values between corresponding pixels in the reference image plane and the image plane being processed.

3. A moving image brightness variation compensation method in accordance with

claim 1

, wherein the parameters D_Band D_Cfor the entire image plane being processed are determined by using the two parameters D_Cand D_Bfor each of n blocks, n being an integer of at least 2, composed of K×L pixels, K and L being natural numbers, in the image plane being processed.

4. A moving image brightness variation compensation method in accordance with

claim 3

, wherein the parameters D_Band D_Cfor the entire image plane being processed are determined by taking the mean values of the parameters D_Cand D_Bfor each of n blocks as the parameters D_Cand D_Bof the entire image plane being processed.

5. A moving image brightness variation compensation method in accordance with

claim 1

, wherein said image plane is an image frame or a video object plane.

6. A moving image coding apparatus, comprising:

parameter determining means for determining a parameter D_Bindicating a gain change and a parameter D_Cindicating a contrast change, the parameters representing overall luminance changes between a reference image plane and an image plane being processed, and being determined by using differences in luminance values between corresponding pixels in the reference image plane and the image plane being processed; and

brightness variation compensation means for correcting a luminance value x of each pixel in said image plane being processed using said parameters D_Band D_Caccording to the formula D_C·x+ D_B.

7. A moving image coding apparatus in accordance with

claim 6

, wherein said parameter determining means determines the parameters D_Band D_Cby minimizing the sum of squared differences in luminance values between corresponding pixels in the reference image plane and the image plane being processed.

8. A moving image coding apparatus in accordance with

claim 6

, wherein said parameter determining means determines the parameters D_Band D_Cfor the entire image plane using the two parameters D_Cand D_Bfor each of n blocks, n being an integer of at least 2, composed of K×L pixels, K and L being natural numbers, in the image plane being processed.

9. A moving image coding apparatus in accordance with

claim 8

, wherein said parameter determining means determines the parameters D_Band D_Cfor the entire image plane being processed by taking the mean values of the parameters D_Cand D_Bfor each of n blocks as the parameters D_Cand D_Bof the entire image plane being processed.

10. A moving image coding apparatus in accordance with

claim 6

, wherein said image plane is an image frame or a video object plane.

11. A moving image decoding apparatus, comprising:

a brightness variation compensation means for correcting a luminance value x of each pixel in a reconstructed image plane after decoding has been completed, using parameters D_Band D_Caccording to the formula D_C·x+ D_B, wherein:

the parameter D_Bindicates a gain change and the parameter D_Cindicates a contrast change, the parameters representing overall luminance changes between a reference image plane and an image plane being processed, and determined when an original image plane was being coded.

12. A moving image decoding apparatus, comprising:

a brightness variation compensation means for correcting a luminance value x of each pixel in a reconstructed image plane after decoding has been completed, using parameters D_Band D_Caccording to the formula D_C·x+ D_B, wherein the parameter D_Bindicates a gain change and the parameter D_Cindicates a contrast change, the parameters representing overall luminance changes between a reference image plane and an image plane being processed, and determined when an original image plane was being coded by taking the mean values of parameters D_Cand D_Bas the parameters D_Cand D_Bof the entire image plane being processed.

13. A moving image decoding apparatus in accordance with

claim 11

or

12

, wherein said image plane is an image frame or a video object plane.

14. A computer-readable recording medium recording a moving image coding program, said moving image coding program containing instructions for a computer to execute:

a parameter determining step of determining a parameter D_Bindicating a gain change and a parameter D_Cindicating a contrast change, the parameters representing overall luminance changes between a reference image plane and an image plane being processed, and being determined by using differences in luminance values between corresponding pixels in the reference image plane and the image plane being processed; and

a brightness variation compensation step of correcting a luminance value x of each pixel in said image plane being processed using said parameters D_Band D_Caccording to the formula D_C·x+ D_B.

15. A computer-readable recording medium recording a moving image coding program in accordance with

claim 14

, wherein said parameter determining step includes determining the parameters D_Band D_Cby minimizing the sum of squared differences in luminance values between corresponding pixels in the reference image plane and the image plane being processed.

16. A computer-readable recording medium recording a moving image coding program in accordance with

claim 14

, wherein said parameter determining step includes determining the parameters D_Band D_Cfor the entire image plane using the two parameters D_Cand D_Bfor each of n blocks, n being an integer of at least 2, composed of K×L pixels, K and L being natural numbers, in the image plane being processed.

17. A computer-readable recording medium recording a moving image coding program in accordance with

claim 16

, wherein said parameter determining step includes determining the parameters D_Band D_Cfor the entire image plane being processed by taking the mean values of the parameters D_Cand D_Bfor each of n blocks as the parameters D_Cand D_Bof the entire image plane being processed.

18. A computer-readable recording medium recording a moving image coding program in accordance with

claim 14

, wherein said image plane is an image frame or a video object plane.

19. A computer-readable recording medium recording a moving image decoding program, said moving image decoding program containing instructions for a computer to execute:

a brightness variation compensation step of correcting a luminance value x of each pixel in a reconstructed image plane after decoding has been completed, using parameters D_Band D_Caccording to the formula D_C·x+ D_B, wherein:

20. A computer-readable recording medium recording a moving image decoding program, said moving image decoding program containing instructions for a computer to execute:

a brightness variation compensation step of correcting a luminance value x of each pixel in a reconstructed image plane after decoding has been completed, using parameters D_Band D_Caccording to the formula D_C·x+ D_B, wherein the parameter D_Bindicates a gain change and the parameter D_Cindicates a contrast change, the parameters representing overall luminance changes between a reference image plane and an image plane being processed, and determined when an original image plane was being coded by taking the mean values of parameters D_Cand D_Bas the parameters D_Cand D_Bof the entire image plane being processed.

21. A computer-readable recording medium recording a moving image decoding program in accordance with

claim 19

or

20

, wherein said image plane is an image frame or a video object plane.

22. A computer-readable recording medium recording coded data of moving images, wherein said coded data includes a plurality of areas recording image plane coded data encoded by units of image planes, said areas recording image plane coded data comprising:

a parameter area recording parameters D_Bindicating gain changes and D_Cindicating contrast changes, the parameters representing overall luminance changes between reference image planes and image planes being processed, and being determined by using differences in luminance values between corresponding pixels in the reference image plane and the image plane being processed; and

a coded area recording data in units of image planes relating to differences in pixel values for corresponding pixels between said reference image planes and image planes wherein luminance values x of the pixels in said reference image planes have been corrected by using the parameters D_Band D_Caccording to the formula D_C·x+ D_B.

23. A computer-readable recording medium recording coded data of moving images in accordance with

claim 22

, wherein said image plane is an image frame or a video object plane.

24. A computer-readable recording medium recording coded data of moving images in accordance with

claim 22

, wherein the parameters D_Band D_Crecorded in the parameter area are determined by minimizing the sum of squared differences in luminance values between corresponding pixels in the reference image plane and the image plane being processed.

25. A computer-readable recording medium recording coded data of moving images in accordance with

claim 22

, wherein the parameters D_Band D_Crecorded in the parameter area are determined by using the two parameters D_Cand D_Bobtained from each of n blocks, n being an integer of at least 2, composed of K×L pixels, K and L being natural numbers, in the image plane being processed.

26. A computer-readable recording medium recording coded data of moving images in accordance with

claim 25

, wherein the parameters D_Band D_Crecorded in the parameter area are determined by taking the means value of the parameters D_Cand D_Bfor each of n blocks as the parameters D_Cand D_Bof the entire image plane being processed.