WO1999017254A1

WO1999017254A1 - Digital redeye removal

Info

Publication number: WO1999017254A1
Application number: PCT/US1998/019428
Authority: WO
Inventors: Ronald E. Boucher; John C. Debossu
Original assignee: Polaroid Corporation
Priority date: 1997-09-26
Filing date: 1998-09-17
Publication date: 1999-04-08

Abstract

A system and method for correcting redeye in a digital image includes the functions or steps, respectively, of: presetting threshold values of luminance, hue and saturation; selecting at least one pixel of the image for comparison with the threshold values; identifying redeye pixels of the image by comparing luminance, hue and saturation values which are derived from color components of the selected pixels with the threshold values; indentifying redeye regions of the image having one or more adjacent redeye pixels which are surrounded by non-redeye pixels; replacing the luminance, hue and saturation values of the identified redeye pixels with values derived from weighted functions of the color components subject to a first set of color weights; identifying edge pixels as the redeye pixels which border non-redeye pixels; and smoothing each redeye region by replacing the luminance, hue and saturation values of the edge pixels with the weighted functions of the color components subject to a second set of color weights.

Description

DIGITAL REDEYE REMOVAL

BACKGROUND OF THE INVENTION

In some color and black-and-white pictures taken by an electronic flash the eyes of a human

or an animal have an unnatural "redeye" effect which causes pupils to appear red rather than black, brown, blue, green or gray. It is generally accepted that the redeye effect is caused by light from

the electronic flash that is reflected from the retina of the subject's eye. Thus, light from the flash is imaged at the retina where a fraction of it is reflected back out the pupil of the eye. If the eye were

a perfect optical system, this light would be reflected entirely back into the flash source. However,

since the eye is not perfect, and since the original path of the light originating at the electronic flash is proximate to the lens axis, then some of the spreading reflected light from the retina overlaps the

lens axis and is sensed by the lens. Since the reflectivity of the retina is high in the red light portion of the visible light spectrum, the pupil spot effected by the reflected light is highly red. Although significant subject variations in susceptibility to redeye have been noted,-these can be explained by

variability in retinal-reflectaance, accommodation, pupil size and ocular optics of the subjects. The redeye effect can be avoided by locating the flash unit some distance (perhaps six

inches or more) from the lens axis, viewing the subject from an angle, having the subject look away from the camera, or raising the ambient light level so that the pupils will close to a significant

degree. Many other methods and systems have been utilized for minimizing redeye such as

reducing the size of the pupil by sending a pulsating light from the electronic flash just prior to taking the picture.

It is a primary object of the present invention to provide a system and method for redeye removal in a digital image. SUMMARY OF THE INVENTION

A system and method for correcting redeye in a digital image includes the functions or

steps, respectively, of: presetting threshold values of luminance, hue and saturation; selecting at

least one pixel of the image for comparison with the threshold values; identifying redeye pixels of the image by comparing luminance, hue and saturation values which are derived from color

components of the selected pixels with the threshold values; identifying redeye regions of the image having one or more adjacent redeye pixels which are surrounded by non-redeye pixels; replacing

the luminance, hue and saturation values of the identified redeye pixels with values derived from

weighted functions of the color components subject to a first set of color weights; identifying edge pixels as the redeye pixels which border non-redeye pixels; and smoothing each redeye region by

replacing the luminance, hue and saturation values of the edge pixels with the weighted functions of the color components subject to a second set of color weights.

BRIEF DESCRIPTION OF THE DRAWINGS

The aforementioned aspects and other features of the invention are described in detail in

conjunction with the accompanying drawings where:

Figure 1 is a diagram of an image acquisition and rendering system including an image

processing system constructed in accordance with the principles of the invention;

Figure 2 is a block diagram of a preferred sequence of steps of the inventive method; and

Figure 3 is a representation of a redeye region composed of redeye pixels. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

A digital image consists of pixels each containing color and luminance information which

can be represented in a variety of domains. For instance, the pixel information can be resident in RGB (red, green and blue) device-dependent color space having R, G and B color components for use with a color monitor, or in CMYK (cyan, magenta, yellow black) device-dependent color space having C, M, Y and K color components for use with a color printer. When transferring

information between such device-dependent color spaces, use is made of device-independent color

space such as, but not limited to, proprietary color spaces, CIE L*a*b*, CIE XYZ, YUV and YIQ.

In one embodiment of the redeye removal method, the properties of a pixel are represented in RGB color space and the pixel values are transformed into YIQ color space as defined in the following equations (1) through (5) where, for each pixel, Y represents luminance, I and Q represent chrominance, H represents hue and S represents saturation.

Y = 0.30R + 0.59G + 0.11B (1)

I = 0.5R - 0.23G - 0.27B (2) Q = 0.203R - 0.5G + 0.297B (3)

H = tan-¹(Q/I) (4)

S^ I' + Q²}^ (5)

The transformations of equations (1) through (5) are conventional as described in "Colour Space

In Television And Display Systems" by W. N. Sposon, Adam Hilger Ltd., Bristol (1983), pages 19 and 64.

Maximum and minimum threshold values of luminance, hue and saturation for defining a redeye region of a pixel are determined in block 30 of Figure 2 by testing a database of images

where: MaxLum represents the maximum luminance threshold value; MinLum represents the- minimum luminance threshold value; MaxHue represents the maximum hue threshold value; MinHue represents the minimum hue threshold value; MaxSat represents the maximum saturation

threshold value; and MinSat represents the minimum saturation threshold value. Threshold ranges having both upper and lower limits are necessary for preventing spurious redeye readings. These

threshold values, the calculation of which will be detailed further on, are preferably precalculated and prestored in a configuration file, database, look-up table or other memory structure prior to

initiation of the redeye removal method. Furthermore, the above threshold values may be varied if

desired by the user.

One or more pixels of the image are selected in block 32 for comparison against the

predetermined threshold values, MaxLum, MinLum, MaxHue, MinHue, MaxSat and MinSat. The

comparison starts with one seed pixel and continues in some predetermined pattern until the luminance, hue and saturation properties of all pixels for a given area (which could be a part, or the

whole, of an image) are compared against the predetermined threshold values. If each of the luminance, hue and saturation values of a pixel being tested is both greater than the respective

minimum threshold value and less than the respective maximum threshold value, then that pixel in

block 34 is identified and labeled as a redeye pixel, i.e. a pixel which exhibits the redeye effect. For

instance, a pixel at a spatial location (x,y) could be flagged or labeled as a redeye pixel by setting

the appropriate element in the array Flags(x,y) equal to 3 whereas a non-redeye pixel could be indicated by setting Flags(x,y) equal to 0.

Redeye regions are identified in block 36 as regions of the image having one or more adjacent redeye pixels which are surrounded by non-redeye pixels. After all redeye pixels and

redeye regions have been identified in block 36, then the original luminance, hue and saturation

values of those pixels are replaced in block 38 with values of luminance, hue and saturation derived from weighted functions of the color components. The weighted functions of the color components are determined in equations (7), (8) and (9), and the replacement luminance, hue and

saturation values are derived by inserting the weighted functions R', G' and B' into equations (1),

(4) and (5). The density of each pixel after insertion of replacement values is typically dark neutral

gray to closely resemble a pupil. Other Y, H and S replacement values corresponding to! different

shades of gray, or different eye colors such as blue, green, brown or gray could be preselected by

the user if desired.

In order to provide a more natural effect for the resulting pupils which have had redeye

removed, the edges of the redeye regions are smoothed or feathered so that the colors of the pupil

and the retina will gradually blend together. This is accomplished by first setting each redeye pixel of each redeye region to dark neutral gray as noted above, then blending the tones of edge pixels of

the redeye region from dark neutral gray to a lighter gray to account for flare and specular reflection.

The edge pixels are defined and identified in block 40 as redeye pixels in a redeye region

which border non-redeye pixels. Figure 3 illustrates an 8x8 block 50 of redeye pixels, i.e. a redeye region 50, where each pixel 1 is an edge pixel. Thus feathering or smoothing of redeye region 50 in

the simplest form amounts to setting all pixels 1, 2, 3 and 4 to dark neutral gray, and then replacing the edge pixels 1 with an intermediate gray value. However, the extent of feathering can be

extended beyond-one layer of edge pixels. This is accomplished in block 42 by flagging layers of

edge pixels. Each pixel is tested to determine if it should be designated as a redeye pixel, and if so, then the particular tone or level of gray which should be assigned to each layer of pixels is

determined by the amount of feathering desired. For inst-ance in Figure 3, the redeye region 50 is surrounded by non-redeye pixels (not shown) at locations (x,y) which are flagged as Flags(x,y) = 0. All redeye pixels 1, 2, 3 and 4 are initially flagged as Flags(x,y) = 3 corresponding to pixels to be replaced with dark neutral gray. Thereafter, pixels 3 and 4 remain flagged as Flags(x,y) = 3 corresponding to dark neutral gray for producing the most correction, pixels 2 are flagged as

Flags(x,y) = 2 corresponding to an intermediate gray for producing medium correction, and pixels 1 are flagged as Flags(x,y) = 1 corresponding to a light gray for producing a small correction. Each

set of pixels designated by a different Flags value will have its original luminance, hue and

saturation values replaced. This feathering, as exemplified by the following pseudocode, produces a more natural, gradual blend between the colors of the pupil and the retina.

for each x,y in oldBorder for xl ,y 1 = each neighbor pixel of x,y and Flags(xl,yl) = 0 add xl,yl to newBorder setFlags(xl,yl) = 2 swap oldBorder and newBorder pointers for each x,y in oldBorder for xl,yl = each neighbor pixel of x,y and Flags(xl,yl) = 0 set Flags(xl,yl) = 1

Equations (6) through (9) calculate weighted functions R', G', and B' from the R, G and B

color components which, in turn, are inserted into equations (1) through (5) to derive replacement

luminance, hue and saturation values for any pixel which is identified as a redeye pixel.

C = min(R, G, B) (6)

R' = {C*n + R*(4-n)}/4 (7)

G' = {C*n + G*(4-n)}/4 (8)

B' = {C*n + B*(4-n)}/4 (9)

where: R', G' and B' are the weighted functions of the color components corresponding to

feathered colorant values which replace the original values R, G and B; and n is a variable defined as the color weighting value so that n = 4 for Flags = 3, n = 2 for Flags = 2, and n = 1 for Flags = 1. The red color component R, the blue color component B and the green color component G are compared with one another and the numerical minimum value of the three color components is set to C, which is then applied to equations (7), (8) and (9).

Three modes are discussed for locating or selecting pbcels in the image for comparison with the threshold values. They are the unassisted mode, the partially assisted mode and the fully

assisted mode.

The predetermined threshold values include a maximum luminance threshold value

MaxLum, a minimum luminance threshold value MinLum, a maximum hue threshold value

MaxHue, a minimum hue threshold value MinHue, a maximum saturation threshold value MaxSat, and a minimum saturation threshold value MinSat. In the unassisted mode, for a given pixel having

a luminance Y, a hue H and a saturation S to be designated as a redeye pixel, the conditions of equations (10), (11) and (12) must be met.

MinLum < Y < MaxLum (10)

^" MinHue ≤ H ≤ MaxHue (11)

MinSat < S < MaxSat (12)

In other words, the predetermined threshold values establish the redeye range for each of the designated pixel properties.

Some of the pixels identified as potential redeye pixels could actually be non-redeye pixels

which exhibit the same over-threshold attributes but which are located in parts of the image away

from the pupils. For instance, a bright red shirt could possibly result in erroneous redeye pixel findings. In order to minimize the occurrence of this type of error, additional threshold values are applied on the size and shape of the potential redeye regions. Since a pupil has an aspect ratio of approximately 1:1, then any potential redeye region having an aspect ratio greater than, say, 2: 1 or 1 :2 would be dismissed as a non-redeye region. Although the unassisted mode identifies all potential redeye pixels, it is relatively slow, computationally extensive and prone to false redeye

pixel identification.

In the fully assisted mode, the coordinates of the seed pixels of each region are supplied by

the user via a data structure input to the algorithm. For instance, the user will move the cursor to a

redeye region in the image, then click the cursor on the redeye region with the mouse. The predetermined threshold values for the fiilly assisted mode are represented as:

MinLum = Y - DeltaLumM (13)

MaxLum = Y + DeltaLumP (14) MinHue = H - DeltaHueM (15)

MaxHue = H + DeltaHueP ( 16)

MinSat = S - DeltaSatM ( 17)

MaxSat = S + DeltaSatP (18)

where as before Y, H and S are, respectively, the luminance, hue and saturation values of the

selected-pixels. The values DeltaLumM, DeltaLumP, DeltaHueM DeltaHueP, DeltaSatM and

DeltaSatP are preselected by testing images from a database.

After the threshold values have been preset as previously described, then a number of points are selected by the user as seed pixels for suspected redeye regions. Although the maximum

number of selected points is theoretically limited by the number of pixels in the image, the practical limitation is in the order of 10 to 20. As the number of selected points increases, the computational

time and processing requirements also increase, so it is preferred to select as few points as possible

for identification of redeye. For instance, in an image having two people with redeye, a minimum number of 4 points is preferably selected, one for each eye. MISSING UPON TIME OF PUBLICATION

AUSENTE EN EL MOMENTO DE LA PUBLICACION

MANQUANT AU MOMENT DE LA PUBLICATION

In the partially assisted mode, the user clicks the mouse cursor near the corner of an eye then pulls the cursor away from that point to create a rectangle around the area of interest where the user identifies redeye. Upon selection by the user of a redeye removal button on the screen, the

method as described herein automatically identifies and replaces any luminance, hue and saturation values of redeye pixels within the selected rectangular region. However, for the partially assisted mode the saturation criterion is augmented by a function based on the color of the pixel. The seed

pixel is determined as the pixel within the selected rectangle which maximizes the expression (19):

S + HueFactor * (HueRedMax - abs(H - HueRed))/HueRedMax (19)

Only the pixels located within the user-specified rectangle are tested for redeye, so computational efficiency and processing times are minimized. The above steps can be repeated as desired for additional redeye regions with the image.

Figure 1 is a diagram of an image acquisition and rendering system 26 built in

accordance with the principles of the invention. The system 26 includes a computer 14, one or more image acquisition devices such as the camera 12 and the scanner 10 which are useful for

acquiring an image and converting it to digital image data, and one or more image rendering devices such as the monitor 16 and the printer 20 which are useful for generating an output

image.

The system 26 includes conventional operator interface elements such as the keyboard

18, the mouse 22, the monitor 16 and a compact or floppy disk 24. During operation of the system 26, the computer 14 processes certain programs which enable selected image processing operations such as the redeye removal algorithm described hereinbefore. The

system 26 may include a variety of components in addition to or instead of those depicted in Figure 1. For instance, the various elements of the system 26 may be replaced by special-purpose hardware, firmware or any combination thereof. Also, those skilled in the art understand that many, if not all, of the processing capabilities of the computer 14 can alternatively be performed in intelligent system components, such as the camera 12, the scanner 10, the monitor 16, the keyboard 18 and the printer 20. The redeye removal algorithm, as described above, is loaded into the computer 14, for instance, via a CDROM 24 and then run by the user who interacts with the computer 14 by way of the keyboard 18 and the monitor 16. First, an image is acquired, for instance, through an image acquisition device 10 or 12, or from a file stored within the computer 14. Means for selecting one or more pixels for comparison with the threshold values is defined as the computer 14 when the algorithm is operating in the unassisted mode, or as the mouse 22 when the algorithm is operating in either the assisted and partially assisted modes. The computer 14 then performs the functions of identifying redeye pixels, identifying redeye regions, replacing redeye pixel values, identifying edge pixels and the smoothing.

The above embodiments of the present invention are merely illustrative of the many possible specific embodiments and variations of the invention as claimed. Many other equivalents of the claimed devices and methods are within the purview of the claims as understood by those of ordinary skill in the art.

Claims

CLAIMS:

1. A method for correcting redeye in a digital image, comprising the steps of: presetting threshold values of luminance, hue and saturation; selecting at least one pbcel of the image for comparison with said threshold values; identifying redeye pixels of the image by comparing luminance, hue and saturation values which are derived from color components of said selected pixels with said threshold values;

identifying redeye regions of the image having one or more adjacent redeye pixels which

are surrounded by non-redeye pixels; replacing the luminance, hue and saturation values of the identified redeye pixels with values derived from weighted functions of the color components;

identifying edge pixels as said redeye pixels which border non-redeye pixels; and smoothing each said redeye region by replacing the luminance, hue and saturation values of

the edge pixels with the weighted functions of the color components subject to a second set of color weights.

2. The method of claim 1 wherein said threshold values of luminance, hue and saturation are determined by testing on a database of images.

3. A system for correcting redeye in a digital image, comprising: means for selecting at least one pixel of the image for comparison with preset threshold values of luminance, hue and saturation; means for identifying redeye pixels of the image by comparing luminance, hue and saturation values which are derived from color components of said selected pixels with said threshold values; means for identifying redeye regions of the image having one or more adjacent redeye pixels which are surrounded by non-redeye pixels; means for replacing the luminance, hue and saturation values of the identified redeye pixels

with values derived from weighted functions of the color components subject to a first set of color weights;

means for identifying edge pixels as said redeye pixels which border non-redeye pixels; and means for smoothing each said redeye region by replacing the luminance, hue and

saturation values of the edge pixels with the weighted functions of the color components subject to a second set of color weights.

4. The system of claim 3 wherein said threshold values of luminance, hue and saturation are determined by testing on a database of images.