WO2013124664A1

WO2013124664A1 - A method and apparatus for imaging through a time-varying inhomogeneous medium

Info

Publication number: WO2013124664A1
Application number: PCT/GB2013/050428
Authority: WO
Inventors: Antony Joseph Frank Lowe
Original assignee: Mbda Uk Limited
Priority date: 2012-02-22
Filing date: 2013-02-22
Publication date: 2013-08-29

Abstract

An apparatus comprises a coded-aperture mask configured to form a coded image, an imager arrange to record the coded image, a decoder configured to decode the coded image using a plurality of decoding masks, and an image processor. The decoder is configured to form a plurality of different sets of first, second and third decoded images using a plurality of different sets of first, second and third decoding masks. The image processor identifies focused portions of the decoded images, and forms a composite focused image from the identified focused portions of the second decoded images.

Description

A method and apparatus for imaging through a time-varying inhomogeneous medium

Field of the Invention

The invention relates to the field of imaging through a time-varying inhomogeneous medium, for example the atmosphere.

Background of the Invention

An attempt to obtain a clear image of an object through a medium such as the atmosphere can be hampered by the effects of localised variations in the refractive index of the medium, for example resulting from turbulence. Such localised inhomogeneities create minor distortions in the wavefront of the light reaching the imager, which in turn results in degradation of the image of the object. The degradation typically manifests itself as a distortion of the image (i.e. the separation between equally spaced points in the object varies between pairs of points in the image) and as a defocusing or blurring (i.e. infinitesimal points in the object have a finite extent in the image). The distortion and blurring changes rapidly (for example, for turbulent atmosphere, over the course of a few milliseconds). Temporally varying blurring and distortion effects in the captured image limit the maximum magnification which can be used, and thus the maximum range at which a given resolution can be achieved. Those effects can be particularly strong in littoral regions and in hot environments.

A known technique for imaging through the atmosphere is known as "lucky imaging" or "lucky sub-frame selection". This technique relies on the fact that the random variations in blurring will very occasionally and briefly happen to result in an image (also referred to herein as a "frame") having a portion that is in focus (a "lucky sub-frame"). These moments of clarity will occur at different times for different portions of the image. The technique involves obtaining a sequence of images over a period of time that is sufficiently long for substantially all of the imaged object to have been in focus at some moment, and then forming a composite in-focus image of the object by mosaicing together the in-focus portions or sub-frames from the sequence of images or frames so obtained.

In order to do that, it is necessary to identify any subframes that are in focus within a given frame. Various techniques for doing that are known. Typically, those techniques involve calculation of a "sharpness metric" for the image, i.e. an assessment of how sharp the image is, with greater sharpness taken to indicate a better-focused sub-frame. An automated system can asses an area of an image to determine whether or not the area is in sharp focus (examples of such methods are used in cameras having an autofocus capability). Many different sharpness metrics are known, including use of, for example, the maximum spatial frequency having an amplitude above a certain threshold, a gradient within the image, wavelets, entropy, eigenvalue sharpness, bicoherence, or Chebyshev moments. There is a trade-off between the performance of different ones of those metrics, and the complexity of the calculations they involve (and hence the time and/or computing power required to carry them out). Such methods can have significant drawbacks, such as being misled by image noise.

S C Woods, P J Kent and J G Burnett, in "Lucky Imaging Using Phase Diversity Image Quality Metric", Paper B3, 6^th EMRS DTC Technical

Conference, Edinburgh 2009, describe a metric that relies upon an insight into the optics of the problem to reduce the computations required. The insight is the observation that, when a subframe is in focus at an imaging plane in a detector, the subframe will be out of focus by an equal amount at equal distances in front of and behind that plane. In contrast, when the subframe is not in focus at the imaging plane, the subframe will be differently defocused at equal distances in front of and behind that plane, because it will be more in focus on whichever side of the plane is closer to the point at which it is actually in focus (so, if the subframe is actually in focus some way behind the imaging plane, it will be more in focus at a distance behind the imaging plane than at the same distance in front of the imaging plane). Thus, in the method described by Woods et al., incoming light from the object is split into three using a quadratically distorted grating in the form of an off-axis Fesnel zone plate (described in more detail in PowerPoint presentation "Lucky sub-frame selection using phase diversity", 7 July 2009, QINETIQ/AT/PS/PUB0900701 , by the same authors), which results in spatially separated +1 , zero, and -1 orders that are converging, collimated and diverging, respectively. The zero order is focused at the imaging plane of a CCD array, whereas the +1 and -1 orders are defocused by equal amounts (in opposite directions, i.e.

converging and diverging) at the CCD array. A metric is then calculated based on the difference between the CCD image of the +1 order and the CCD image of the -1 order, with sub-frames that score well (i.e. a small difference) being selected as "lucky" subframes.

The Woods et al. device also addresses the problem of distortion. In this aspect of their disclosure, the zero-order images are summed to provide a low-resolution but undistorted (i.e. geometrically correct) reference image. The transformation required to map each individual zero-order image onto the reference image is calculated, and used to remove distortion from the +1 and- 1 order images prior to calculation of the metric. The "lucky sub-frame" is then selected from the transformed zero-order image, according to the metric, as just described.

However, the approach of Woods et al. has several significant drawbacks which include the following:

• the optics described uses potentially expensive components, including lenses and a diffraction grating;

• because of the diffraction grating, it suffers from chromatic

aberration;

• the image is split into three parts which are projected onto three different regions of the detector, thus reducing the overall field of view (FOV) of the detector by a factor of 3;

• differences in response (detector non-uniformity) between the three sets of pixels can lead to errors in the formation of the clear image; • to build up enough clear patches to cover the whole FOV requires many frames and therefore is slow;

• because the generation of a clear image relies on the serendipitous spread of clear patches to cover the whole FOV, the time taken to generate an optimal clear frame is variable and unpredictable; and

• because the distortion-removal process is performed on a

composite frame generated over quite a long period compared to the rate at which the atmospheric changes occur, the clear sections are essentially discontinuous, which limits the degree to which distortion can be removed from the image.

US2008/0259176 A1 describes an image pickup apparatus in which multiple images are transformed so that positions of corresponding points coincide between the images. The images are composited with the

corresponding points matched. An all-in-focus image or blur-emphasized image is said to be obtained even if there is camera shake or subject movement. However, use of multiple images is vulnerable to phenomena that vary with time, such as atmospheric shimmer. There is therefore an inherent degradation of the raw data. The method of US2008/0259176 A1 also requires complicated optics and moving stages.

The present invention seeks to mitigate the above-mentioned problems. Alternatively or additionally, the present invention seeks to provide an improved imaging apparatus. Alternatively or additionally, the present invention seeks to provide an improved method of imaging.

Summary of the Invention

The invention provides, in a first aspect, a method of imaging an object through a time-varying inhomogeneous medium, the method comprising:

(i) imaging the object through a coded-aperture mask to form a coded image at an imaging plane; (ii) decoding the coded image using a first decoding mask to form a first decoded image corresponding to a first separation of the coded-aperture mask and the imaging plane;

(iii) decoding the coded image using a second decoding mask to form a second decoded image corresponding to a second separation of the coded-aperture mask and the imaging plane, the second separation differing from the first separation by an increment distance;

(iv) decoding the coded image using a third decoding mask to form a third decoded image corresponding to a third separation of the coded-aperture mask and the imaging plane;

(v) identifying focused portions in a plurality of the decoded images; and

(vi) forming a composite focused image from the identified focused portions of the decoded images.

It may be that the decoding mask is the pattern of the coded aperture mask projected onto a detector. It will be understood that the decoding mask is used to deconvolve the focused image from the initially captured image.

It may be that the imaged object is itself an image of the object. It may be that the first separation of the coded-aperture mask and the imaging plane is calculated so as to provide a focused image at a range R. It may be that the second separation of the coded-aperture mask and the imaging plane is a second focused range such that the conjugate image distance from the coded aperture corresponding to that object range differs from the first separation by an increment distance. (Note that the actual distance between the coded-aperture mask and the imaging plane has not changed; rather the change is in the range used to calculate the decoding pattern.) It may be that the third separation differs from the second separation by the increment distance, so that the second separation is midway between the first separation and the third separation. It may be that the step of identifying focused portions in a plurality of the decoded images comprises the step of

(a) identifying any corresponding portions of, on the one hand, the first decoded image and, on the other hand, the third decoded image, in which the object is defocused by the same amount, and thereby identifying a focused portion of the second decoded image that corresponds to the identified corresponding portions of the first decoded image and the third decoded image.

It may be that steps (ii), (iii), (iv) and (a) are repeated using a different set of first, second and third decoding masks, thereby identifying further focused portions of further second decoded images. It may be that the composite focused image is formed from the identified focused portions of the second decoded images.

The invention provides, in a second aspect, a method of imaging an object through a time-varying inhomogeneous medium, the method

comprising:

(i) imaging the object through a coded-aperture mask to form a coded image at an imaging plane;

(ii) decoding the coded image using a first decoding mask to form a first decoded image corresponding to a first separation of the coded-aperture mask and the imaging plane;

(iii) decoding the coded image using a second decoding mask to form a second decoded image corresponding to a second separation of the coded-aperture mask and the imaging plane, the second separation differing from the first separation by an increment distance; (iv) decoding the coded image using a third decoding mask to form a third decoded image corresponding to a third separation of the coded-aperture mask and the imaging plane, the third separation differing from the second separation by the increment distance, so that the second separation is midway between the first separation and the third separation;

(v) identifying any corresponding portions of, on the one hand, the first decoded image and, on the other hand, the third decoded image, in which the object is defocused by the same amount; (vi) thereby identifying a focused portion of the second decoded image that corresponds to the identified corresponding portions of the first decoded image and the third decoded image;

(vii) repeating steps (ii) to (vi) using a different set of first, second and third decoding masks and thereby identifying further focused portions of the second decoded images;

(viii) forming a composite focused image from the identified focused portions of the second decoded images.

When atmospheric non-uniformities distort a scene, the effect is the equivalent to causing different parts of the scene to be focused at different distances from the aperture. Given that in a conventional camera, for any single image, the focal plane is a fixed distance from the aperture, this means that, to use "lucky sub-frame selection" to generate a composite clear image, one needs to wait until each part of the image happens to have been in focus on the focal plane. If, instead, one could somehow freeze the atmosphere and then move the focal plane at will, each individual part of the scene could be brought into focus at some focal plane position. These clear sections can then be stitched together to create an overall clear image of the scene.

Coded aperture imaging (CAI) effectively makes this possible. Coded aperture imaging (CAI) is a known process in which an aperture consisting of a pattern of transparent and opaque areas is used (either alone or in conjunction with a conventional lens) to project an image onto a detector. The image is somewhat analogous to a hologram of the imaged scene, in that it simultaneously contains within it information about the clear image of the scene at all ranges, compressed into one layer. An image of the scene that is focused to any single chosen range can be extracted from this "hologram" image by Fourier deconvolution of the aperture function as projected onto the imager focal plane at the focal length which would render the image at that range in focus. Because a single image captured with CAI can be used to generate images which appear to be focused at any chosen focal length, it can also be used to generate images at all focal lengths. That means that a single CAI image does effectively freeze the atmosphere and allows the possibility of finding the focal lengths at which each individual section of the image is in focus. Thus, it will be understood that, as used herein, a decoded image "corresponding to" a separation of the coded-aperture mask and the imaging plane means a decoded image that would be produced if the coded- aperture mask were moved to that separation, but in fact the coded-aperture mask remains stationary and the "moving" is effected by changing the decoding mask, e.g. by scaling it (as discussed further below). Thus one no longer needs to wait until each part of the image happens to be in focus at the actual focal plane.

Thus, it may be that the composite focused image is generated from one coded image.

The method may comprise forming the decoded images corresponding to a range of separations of the coded aperture mask and the imaging plane that is sufficiently wide that the whole of the object is imaged in focus in the composite focused image.

The object being imaged may for example be a discrete object, e.g. a ship, or a collection of objections, e.g. a scene. It may be that the varying inhomogeneous medium is the atmosphere. It may be that the varying inhomogeneous medium is seawater, e.g. the sea or ocean.

It may be that the first and third decoded images are formed before the second decoded image. It may be that only the portions of the second decoded image that correspond to identified corresponding portions of the first decoded image and the third decoded image are decoded from the coded image. That approach avoids the need to generate parts of the second decoded image that are not identified as being in focus.

It may be that the steps of identifying any corresponding portions of, on the one hand, the first decoded image and, on the other hand, the third decoded image in which the object is defocused by the same amount comprises identifying portions of the first decoded image and the third decoded image that are identical to each other.

The method may include recoding the coded image at the imaging plane using a plane array. The plane array may be a 2D array of

photoreceptors. The plane array may be a charge-coupled device (CCD) array.

The steps of decoding the coded images may comprise deconvolving the Fourier transform of the coded-aperture mask from the coded images.

The coded aperture mask may comprise a pattern of areas forming at least two sets of different transmissivities. It may be that the pattern of areas consists of a first set of opaque areas and a second set of transparent areas; in that case, the coded aperture mask may comprise a pattern of apertures.

The steps of decoding the coded image using each decoding mask to form each decoded image corresponding to each separation of the coded- aperture mask and the imaging plane may include the step of scaling the pattern of areas of the coded-aperture mask according to each said

separation. The steps of decoding the coded image using each decoding mask to form each decoded image corresponding to each separation of the coded-aperture mask and the imaging plane may include the step of deriving a ranged image focused at a range R. The pattern scaling factor may be f/fO where fO is a fixed separation of the coded-aperture mask from the imaging plane and 1/f + 1/R = 1/f0 (which is equivalent in a conventional system to changing the distance between the aperture and the detector from fO to f).

A series of the second decoded images may be formed. The images in the series may be derived at different values of f separated by amounts, preferably small amounts, the values of which are related to the effective depth of focus of the system at that value of f. The amounts may be half the difference in values of f equivalent in values of f to the depth of field at that value of f. A series of the first decoded images and a series of the third decoded images may also be formed by using two other series of values of f(fa and fb) to generate the decoded images, which may have the same f spacing as the first series, but with a small positive and negative offset respectively, i.e. (fa - f) = (f - fb) « f.

The step of identifying any corresponding portions of, on the one hand, the first decoded image and, on the other hand, the third decoded image in which the object is defocused by the same amount may comprise comparing a phase diversity metric of the first decoded image and the third decoded image. An example phase diversity metric corresponds to that given by Woods et al., i.e.

where, here, Z_kj is the value of the ith pixel in the kth sub-frame of the second decoded image, P_{k i} is the value of the ith pixel in the kth sub-frame of the first decoded image, and M_k,i is the value of the ith pixel in the kth sub- frame of the third decoded image. Q_k varies between 0 (poor) and 1 (good) image quality.

The step of forming the composite focused image may comprise the step of scaling the identified focused portions of the decoded images to take into account changes in projected size and/or position of the portions due to the different separations of the coded aperture mask and the imaging plane for which those focused portions are obtained.

Furthermore, because the clear but distorted image may be derived from a single frame, the distortion effects across the clear image may be essentially coherent and smoothly changing. This makes them potentially much more amenable to recovery by the de-distortion process than composite images created over many frames.

The method may further comprise the step of removing distortion arising from the time-varying inhomogeneous medium from the composite focused image, to form an undistorted, focused image. The step of removing distortion may comprise the steps of:

(a) imaging the object through the coded-aperture mask a plurality of times, to form a plurality of the coded images at the imaging plane; (b) forming a cumulative image frame from the plurality of the

coded images, or carrying out steps (ii) to (viii) on each coded image and forming a cumulative image frame from the resulting plurality of composite focused images;

(c) using the cumulative image frame as a reference frame by

comparing each of the plurality of images used to form the cumulative image with the cumulative image itself to determine a vector distortion field for each of the plurality of images; and

(d) using the vector distortion field for each of the plurality of images to remove the distortion from that respective image. The cumulative image may be a mean image, i.e. an average of the plurality of images used to form the cumulative image.

It may be that, after a predetermined number of images have been used to form the cumulative image frame, an oldest image of the plurality is removed from the cumulative image frame when the next image is added. Thus, the cumulative image frame may be formed from a constant number of successive ones of the plurality of images, up to and including a most-recent image. The constant number may be chosen to provide sufficient images for the cumulative image to be acceptably stable.

The method may further comprise the step of deconvolving the undistorted, focused image from at least one of the decoded images to provide a measure of the distortion and/or blurring caused by the time-varying inhomogeneous medium when the coded image was formed.

The coded-aperture mask may be a programmable mask, for example a spatial light modulator. The method may include the step of determining an optimal available coded-aperture pattern for the measured distortion and/or blurring, and applying that pattern to the programmable mask.

It may be that the focused portions are identified in the plurality of decoded images by identifying a surface of best focus.

It may be that the focused portions are identified in the plurality of decoded images by identifying corresponding portions in each decoded image, measuring the intensity of each corresponding portion, calculating the mean of the measured intensities of all corresponding portions and identifying as the focused portions the portions having an intensity having the greatest absolute deviation from the calculated mean intensity.

The invention provides, in a third aspect, an apparatus for imaging an object through a time-varying inhomogeneous medium, the apparatus comprising:

(i) a coded-aperture mask configured to form a coded image at an imaging plane; ager arranged to record the coded image;

(iii) a decoder configured to decode the coded image using a

plurality of decoding masks, said plurality comprising: a a first decoding mask configured to form a first decoded image corresponding to a first separation of the coded- aperture mask and the imaging plane;

b a second decoding mask configured to form a second

decoded image corresponding to a second separation of the coded-aperture mask and the imaging plane, the second separation differing from the first separation by an increment distance;

c. a third decoding mask configured to form a third decoded image corresponding to a third separation of the coded- aperture mask and the imaging plane;

(iv) an image processor configured to: a. identify focused portions in a plurality of the decoded images; and

b. form a composite focused image from the identified focused portions of the decoded images.

The coded image is a single coded image. The decoder is configured to decode the single coded image into a plurality of images focused at a plurality of ranges.

It may be that the third separation differs from the second separation by the increment distance, so that the second separation is midway between the first separation and the third separation. It may be that the decoder is configured to form a plurality of different sets of the first, second and third decoded images using a plurality of different sets of first, second and third decoding masks. It may be that the image processor is configured to:

a. for each set of first, second and third decoded images,

identify any corresponding portions of, on the one hand, the first decoded image and, on the other hand, the third decoded image in which the object is defocused by the same amount; and

b thereby identify a focused portion of the second decoded image that corresponds to the identified corresponding portions of the first decoded image and the third decoded image;

C. repeat steps b and c for the plurality of different sets of the first, second and third decoded images and thereby identify further focused portions of the second decoded images. It may be that the composite focused image is formed from the identified focused portions of the second decoded images.

The invention also provides, in a fourth aspect, an apparatus for imaging an object through a time-varying inhomogeneous medium, the apparatus comprising:

(i) a coded-aperture mask configured to form a coded image at an imaging plane;

(ii) an imager arranged to record the coded image;

(iii) a decoder configured to decode the coded image using a

plurality of decoding masks, said plurality comprising: a. a first decoding mask configured to form a first decoded

image corresponding to a first separation of the coded- aperture mask and the imaging plane;

b. a second decoding mask configured to form a second

c. a third decoding mask configured to form a third decoded image corresponding to a third separation of the coded- aperture mask and the imaging plane, the third separation differing from the second separation by the increment distance, so that the second separation is midway between the first separation and the third separation;

the decoder being configured to form a plurality of different sets of the first, second and third decoded images using a plurality of different sets of first, second and third decoding masks;

(iv) an image processor configured to: a. for each set of first, second and third decoded images,

b. thereby identify a focused portion of the second decoded image that corresponds to the identified corresponding portions of the first decoded image and the third decoded image;

c. repeat steps a and b for the plurality of different sets of the first, second and third decoded images and thereby identify further focused portions of the second decoded images; and d. form a composite focused image from the identified focused portions of the second decoded images.

The image processor may include the decoder.

The apparatus may be or form part of a passive optical system. The passive optical system may be, for example, a digital camera, binoculars, telescopic sights, an airborne sensor platform, a missile sensor system, or a satellite imaging system.

The apparatus may further comprise an active optical element, for example an adaptive optical element, for example an adaptive mirror. The active optical element may be a spatial light modulator, which may also be used to provide the encoding mask. The image processor may be configured to generate an undistorted focused image from the composite focused image and to deconvolve the undistorted focused image from at least one of the decoded images to provide a measure of the distortion and/or defocusing caused by the time-varying inhomogeneous medium when the coded image was formed. The image processor may be configured to provide to the active optical element a signal indicative of the measured distortion and/or defocusing; thus, the active optical element may be configured to compensate for the distorting and/or defocusing effects, preferably before the

inhomogeneous medium has changed significantly. The apparatus may for example form part of a laser-focusing system, for example in a communications system, and the adaptive mirror may be used to direct and/or to focus the laser.

It will of course be appreciated that features described in relation to one aspect of the present invention may be incorporated into other aspects of the present invention. For example, an apparatus of the invention may

incorporate any of the features described with reference to a method of the invention and vice versa.

Description of the Drawings

Embodiments of the present invention will now be described by way of example only with reference to the accompanying schematic drawings of which:

Figure 1 is a schematic view of an apparatus according to a first example embodiment of the invention;

Figure 2 is a schematic side view of an apparatus according to a second embodiment of the invention;

Figure 3 is a schematic illustration of the change in a projected mask deconvolution pattern between different chosen focus planes;

Figure 4 is a further illustration of the patterns of Fig. 3;

Figure 5 is a plot of the variation for a first example pixel of the intensity with distance from best focus, along the optic axis of the apparatus of Fig. 2; and

Figure 6 is a plot of the variation for a second example pixel of the intensity with distance from best focus, along the optic axis of the apparatus of Fig. 2

Detailed Description

In a first example embodiment, a coded aperture 10 is used to project an image A, separated by a fixed distance fO from the aperture, onto a focal plane array detector FPA, which is a 2D array of photoreceptors, the whole output of which is captured in a short period of time (a single frame).

The effect of the atmosphere is to defocus patches of the image such that the whole image no longer appears to come from a single range.

However, the "focusing" effect of the system can be selected by appropriately scaling the aperture pattern (which is deconvolved from the image A) to derive a ranged image Bf focused at range R such that the pattern scaling factor is f/fO where 1/f + 1/R = 1/f0. This is equivalent in a conventional system to changing the distance between the aperture 10 and the detector FPA from fO to f .

In the proposed system, a series of deconvolved images are derived at different f values separated by small amounts whose values are related to the effective depth of focus of the system at that f value. In this example, the step-size is given by half the f difference equivalent to the depth of field at that f value. In addition, two other series of f values (fa and fb) are also used to generate deconvolved images. These are spaced the same as the first series, but with a small positive and negative offset respectively, i.e. (fa - f) = (f - fb) «f.

For the image generated, in the process above, at each value of f, the images at fa and fb are compared, using the phase diversity metric, to determine which if any areas in the f-sequence images, are in focus at that f value. If the range of f values is wide enough, each point in the image will be at a best focus at some value of f.

These best-focus image sections are then stitched together digitally (and suitably scaled for changes in projected size and position due to changes in f value) into a single clear image for that frame.

This process is repeated on each frame. Each frame's data (up to a total of n frames) will also be added to a mean image frame. This mean image frame will eventually generate a blurred but stable image of the target scene. This is then used as a reference against which to compare each individual clear frame to determine the vector distortion field for each frame. The vector distortion field for each frame is then used to remove the distortion from each clear frame, thus enabling the creation of an image which is both clear and undistorted, almost in real time. After n frames, the first frame in the sequence is removed from the calculated mean frame when the nth frame is added, thus the mean frame is formed by a constant number of frames derived over a period of n frames up to the present frame. The value chosen for n will depend on the degree of distortion experienced and the parameters of the camera, but essentially represents the period required to generate a mean frame which is static within the tolerances of the system.

In use, it is possible to move the FOV of the camera, or to image changing scenes, if the change in the image is relatively small over the period taken to collect n frames. Thus the degree of distortion tolerable in the image can be traded off against, for example, the rate at which a camera including the system can pan.

Thus, in the method, a single image is captured, and all of the "lucky sub-frames" are extracted, not by physical movement of masks or other optical components, but rather in the post-processing stage. Advantageously, there is no need for complicated optics and moving stages. The effects of time-varying phenomena, such as atmospheric distortion, are avoided as the resultant composite image is obtained from a single image and hence from a single moment in time.

The image generated by a coded aperture system can be thought of as the convolution of the image that would be projected onto the detector using a single pinhole, and the projection of the pattern of holes in the mask from all points on the optical axis. To obtain an image of the scene focused at any particular range, the image of the mask as projected onto the detector from the point on the optical axis which lies at that range is deconvolved from the whole image. Thus, by choosing the range from which to project the mask image that will be deconvolved from the captured image, one can focus the final image at any desired range. Note that, to do that, one does not need to capture multiple images or use multiple masks: the only change is in the image-processing computation. The proposed system is capable of generating and displaying a complete clear (de-blurred) image from a single captured frame. To also remove distortion from the displayed clear image will initially take a little longer (typically less than 1 s), but, once generated, undistorted, de-blurred frames can be produced continuously at the full camera frame rate.

The exact level of improvement (in maximum magnification or range) will depend on the hardware being used and the atmospheric state, but typically, a factor of at least 3 to 5 is expected and up to a factor of 10 is quite feasible.

Example methods according to embodiments of the invention can be understood to involve measuring the whole Surface Of Best Focus (SOBF) of incoming light.

In the example system 100 shown in Fig. 2, a lens 1 10 casts an aerial image (otherwise known as an intermediate image - i.e. one which is not actually projected onto a surface but exists in a space volume) of a scene which is regarded as being effectively at infinity. The nominal plane of focus 120 for this image is at a distance c from the lens, given by the focal length of the lens.

At some distance f from the lens 1 10 lies an obscuring baffle 130, into which is inserted a coded aperture 140. The aperture 140 is in this example of the MURA type, but the aperture parameters, including pattern type, and size, and the values of f and g, will in general be chosen to optimise the system's performance given the constraints imposed by considerations including for example the number, size and sensitivity of the detector pixels, the focal length of the lens 1 10 and the desired field of view of the system

100.

Behind the coded aperture 140, at a range of g from the lens 1 10, lies a pixellated detector 150. The detector 150 in this example operates in the visual and infra-red wavebands, but detectors operating in other wavebands may of course be used. The detector 150 is operated in a sequence of

"frames", at a rate of F frames/second, and each frame period includes an integration period, during which each pixel of the detector 150 integrates the signal generated by the flux which illuminates it during that period. After the integration period is complete, the stored signal from each pixel in the detector 150 is read out by suitable electronics and stored in computer memory, as an array of numeric data. Between each frame, that array is processed to calculate the shape of the SOBF of the light in the region surrounding the nominal aerial focus 120.

In an ideal scenario, with a perfect optical system, the SOBF would be a plane coincident with the detector surface 120. Such a situation might be approachable when a well-corrected optical system is used to image a scene at infinity through a vacuum. More typically, however, a long-range target is viewed using a system with imperfectly corrected optics, through a distorting atmosphere which varies over time. That latter scenario typifies the kind of case principally envisaged here.

This SOBF information is used to calculate the best estimate of an undistorted image, which is then output by the system 100 for use as required. The SOBF information itself may also be output by the system 100, if required.

An example of a process by which the SOBF information can be extracted will now be described.

When an optical system casts an image onto a detector, the image will appear to be in focus in those areas of the SOBF which lie in the detector plane. In particular, within the constraints imposed by limiting factors such as diffraction, optical aberrations and transmission through intervening media, the intensity detected at a given point on such a plane corresponds directly to the intensity of the scene imaged at that point (indeed, that is the nature of imaging devices and is the means by which we can recognise an image as being a representation of the thing of which it is an image).

However, when the SOBF does not lie in a single plane, the intensity experienced at a given point on that plane will also partly be influenced by the intensity of the scene in areas immediately surrounding the imaged point. In the limit, as the SOBF diverges from the detector plane, the intensity of a point on the detector plane eventually begins to be determined by the mean intensity of the whole scene.

The SOBF can be understood to be the locus of the points where the imaged intensity most closely corresponds to that of the intensity of the imaged point of the scene.

If a detector were moved parallel to the optical axis, from the side of the SOBF nearest to the lens to the side further away, and the changing output from each pixel observed, one would thus expect to see each pixel intensity varying from near the scene mean, through a maximum absolute (not algebraic) deviation from that mean, and then back to the mean again. The point along the optical axis at which maximum absolute deviation for each pixel was seen would represent the "height" of the SOBF along the optical axis at the x and y co-ordinates represented by that pixel. However, such a system would be complex and costly and would necessitate multiple measurements over a relatively long period. When trying to measure a rapidly varying SOBF, such as one induced by atmospheric disturbances, such long measurement periods are unacceptable. Instead, in example embodiments of the present invention, a coded aperture is used to capture simultaneously all of the information from which the images at any desired plane in the region of the SOBF can subsequently be extracted through suitable processing.

Figure 3 shows a cross-sectional view of a simple imaging system 200 with a coded aperture 210 and a detector 220. Objects to the left of the coded aperture 210 will form a pseudo-image on the detector 220, but none will be in focus regardless of range. However, focused images can be generated from the captured data. If it is required to generate an image of the scene focused at plane A, the pattern 230A of the coded aperture 210 as projected onto the detector 220 from the point on the optical axis lying in plane A must be deconvolved from the captured data. An image focused at plane B can be obtained in an equivalent manner, using the pattern 230B of the coded aperture 210 as projected onto the detector 220 from the point on the optical axis lying in plane B. The required projected deconvolution patterns 230A, 230B are obtained by calculation or by measurement (for example during the manufacturing process).

Figure 4 shows a comparison between the patterns required to focus the above system at planes A and B respectively.

So, referring back to Fig. 2, it can be seen that if the system is used to image a scene at long range through an inhomogeneous atmosphere, a SOBF will be generated in the region of the plane 120 at a range of c from the lens 1 10 and may extend some distance in front of and behind that plane - say from a plane 170 at a range of b from the lens 1 10 to a plane 180 at a range of d from the lens 1 10, for example.

The coded aperture 140 and detector 150 shown in Fig. 1 may be used, in the manner described above, to decode focused images of for example the planes 160, 170, 120, 180, 190 at a to e respectively (or any desired planes for that matter). (The "focused images" extracted at each of the required planes may have to be re-scaled appropriately, so that the x and y values consistently represent the same parts of the scene.) The intensity of each pixel in each of these images, plotted as a function of distance from the nominal focal plane, will result in a plot which looks something like those in Figs. 5 or 6.

In the simplest approach, the above information is sufficient to generate an image with much of the atmospheric defocus effects reduced or eliminated. To do so, one simply needs to choose, for each pixel, the value where it shows the greatest absolute deviation from the mean of its values in all planes imaged.

However, alternative processing methods are also possible. These revolve around the idea that the data generates an estimate of the shape of the SOBF, and that that estimate may be imperfect but can be improved by using all the available information.

For example, one can improve the methodology for determining the point at which the best estimate of the SOBF position lies for a given pixel by modelling how the function is constrained to vary as a function of distance along the optical axis, and finding the best fit of the points to that function. That minimises the effect of errors on individual points due to noise. The rate of variation of the SOBF would normally also be expected to be constrained in the plane normal to the optical axis, which would mean that locally weighted means of the value of the SOBF at a given pixel can be obtained using an appropriately weighted local shaped function.

The above functions vary with a number of factors, including system- related parameters such as f-number, diffraction limit and projected pixel size. The effect of those on the required functions could be pre-determined. Other relevant parameters are user selected or fixed, including range to target. Those categories of features are expected to be sufficient to generate functions which can be used to improve the system performance in generating accurate SOBFs. One can also use other means to improve the signal such as applying filtering. For example, to find the height of the SOBF on pixel x,y, one could find the weighted centroid of the value d2l/dz^A2, over the range from the minimum to the maximum distance along the optical axis, where I = the intensity of the chosen pixel at each of the positions along the optical axis and z represents that distance.

Further improvements may be included by accounting for features constrained by the environment, such as the characteristic angle over which distortion is correlated in a given type of atmosphere. These may be incorporated in a fixed value (based on measurements) to allow for the most common types encountered, or be user selectable, or be self-determined by the system using iterative methods.

Distortion will now be discussed. Distortion (rather than defocus) is the stretching of the points of the image in the xy plane, rather than along the optical axis. The shape of the SOBF is believed to be affected by distortion, and it is speculated that generating a good enough SOBF enables one to calculate the degree of distortion present and remove it.

In any case, removal of distortion is relatively simple problem solvable by known techniques (e.g. by creating a mean image over time, noting the significant points in the image, associating those points in a specific frame with the mean frame, for each point noting its shift in x and y, and for each pixel, interpolating its shift from the nearest points).

Whilst the present invention has been described and illustrated with reference to particular embodiments, it will be appreciated by those of ordinary skill in the art that the invention lends itself to many different variations not specifically illustrated herein. By way of example only, certain possible variations will now be described.

In theory, many patterns of apertures could be used to perform the above process. In practice, the nature of the pattern will influence certain aspects of the image generation process, e.g. intensity, resolution, image uniformity, directional spatial frequency response etc. Similarly, some atmospheric effects could be better corrected by some patterns than others. In another example embodiment of the invention, after a clear, undistorted image has been generated, it is deconvolved from the original image to give a measure of the atmospheric distortion on that frame. Knowing the nature of the atmospheric distortion over a few frames thus enables the camera to determine what the optimum aperture pattern is for the next frame. In example embodiments, the aperture is not fixed, but rather is generated by means of an SLM (Spatial Light Modulator, essentially a controllable, transmissive LCD); thus the optimum pattern can be used, thus improving the overall camera performance.

The technique described herein has many potential applications: it would potentially be a useful technique in any case where high resolution images are required at very long ranges. Example embodiments may be sufficiently light to be used in very small, hand-held devices. Example applications involving passive systems include:

• in digital cameras or binoculars

• in telescopic sights

• from an airborne sensor platform

· in satellite imaging (though the improvement would be less marked). Using the method described above, it would be possible to infer the form of the wavefront distortion created by the atmosphere at a given instant. If a system with a fast enough frame rate is used, it is possible to compensate for this distortion before the atmospheric distortion wavefront has changed significantly. Thus, in an alternative example embodiment, a laser focusing system is adjusted to compensate for the measured wavefront distortion, providing an active system that is robust against atmospheric distortion. This is accomplished using, for example, an adaptive optics element programmed using the measured wavefront distortion information, or (for low power applications) using the sensor aperture itself (if an SLM is employed). Such a laser focusing system may be useful in for example weapons or

communications systems.

Thus, the system is capable of more than just removing long-range atmospheric blur. Further applications include:

· Imaging through distorting surfaces (rather than volumes).

• Athermalising an optical system (to stop it going out of focus when its temperature changes and its components expand or contract). This can be a very difficult and expensive step in optics development.

· Imaging through exceptionally turbulent media at short range- e.g. in furnaces & through jet efflux.

Where in the foregoing description, integers or elements are mentioned which have known, obvious or foreseeable equivalents, then such equivalents are herein incorporated as if individually set forth. Reference should be made to the claims for determining the true scope of the present invention, which should be construed so as to encompass any such equivalents. It will also be appreciated by the reader that integers or features of the invention that are described as preferable, advantageous, convenient or the like are optional and do not limit the scope of the independent claims. Moreover, it is to be understood that such optional integers or features, whilst of possible benefit in some embodiments of the invention, may not be desirable, and may therefore be absent, in other embodiments.

Claims

1 . A method of imaging an object through a time-varying inhomogeneous medium, the method comprising:

(v) identifying focused portions in a plurality of the decoded images; and

2. The method of claim 1 , wherein the third separation differs from the second separation by the increment distance, so that the second separation is midway between the first separation and the third separation; and wherein the step of identifying focused portions in a plurality of the decoded images comprises the step of (a) identifying any corresponding portions of, on the one hand, the first decoded image and, on the other hand, the third decoded image, in which the object is defocused by the same amount, and thereby identifying a focused portion of the second decoded image that corresponds to the identified corresponding portions of the first decoded image and the third decoded image; and wherein steps (ii), (iii), (iv) and (a) are repeated using a different set of first, second and third decoding masks and thereby identifying further focused portions of the second decoded images; and wherein the composite focused image is formed from the identified focused portions of the second decoded images.

3. A method as claimed in claim 1 or claim 2, in which the composite focused image is generated from one coded image.

4. A method as claimed in any preceding claim, in which the steps of decoding the coded images comprise deconvolving the Fourier transform of the coded-aperture mask from the coded images.

5. A method as claimed in any preceding claim, in which the coded aperture mask comprises a pattern of areas forming at least two sets of different transmissivities.

6. A method as claimed in claim 5, in which the steps of decoding the coded image using each decoding mask to form each decoded image corresponding to each separation of the coded-aperture mask and the imaging plane include the step of scaling the pattern of areas of the coded- aperture mask according to each said separation.

7. A method as claimed in claim 2, in which the step of identifying any corresponding portions of, on the one hand, the first decoded image and, on the other hand, the third decoded image in which the object is defocused by the same amount comprises comparing a phase diversity metric of the first decoded image and the third decoded image.

8. A method as claimed in any preceding claim in which the step of forming the composite focused image comprises the step of scaling the identified focused portions of the decoded images to take into account changes in projected size and/or position of the portions due to the different separations of the coded aperture mask and the imaging plane for which those focused portions are obtained.

9 A method as claimed in any preceding claim, further comprising the step of removing distortion arising from the time-varying inhomogeneous medium from the composite focused image, to form an undistorted, focused image.

10. A method as claimed in claim 9, wherein the step of removing distortion comprises the steps of:

(a) imaging the object through the coded-aperture mask a plurality of times, to form a plurality of the coded images at the imaging plane;

(b) forming a cumulative image frame from the plurality of the

coded images, or carrying out steps (ii) to (vi) on each coded image and forming a cumulative image frame from the resulting plurality of composite focused images;

(c) using the cumulative image frame as a reference frame by

(d) using the vector distortion field for each of the plurality of images to remove the distortion from that respective image.

1 1 . A method as claimed in any preceding claim, in which the focused portions are identified in the plurality of decoded images by identifying a surface of best focus.

12. A method as claimed in any preceding claim, in which the focused portions are identified in the plurality of decoded images by identifying corresponding portions in each decoded image, measuring the intensity of each corresponding portion, calculating the mean of the measured intensities of all corresponding portions and identifying as the focused portions the portions having an intensity having the greatest absolute deviation from the calculated mean intensity.

13. An apparatus for imaging an object through a time-varying

inhomogeneous medium, the apparatus comprising:

(i) a coded-aperture mask configured to form a coded image at an imaging plane;

(ii) an imager arranged to record the coded image;

(iii) a decoder configured to decode the coded image using a

b. a second decoding mask configured to form a second

14. An apparatus as claimed in claim 13, wherein the third separation differs from the second separation by the increment distance, so that the second separation is midway between the first separation and the third separation; and wherein the decoder is configured to: a. form a plurality of different sets of the first, second and third decoded images using a plurality of different sets of first, second and third decoding masks;

and wherein the image processor is configured to:

a. for each set of first, second and third decoded images,

c. repeat steps b and c for the plurality of different sets of the first, second and third decoded images and thereby identify further focused portions of the second decoded images. and wherein the composite focused image is formed from the identified focused portions of the second decoded images.

15. An apparatus as claimed in claim 13 or claim 14, in which the image processor is configured to generate an undistorted focused image from the composite focused image and to deconvolve the undistorted focused image from at least one of the decoded images to provide a measure of the distortion and/or defocusing caused by the time-varying inhomogeneous medium when the coded image was formed.

16. An apparatus as claimed in any of claims 13 to 15, further comprising an active optical element.

17. An apparatus as claimed in claim 16, in which the image processor is configured to provide to the active optical element a signal indicative of the measured distortion and/or defocusing.

18. An apparatus as claimed in claim 17, in which the active optical element is configured to compensate for the distortion and/or defocusing.