WO1996009207A1

WO1996009207A1 - Autonomous video-based aircraft docking system, apparatus, and method

Info

Publication number: WO1996009207A1
Application number: PCT/US1995/011803
Authority: WO
Inventors: Ming Fang; Indranil Chakravarty; Long-Ji Lin
Original assignee: Siemens Corporate Research, Inc.
Priority date: 1994-09-19
Filing date: 1995-09-18
Publication date: 1996-03-28

Abstract

An autonomous video-based aircraft docking system comprises a camera arrangement, an image processing unit coupled to the camera arrangement and a communications unit coupled to the image processing unit, the camera arrangement being oriented to obtain an image of an aircraft through a designated portion of its approach for docking, the image processing unit processing the image for detecting in accordance with prestored data a location of a predetermined feature of the image and causing the communications unit to output a signal indicative of the orientation and location of the aircraft.

Description

AUTONOMOUS VIDEO-BASED AIRCRAFT DOCKING SYSTEM, APPARATUS, AND METHOD

The present application relates to vehicle docking systems and, more particularly, to a system that enables a pilot of an aircraft to dock at a gate for passenger or cargo transfer with minimal or no guidance or assistance at all from ground personnel.

With the increasing volume of air-traffic world-wide, it has become important to develop systems that properly manage the ground movement of aircraft at an airport in order to improve safety and operational efficiency. A docking system monitors the movement of an aircraft as it approaches the gate for deplanning or enplaning, and provides signals to the pilot so that the aircraft is correctly positioned at the gate. Correct positioning of the aircraft requires that the aircraft nose wheel be positioned to within 0.5 meter (0.5m) or about 20 inches (20") of a pre-defined mark on the airfield running surface or tarmac and the aircraft body positioned along a centering line. Typically, different marks are delineated for different types of aircraft. The centering line is also usually indicated on the tarmac to help the pilot taxi the plane correctly to the gate area. A docking system must be able to track an aircraft for at least 20m so that the pilot has enough time and maneuvering room to correct any deviations in aircraft position and/or orientation.

Currently in US airports, a member of ground personnel, known as a marshall on the ground, guides the pilot to a desired gate by signalling from the tarmac. A docking system automates this process by providing the pilot with a visual display, visible to the pilot, that is mounted on either the terminal building or close to the docking gate. Present docking systems in use in high-volume European airports are based on either induction based measurements or laser propagation delay (^Mtime-of-flight") based measurements to detect the position of the aircraft with respect to the gate. A mechanical parallax device is mounted on the display so that a pilot can determine the correct orientation when the parallax device is in alignment with the pilot's view of the display.

Typically for such systems, the system comprises sensor loops, a scanning unit, an operator's panel, a visual display board with central unit and serial interfaces connecting the units. The sensor coils are laid in slots in the apron concrete along the center parking lane. The scanning unit is a microprocessor connected to the sensor loops and to the central unit by a serial channel. The purpose of the scanner is to examine the sensor loops in the apron for faults and to activate the right number and positions of loops.

Typically, when an aircraft type is selected on the operator panel, the first three loops to be used are activated. As the aircraft proceeds, a new loop is activated in front of the nose-wheel so that three loops are always in use up to the stop position.

Disadvantages of existing systems further include the factors that the installation is permanently fixed; loops have to be imbedded in the tarmac; the system cannot distinguish aircraft and vehicles; each taxi line needs its own system installation and any changes are expensive to implement; and the systems can be sensitive to the wheel structural materials. Laser radar sensors have also been developed. Such systems generally require very exacting installations and are typically hampered by weather conditions such as fog and or snow. In accordance with an aspect of the invention, an autonomous video-based aircraft docking system comprises a camera arrangement, an image processing unit coupled to the camera arrangement and a communications unit coupled to the image processing unit, the camera arrangement being oriented to obtain an image of an aircraft through a designated portion of its approach for docking, the image processing unit processing the image for detecting in accordance with prestored data a location of a predetermined feature of the image and causing the communications unit to output a signal indicative of the location.

In accordance with another aspect of the invention, the image processing unit is a display unit and the signal indicative of the location comprises a visual display.

In accordance with another aspect of the invention, the predetermined feature comprises a windshield of the aircraft.

In accordance with another aspect of the invention, the predetermined feature comprises a set of characteristic points on the windshield.

In accordance with another aspect of the invention, a computer-implemented method for enabling a pilot of an aircraft to dock at a gate comprises the steps of: obtaining an image of at least a portion of the aircraft from a camera; digitizing and enhancing the image; creating an image bounding box so as to localize therein a windshield of the aircraft by reference to a prestored template; localizing corners of the windshield; verifying in accordance with a predetermined pattern whether the windshield localized correctly represents a true windshield of the aircraft, and if not, returning to the step of digitizing, and if so, processing the image for eliminating outliers; determining from the localized windshield a position for the aircraft; and communicating the position to the pilot.

In accordance with another aspect of the invention, apparatus for communicating position to a pilot of an aircraft approaching a docking station, comprises: camera apparatus for deriving an image of at least a portion of the aircraft; computer apparatus for:

(a) digitizing and enhancing the image; (b) localizing an aircraft windshield in the image;

(c) forming a bounding box around the windshield;

(d) localizing corners of the windshield;

(e) verifying the windshield localized in step (b) ;

(f) eliminating outliers;

(g) determining position and orientation of the aircraft; and

(h) displaying the position and orientation to the pilot.

In accordance with another aspect of the invention, the computer apparatus further compares template information stored in its data base memory on aircraft configuration, based on model selection, and compares the information with the image in forming the bounding box in step (c) .

In accordance with another aspect of the invention, a computer-implemented method for enabling a pilot of an aircraft to dock at a gate comprising the steps of: obtaining an image of at least a portion of the aircraft from a camera; digitizing and enhancing the image; creating an image bounding box by comparing the image with a prestored template of a corresponding portion of a selected aircraft type so as to localize therein a windshield of the aircraft; determining from the localized windshield a position for the aircraft; and communicating the position to the pilot.

In accordance with another aspect of the invention, the step of creating an image bounding box.comprises localizing therein a plurality of portions of the windshield and comparing the portions with prestored sub-templates of corresponding portions of the selected aircraft type.

In accordance with another aspect of the invention, the step of creating an image bounding box comprises utilizing a succession of templates of increasingly fine resolution as the aircraft approaches.

In accordance with another aspect of the invention, a computer-implemented method for enabling a pilot of an aircraft to dock at a gate comprises the steps of: obtaining an image of at least a portion of the aircraft from a camera; digitizing and enhancing the image; creating an image bounding box so as to localize therein a windshield of the aircraft; localizing corners of the windshield by comparing the image with a prestored template of a corresponding portion of a selected aircraft type so as to localize therein a windshield of the aircraft; determining from the localized corners a position for the aircraft; and communicating the position to the pilot.

In accordance with another aspect of the invention, the step of creating an image bounding box comprises localizing therein a plurality of portions of the corners and comparing the portions with prestored sub-templates of corresponding portions of a selected aircraft type.

In accordance with another aspect of the invention, a computer-implemented method for enabling a pilot of an aircraft to dock at a gate comprises the steps of: obtaining an image of at least a portion of the aircraft from a camera; digitizing and enhancing the image; creating an image bounding box by comparing the image with a prestored template of a corresponding portion of a selected aircraft type and comparing portions of image with prestored subtemplates of corresponding portions of the selected aircraft type so as to localize therein a windshield of the aircraft; determining from the localized windshield a position for the aircraft; and communicating the position to the pilot.

The invention will be best understood from the following detailed description of preferred exemplary embodiments in conjunction with the Drawing, in which

Figure 1 shows a system overview in accordance with the invention in a flow chart type of format; Figure 2 shows a plan view of a gate geometry in accordance with the invention;

Figure 3 shows a side view or elevation of a gate geometry in accordance with the invention; Figure 4 shows a procedure in accordance with the invention for windshield detection and corner detection;

Figure 5 shows corner detection in accordance with the invention and the presence of an outlier;

Figure 6 shows windshield verification in accordance with the invention by spatial relationships of detected corners;

Figure 7 shows windshield localization by using a resolution reduction factor of 8 for template matching, with a focused and a strongly defocused image; Figure 8 shows an example of a windshield template with two subtemplates marked with rectangles for multiple evidence based template matching in accordance with the invention;

Figure 9 shows windshield localization using global histogram modification in accordance with the invention under uniform lighting change, Figure 9a, 9b, 9c, and 9d, respectively showing conditions of dusk, night, fog, and sunny conditions;

Figure 10 shows an illustration of the partitioning concept in accordance with the invention for solving problems caused by nonuniform illumination change;

Figure 11 shows and example of aircraft docking under non-uniform lighting change, in accordance with the invention; Figure 12 shows an illustration of a size sensitive template matching scheme in accordance with the invention; and

Figure 13 shows an illustration of a size sensitive template matching scheme in accordance with the invention using only three adjacent templates. Figure 1 illustrates basic components of a video-based aircraft docking system in accordance with the present invention. As an aircraft proceeds to a docking gate, the airport management system notifies the docking system of the aircraft type scheduled for docking at the gate. Using a video sensor, the docking system continuously checks to see whether an aircraft has entered its field of view. Once an aircraft is detected, the docking system attempts to localize the windshield of the aircraft, by matching the input video image against a computer model of the incoming aircraft. An aircraft data-base is provided that contains a model of each possible aircraft type expected at a gate for docking. Images are preferably acquired by a video sensor, such as a CCD camera. The analog image data is digitized and enhanced for windshield localization (box 1) . The windshield localization process finds an object in the image that most resembles the windshield and then locks the feature by placing a bounding or circumscribing box around it. This procedure uses a template from a data-base to match and localize the windshield (box 2) . The bounding box is passed to the corner localization process (box 3) . The purpose of this process is to perform a detailed analysis of the windshield and locate the corners for (i) verifying the windshield and for (ii) matching the windshield shape against a model or aircraft type verification. Once the windshield is verified (box 4) the windshield.corners are passed to an outlier elimination process (box 5) . This process checks the spatial relationship among the corners and eliminates false matches. The spatial coordinates of the corners are passed for position and orientation determination (box 6) . A succession of frames may be used for measuring the position and orientation.

The result of this processing is passed on to a display unit, which shows the current aircraft position and orientation and provides guidance to the pilot (box 7) . The entire process is updated several times a second in order to provide real-time guidance to the pilot.

The system in accordance with the present invention uses a reduced resolution image to detect and place a bounding box around the windshield of the incoming aircraft. Once the windshield is detected, the bounding box is used to restrict the search area for corner localization in the image.

In an operational setting, the corner localization and windshield localization would not be perfect at all times, and therefore the system needs to detect possible outliers or false matches. Outliers can be detected by checking whether each corner has the expected spatial relationships with the others. The result of the outlier processing is shown as set of X-Y coordinate points on the image corresponding to the windshield corners. The central dot in the same figure shows the computed center of the aircraft that needs to be aligned with the center line.

Based on the corner coordinates on the video image, various techniques can be used to determine the aircraft position and orientation. The accuracy obtained is determined by the distance of the video sensor to the aircraft and whether one camera or two cameras are used for distance determination. If two cameras are allocated for each gate, the positional error is estimated to be less than 0.5 meter.

While the above described tasks can be accomplished for a controlled lighting condition, the present invention allows satisfactory operation over a wide dynamic range that can accommodate changes in outdoor lighting for 24 hour operation. In addition, the present invention can also accommodate static and dynamic shadows that may cover the aircraft, and weather related changes, such as water on the aircraft body, that may result in specular reflections.

Next will be described a typical geometry of the docking system. The basic configuration of the docking system is shown in plan view in Figure 2 and in side view or elevation in Figure 3. Two center lines are indicated at DD = 10 meters apart. The distance from the parked plane to the terminal building is approximately 30.0 meters, for example, 31.8 meters along RR on the right and 27.5 meters along LL on the left line, WI, W2 indicating the nose wheel. A video sensor and display unit 16 is mounted on the terminal building. The video sensor (including a camera) and display unit may be in separate cabinets and constitute separate units. In order to guide the pilot, the plane must be tracked for about 20.0 meters. RA indicates a ramp.

A side view of the geometry is presented in Figure 3. The camera is shown to be mounted at a height HH of 20 meters. For a Boeing 747 aircraft, whose nose cone is at height of NN = 4.65m above the ground, a camera tilt angle T of 16.6 degrees will allow tracking the nose and windshield for approximately 20.0 meters. Either one or two cameras WW indicates where the nose wheel stops, at a distance XX of about 30 meters from the terminal building, may be used in this setting. With two cameras, using stereo triangulation, the distance of an aircraft can be determined to within 0.5m accuracy. In this case, the windshield corners are used for correlation between the two images. With one camera, and knowledge of the incoming aircraft height, distance can also be determined. Both scenarios are herein disclosed.

Two key components of the video based aircraft docking system in accordance with the invention are the aircraft windshield and corner localization. In the following, implementation details relating to a preferred embodiment will be explained.

The windshield of an aircraft is used for tracking. This provides the following advantages:

— Stability. Windshields are a consistent feature across all different aircraft types; that is, all aircrafts have a windshield located at its front;

— Visibility. Windshields are in practice always visible while other parts of an aircraft, such as its wings, engines, wheels, or rudder, may be occluded by building infrastructure or service vehicles on the ground; — Contrast. Windshields exhibit good contrast since they are transparent and embedded in an otherwise opaque aircraft body.

Geometry. Windshields have a well-defined fixed geometry for each commercial aircraft type; Windshields are localized by template matching techniques, which are described in further detail below.

In accordance with the present invention, windshield corners are herein for determining aircraft position and orientation. The use of windshield corners provides the following advantages.

Since the windshield is in practice always visible from the display, windshield corners are very stable, reliable, and distinctive features.

Because windshield corners are very distinctive, their location can be determined accurately. Each corner location can be found with an error less than 3 pixels. See Figure 4, in which the bounding box indicates the detection of a windshield. Eight corners, indicated by dots, have been correctly located. The location of the windshield center, indicated by the dot in the center of the bounding box, is then computed from these corners. Because of shadows or lighting related changes, false corner matches or outliers may occur. However, outliers usually do not cause a problem, because they can be identified by verifying the spatial relationships of the detected corners. Thus, Figure 5 shows corner detection and an outlier. One of the eight detected corners, which are the white dots in the bounding box is obviously an outlier because it does not have the expected spatial relationships with the other corners.

In the course of docking, the windshield exhibits a large change in scale. The distortion of the windshield size in the image can be as large as 10 times or more from the time a windshield is detected to final docking position. For robust template matching, this may result in the use of many corner templates of different sizes In order to compensate for this size distortion. Since a projective transformation preserves straight lines, the use of corners limits the number of templates needed to accommodate scale changes. Thus, proper windshield detection in accordance with the invention is resistant to perspective distortion.

Since the approach angle of the aircraft to the gate may vary, it may be viewed in different orientations and several templates in accordance with the invention may be needed. Windshield corners are sufficient to handle variations in approach angle up to 30 degrees.

A windshield has a plurality of corners. Once the positions of all corners or just a subset of corners is known, the aircraft position and its orientation can be determined accurately in accordance with the invention. If the error of each corner position is within 3 pixels (after outlier elimination) , the sub-pixel accuracy of aircraft positions can be obtained, simply because the errors can be averaged out when multiple corner locations are combined. Depending on the accuracy requirements and the hardware cost tradeoffs, a choice can be made to use a one-camera system or a two-camera system for each gate area.

In accordance with the invention, windshield corners are utilized for verifying if the localized windshield is indeed a windshield. Figure 6 shows windshield verification by spatial relationships of detected corners. Because the aircraft windshield is not entirely visible, the windshield localization does not locate the windshield correctly and a wrong bounding box is drawn. Consequently, within the bounding box, the corner detector finds 8 phony corners. Because these 'corners' clearly do not have the expected spatial relationships, it can be concluded that the windshield has not been recognized and the corner information should not be trusted. Thus, in Figure 6, because the relative positions of the detected corners are completely wrong, the present invention can determine that the object is not a windshield even though the initial template matching might recognize the presence of windows.

Generally, different aircraft may have different windshield shapes. Based on relative corner positions, it is possible to distinguish between certain types of aircraft. However, a general, accurate classification of aircraft types may require additional information such as the engine location, wing-span etc.

By matching all of the templates in the data-base, the aircraft type or class can be determined.

In summary, the use of windshield corners for detection in accordance with.the invention is reliable. The corner locations can be determined accurately, and when combined, they can determine the aircraft position and orientation accurately and reliably. A small number of corner templates is sufficient to deal with large perspective distortions and approach angles in the course of docking.

Template matching is a technique used to detect the presence of an object in an image. Thus, for example, see A. K. Jain, "Fundamentals of digital image processing", Prentice-Hall, New Jersey, 1986. If a picture of an object to be detected, herein referred to as the target object, is available, this picture can be stored as a template. To detect the presence of an object in a new image, a similarity score is-computed reflecting how well the new image matches the template for each possible template location. The location exhibiting a maximal similarity score obtains is considered as.the object location. A common similarity measure between a template t(i) and an input x(i) is computed as

Similarity = Σ | x ( i) - t ( i) | ^p i

where i is the index to each pixel in the template or the input, and p is a similarity metric. The exemplary docking.system in accordance with the invention uses p = 1.

There are three major issues of concern with template matching as a technique for object detection/localization. These are:

Efficiency: The time used to find the best match is determined by two factors: the size of the template and the size of the search space (i.e., all of the template locations to be tested) . To improve efficiency, the template size can be reduced, or.the search space can be reduced, or both can be done. Illumination changes: The input images and the template images may be taken in very different illumination conditions such that their pixel-by-pixel differences are very large although they are images of the same object. The primary issue here is how to "normalize" the input image so that the object in the normalized image will exhibit similarity to the one in the template, pixel by pixel, in spite of illumination or weather related changes.

Confidence measure: Template matching will always find the best match of the target object in the input image. However, the best match found is not guaranteed to be the target object. For instance, even when the input image does not contain the target object, template matching may still determine a "best match". Therefore, it is herein recognized that some method is needed of knowing whether the localized object is indeed the object being looked for or whether it corresponds to a false match.

In the following, advanced template matching techniques are addressed as they may be applied in the context of the present invention in order to address the above-mentioned issues.

In accordance with an embodiment of the present invention, a template matching scheme with high efficiency is described, which addresses basically the efficiency issue of the template matching. This efficient scheme comprises three components: (i) a multi-resolution processing scheme, (ii) a coarse-to-fine search strategy, and (iii) a limited search space based on a priori information available to the system or provided by previous processing steps.

A multi-resolution processing scheme for template matching is applicable in accordance with the invention. Since an aircraft should be docked in real time, low-resolution is used for the initial window.localization. The resolution reduction factor for both image and templates is determined by the size of the smallest dimension of the templates used for the matching. According to the experience of the present inventors, it is preferred that the smallest dimension at the low resolution should not be smaller than 8 pixels. This selection is somewhat subjective and it is also influenced by the object to be matched. For the windshield localization for aircraft docking, this number was found to be suitable. The performance of the template matching for windshield localization will be strongly affected if the smallest dimension is smaller than 8, due the lack of information or constraints in that particular small dimension. This determines the resolution reduction factor that should be used for the template matching. For example, if an image has a size of 512 x 512 pixels, and 2 templates to be matched with the image have 80 x 70 and 120 x 96 pixels, . the smallest dimension of these templates is 70. Hence, the resolution reduction factor should be 8. Once this factor is determined, all of the templates and the image will be down-sampled by a factor of 8, and the matching is then conducted at that resolution. Since this resolution reduction is template-dependent, it will be dynamically changed if the-smallest dimension of the templates for the matching is changed.

There are a number of advantages to this adaptive resolution reduction among which are:

low resolution means low computational cost, and therefore high processing speed; low resolution makes possible for bigger search area for the matching; low resolution makes the system insensitive to jittering or defocusing of the video sensor; and low resolution permits the system to perform fast noise filtering and/or enhancement, if the video images are relative noisy.

For example, Figure 7 shows an example of the windshield localization in the situation where the image is strongly defocused. Figure 7 shows windshield localization by using a resolution reduction factor of 8 for the template matching. Figure 7a is the original image and the image in Figure 7b is strongly defocused. Nevertheless, the windshield localization still provides relatively accurate location of the windshield.

Once the windshield is localized, high resolution is then used for corner localization for high accuracy.

In accordance with the present invention, prior knowledge is advantageously utilized to constrain the search space. A facile approach to localize an object in a 512 x 512 image is to process the entire image, testing each possible template location to find the best match. This approach is very time-consuming, however. To be efficient, the docking system in accordance with the present invention utilizes prior knowledge to constrain the search space.

Windshield localization is also carried out to advantage in accordance with the present invention. Once the video sensor is installed and fixed, there is a limited region in the image where the windshield is.expected to appear when the aircraft enters the field of view. Therefore, it is herein recognized that it should only be necessary to perform template matching in this relatively small region, thereby saving much effort. After the windshield has been localized the first time, there is only a small local region to which the windshield can next move, since the speed of the aircraft is limited. Therefore, by predicting where the aircraft will move to, the search space can be further constrained.

Corner localization is also advantageously addressed in accordance with the present invention. Since the corners are on the windshield, there is a limited region where each corner may appear within the bounding box of the windshield. It is herein recognized that it should therefore not be necessary to have to search the entire image for corners, thereby saving additional computational time.

Transition from coarse-to-fine search is also accomplished with advantage in accordance with the present invention. Even after the search space is constrained, the number of locations to be considered may still be very large. To further improve efficiency, a coarse-to-fine search strategy is utilized to find the best match within a constrained search space: first, only every fourth (for example) location in the constrained search space is considered as candidate template location. After the best match in this coarse-resolution search space is found, a finer-resolution search is performed within a small bounding box centered at the best match found previously. If the search space is very large, more than 2 levels of search resolutions can be used. Currently, in the preferred embodiment, 1-3 levels of resolution are used, depending on the size of the windshield in the image.

The confidence issue of the template matching is next reviewed. In accordance with the present invention, a template matching technique is utilized for windshield localization. A known windshield in a scene can be detected by searching for the location of a match between the windshield template and the scene image. Template matching can be conducted.by searching the displacement of the windshield template, where the mismatch energy is minimum or the correlation is maximum. However, this approach to conventional template matching will not indicate directly whether the object to be detected is present in the scene. If the match score is high, the confidence of the presence of the windshield is high. If the match score is medium or low, then the template matching does not indicate whether this medium or low match score is due to the deformation or defocusing of the object, or due to noise and lighting influences on the object, or due to the absence of the object. Hence, it is necessary for the template matching algorithm to have some additional means to determine if it is not known a priori whether the object is present

To that end, in accordance with the present invention, multiple evidence based template matching is utilized. The basic idea of the multiple evidence based template matching is to use not only a whole object template but also some sub-templates (parts) of the this object to conduct multiple matching with the scene image. Such templates are shown in Figure 8, which shows an example of a windshield template with two sub-templates marked with rectangles for multiple evidence based template matching. The circles in Figure 8 represent the localized positions of these three templates with the image. This multiple matching provides the best matches for the whole object and for the parts of the object corresponding to the sub-templates. Since the parts of the object have some fixed geometrical, that is, spatial, topological, relations, the detected best-matched positions of the whole and the sub-templates should also have the similar geometrical relations. If the geometrical relations are not as expected, the confidence is low even though the match score may be high. Conversely, when relations are similar to the relations to be expected, the confidence of the presence of the object to be detected is very high, even though the matched scores are medium or low. It is also possible to integrate the windshield and the corner localization together to implement a fast multiple evidence based windshield localization. Since corners of the windshield can be viewed as parts of the windshield, some regular topological relations among the corners and the windshield are to be expected. This property of the windshield corner can be utilized to verify if the localized windshield is indeed a windshield. In Figure 6, because the relative positions of the detected corners are far away from those to. be expected from the windshield corners, a determination can be made that the object is not a windshield even though the initial template matching might show high match score at that location. The multiple evidence based template matching provides an important confidence measure which is independent of the match score. This confidence measure allows the system to make correct decision about the presence of the object to be found, even if the difference, and therefore the tolerance, between the template and the real object is relatively large. A large difference, that is a high tolerance, which means large distortions and illumination changes, as well as high noise level, are allowed by using this technique for the initially locating the object to be searched. This technique is essential for locating the correct object with high confidence for further processing steps, such tracking.

A remaining major issue of the template matching is the illumination problem. In fact, one of the most challenging problems for any outdoor vision based system is the illumination change. The illumination change can be uniform or non-uniform. There are known histogram based techniques which can deal with the uniform lighting change. A well-known histogram modification technique is histogram equalization. See for example, W.K. Pratt, "Digital Image Processing", Section 12.1 and 12.2, Wiley, New York, 1978 and the afore-mentioned A. K. Jain, "Fundamentals of digital image processing", Prentice-Hall, New Jersey, 1986.

The present inventors have found that histogram equalization does not work very well for windshield/corner localization. The technique found to be very successful is as follows (see A. K. Jain, "Fundamentals of digital image processing", Prentice-Hall, New Jersey, 1986): the darkest 10% (for example) of the pixels are assigned the minimum pixel value 0, and the brightest 10% (for example) of the pixels are assigned the maximum pixel value 255. The other 80% of the pixels are scaled linearly to be between 0 and 255.

Histogram modification, however, is a computationally expensive process, if it has to be repeated it for each subimage which is to be matched against a template. To be efficient, each subimage is not individually normalized; there are often many hundred subimages to be considered. Since the search space is known before doing template matching, the bounding box which includes all of the subimages to be considered within this search space can be computed. Then subimage is normalized within this bounding box, and this is done once for all. It is herein recognized that within the bounding box, the subimage can be further partitioned into regions for histogram modification, if necessary.

The histogram modification scheme described above can efficiently compensate the effect of the uniform illumination change for the windshield and corner localization, as shown in Figure 9. In Figure 9, the windshield localization by using global histogram modification can be seen to deal with uniform lighting change. Successful windshield localization is shown under different lighting conditions: Figure 9a, 9b, 9c, and 9d, respectively show conditions of dusk, night, fog, and sunny conditions. The limitation of this kind histogram modification scheme, however, is that the output image will not be invariant under non-uniform illumination changes.

A new partitioning based method in accordance with the invention, for intensity invariant template matching is herein disclosed. This method is aimed to solve the problem of non-uniform illumination change for the situation where the illumination change can be modeled as a slowly occurring change or as a piece-wise function. The basic idea of the technique in accordance with the invention is to divide a given template into several sub-templates, as shown in Figure 10. Figure 10 shows an illustration of the partitioning concept for solving problems caused by non-uniform illumination change. Some sub-templates may be affected by the non-uniform illumination change, but if the majority of the subtemplates are not affected, the matching result will not be affected. This is because a histogram modification will be applied individually to each of those sub-templates rather than to the original template. Since the histogram modification is done individually for each sub-template, the illumination change will be reasonably well compensated, if it is not significant within each sub-template. In fact, even the illumination change in some sub-templates is indeed significant, and hence the result after the local histogram modification is no more intensity invariant, the overall result will still remain the intensity invariant globally, if the number of the sub-templates with significant illumination change is small in comparison with the total number of the sub-templates. A windshield and corner localization algorithm based on this novel partitioning method is implemented and its performance can be seen from an example where some dark shadows and some bright spots are present in the scene, as shown in Figure 11 which shows an example of aircraft docking under non-uniform lighting change. The efficiency issue of the template matching is herein further explained, but with another perspective, that of size or order sensitive template matching. For aircraft docking, the windshield of an aircraft must be located while at different distances. Due to perspective distortion, the windshield of the same aircraft has different sizes depending on the distance between the plane and the video sensor. Therefore, the template matching for locating an aircraft windshield involves in general several templates with different sizes corresponding to the windshield of the aircraft at different distances, as shown in Figure 11. It is time-consuming for the windshield localization if all of the templates are used for the template matching.

Since an aircraft always approaches from a distance the video sensor mounted near the gate at which the plane is being docked, the first time the windshield of the plane appears in the video image it is small. As the plane moves closer to the gate and the video sensor, the windshield appears larger, as shown in Figure 11. During this docking time, the size of the windshield in the images will change continuously from small to large. Hence, the template matching for the docking is size or order sensitive.

An effective scheme utilizing this property in accordance with the invention for reducing the number of templates and hence the computational cost of the template matching with multiple templates, is illustrated in Figure 12, which illustrates size sensitive template matching in accordance with the invention by using only two adjacent templates. Instead of using all templates of the object to be found with many different sizes, the method starts by locating the object at time to only with the two smallest adjacent templates TO and Tl. Since the object at time to has greater distance that at tl, the match score from the template TO is greater than the one from Tl. After a while, say at time tl, as the object is getting bigger, the match score from Tl becomes to be larger than the one from TO, then the next adjacent template T2 and T3 are used for the matching. As the template T3 is active, templates T3 and T4 are used. This scheme allows, at each instant, the use no more than two templates to locate the object whose size is assumed to be changing continuously and monotonically (enlarging or shrinking) .

If the size change is only continuous but not monotonic, three adjacent templates can be used. Figure 13 illustrates size sensitive template matching in accordance with the invention by using only three adjacent templates. As shown in Figure 13, at time to, the template T3 has the best match score, then T3 and two adjacent templates T2 and T4 are used for the matching. At time tl, T2 becomes to be active (has the best match score), then Tl, T2 and T3 will be used. At time t3, T3 is getting active again, then T2, T3, and T4 are again used. At time t4. T4 is active, then the Templates T3, T3, T5 will be used for the tracking. The scheme allows the location of the object whose size is changing continuously, but not necessarily monotonically. The above-mentioned concept can be applied to not only size sensitive but also order sensitive template matching with multiple templates.

Applicants have tested prototype implementation on a SUN workstation using C programming language under UNIX. Sample images on video tape have been taken from Newark International Airport, at different times of the day, for testing lighting conditions, and in different weather conditions, including a rain condition.

Further to the materials cited in the foregoing description, the following will be helpful in gaining a further understanding of the principles underlying the present invention: D.H. Ballard and CM. Brown, "Computer Vision", Section 3.2.1, Prentice-Hall, New Jersey, 1982.

While the present invention has been described by way of exemplary embodiments, it will be understood that various changes and modifications will readily suggest themselves to one of skill in the art to which it pertains. Thus, while aircraft windshields are preferred as detection features, other features also lend themselves to detection. Such features may be normal structural features on aircraft or they may be intentionally provided as targets for detection, such as, for example, painted or illuminated signs or retro-reflective signs. While aircraft docking is the most immediate and desirable application, the invention is clearly applicable to all types of vehicular docking and guidance. Furthermore, it will be understood that while visible light cameras are generally implied in the foregoing examples, the invention is by no means intended to be restricted to the visible spectrum and may be practiced at any convenient spectral range, such as the ultraviolet or infrared. It will also be understood that what has conveniently been referred to as a camera herein need not, in fact, be a conventional type of video camera: instead it may be a video sensing arrangement specifically for obtaining the type of video image herein required for further processing. Furthermore, communication to the pilot need not be by visual display: audio communication by voice synthesizer or other means may be employed. Accordingly, it will be understood that such changes and modifications are intended to be within the spirit of the invention whereof the scope is defined by the features of the claims following.

Claims

1. An autonomous video-based aircraft docking system comprising a camera arrangement, an image processing unit coupled to said camera arrangement and a communications unit coupled to said image processing unit, said camera arrangement being oriented to obtain an image of an aircraft through a designated portion of its approach for docking, said image processing unit processing said image for detecting in accordance with prestored data a location of a predetermined feature of said image and causing said communications unit to output a signal indicative of said location.

2. An autonomous video-based aircraft docking system in accordance with claim 1 wherein said image processing unit is a display unit and said signal indicative of said location comprises a visual display.

3. An autonomous video-based aircraft docking system in accordance with claim 1 wherein said predetermined feature comprises a windshield of said aircraft.

4. An autonomous video-based aircraft docking system in accordance with claim 1 wherein said predetermined feature comprises a set of characteristic points on said windshield.

5. A computer-implemented method for enabling a pilot of an aircraft to dock at a gate comprising the steps of: obtaining an image of at least a portion of said aircraft from a camera; digitizing and enhancing said image; creating an image bounding box so as to localize therein a windshield of said aircraft by reference to a prestored template; localizing corners of said windshield; verifying in accordance with a predetermined pattern whether said windshield localized correctly represents a true windshield of said aircraft, and if not, returning to said step of digitizing, and if so, processing said image for eliminating outliers; determining from said localized windshield a position for said aircraft; and communicating said position to said pilot.

6. A method for enabling a pilot of an aircraft to dock in accordance with claim 5, comprising the step of accessing stored template information on aircraft images and comparing said information with said image after said digitizing and enhancing

7. A computer-implemented method for enabling a pilot of an aircraft to dock at a gate comprising the steps of: obtaining an image of at least a portion of said aircraft from a camera; processing said image so as to localize a predetermined feature on said image; determining therefrom a position for said aircraft; and communicating said position to said pilot.

8. A method for enabling a pilot of an aircraft to dock in accordance with claim 7, wherein said step of communicating said position comprises activating a display.

9. A method for enabling a pilot of an aircraft to dock in accordance with claim 8, wherein said step of processing said image comprises localizing an image of a windshield of said aircraft in a accordance with a stored template.

10. A method for enabling a pilot of an aircraft to dock in accordance with claim 9, wherein said step of processing said image comprising localizing an image of a windshield of said aircraft is followed by a step of localizing corners of said windshield.

11. A method for enabling a pilot of an aircraft to dock in accordance with claim 10, wherein said step of localizing corners of said windshield is followed by a step of verifying said localizing an image of a windshield of said aircraft.

12. A method for enabling a pilot of an aircraft to dock in accordance with claim 11, wherein said step of verifying said localizing an image of a windshield of said aircraft is followed by a step eliminating outliers.

13. Apparatus for communicating position to a pilot of an aircraft approaching a docking station, comprising: camera means for deriving an image of at least a portion of said aircraft; computer means for: (a) digitizing and enhancing said image;

(b) localizing an aircraft windshield in said image;

(c) forming a bounding box around said windshield;

(d) localizing corners of said windshield;

(e) verifying said windshield localized in step (b) ; (f) eliminating outliers;

(g) determining the aircraft-class;

(h) determining position and orientation of said aircraft; and

(i) displaying said position and orientation to said pilot.

14. Apparatus in accordance with claim 13, wherein said computer means further compares template information stored in its data base memory on aircraft configuration, based on model selection, and compares said information with said image in forming said bounding box in step (c) .

15. Apparatus in accordance with claim 14, wherein said step (g) of determining aircraft class is performed by comparing all templates stored in said data base.

16. A computer-implemented method for enabling a pilot of an aircraft to dock at a gate comprising the steps of: obtaining an image of at least a portion of said aircraft from a camera; digitizing and enhancing said image; creating an image bounding box by comparing said image with a prestored template of a corresponding portion of a selected aircraft type so as to localize therein a windshield of said aircraft; determining from said localized windshield a position for said aircraft; and communicating said position to said pilot.

17. A computer-implemented method for enabling a pilot of an aircraft to dock at a gate in accordance with claim 16, wherein said step of creating an image bounding box comprises localizing therein a plurality of portions of said windshield and comparing said portions with prestored sub-templates of corresponding portions of said selected aircraft type.

18. A computer-implemented method for enabling a pilot of an aircraft to dock at a gate in accordance with claim 16, wherein said step of creating an image bounding box comprises utilizing a succession of templates of increasingly fine resolution as said aircraft approaches.

19. A computer-implemented method for enabling a pilot of an aircraft to dock at a gate comprising the steps of: obtaining an image of at least a portion of said aircraft from a camera; digitizing and enhancing said image; creating an image bounding box so as to localize therein a windshield of said aircraft; localizing corners of said windshield by comparing said image with a prestored template of a corresponding portion of a selected aircraft type so as to localize therein a windshield of said aircraft; determining from said localized corners a position for said aircraft; and communicating said position to said pilot.

20. A computer-implemented method for enabling a pilot of an aircraft to dock at a gate in accordance with claim 19, wherein said step of creating an image bounding box comprises localizing therein a plurality of portions of said corners and comparing said portions with prestored sub-templates of corresponding portions of a selected aircraft type.

21. A computer-implemented method for enabling a pilot of an aircraft to dock at a gate in accordance with claim 20, wherein said step of creating an image bounding box comprises utilizing a succession of templates of increasingly fine resolution as said aircraft approaches.

22. A computer-implemented method for enabling a pilot of an aircraft to dock at a gate comprising the steps of: obtaining an image of at least a portion of said aircraft from digitizing and enhancing said image; creating an image bounding box by comparing said image with a prestored template of a corresponding portion of a selected aircraft type and comparing portions of image with prestored sub-templates of corresponding portions of said selected aircraft type so as to localize therein a windshield of said aircraft; determining from said localized windshield a position for said communicating said position to said pilot. AMENDED CLAIMS

[received by the International Bureau on 23 January 1996 (23.01.96); original claims 3 and 9 cancelled; original claims 1,7 and 10 amended; remaining claims unchanged (3 pages)]

1. An autonomous video-based aircraft docking system comprising a camera arrangement, an image processing unit coupled to said camera arrangement and a communications unit coupled to said image processing unit, said camera arrangement being oriented to obtain an image of an aircraft through a designated portion of its approach for docking, said image processing unit processing said image for detecting in accordance with prestored data a location of a windshield of said aircraft on said image and causing said communications unit to output a signal indicative of said location.

7. A computer-implemented method for enabling a pilot of an aircraft to dock at a gate comprising the steps of: obtaining an image of at least a portion of said aircraft from a camera; processing said image so as to localize an image of a windshield of said aircraft in accordance with a stored template; determining therefrom a position for said aircraft; and communicating said position to said pilot.

10. A method for enabling a pilot of an aircraft to dock in accordance with claim 7, wherein said step of processing said image comprising localizing an image of a windshield of said aircraft is followed by a step of localizing corners of said windshield. 11. A method for enabling a pilot of an aircraft to dock in accordance with claim 10, wherein said step of localizing corners of said windshield is followed by a step of verifying said localizing an image of a windshield of said aircraft.

13. Apparatus for communicating position to a pilot of an aircraft approaching a docking station, comprising: camera means for deriving an image of at least a portion of said aircraft; computer means for:

(a) digitizing and enhancing said image;

(b) localizing an aircraft windshield in said image;

(c) forming a bounding box around said windshield; (d) localizing corners of said windshield;

(e) verifying said windshield localized in step (b) ;

(f) eliminating outliers;

(g) determining the aircraft-class;

(h) determining position and orientation of said aircraft; and

(i) displaying said position and orientation to said pilot.

14. Apparatus in accordance with claim 13, wherein said computer means further compares template information stored in its data base memory on aircraft configuration, based on model selection, and compares said information with said image in forming said bounding box in step (c) . STATEMENT UNDER ARTICLE 19

The amendments to the claims being submitted concurrently herewith consolidate the features of originally filed claim 3 into originally filed claim 1, and the features of originally filed claim 9 into claim 7. The dependency of claim 10 is changed from claim 9 to claim 7. No new matter is believed to be introduced thereby.