WO1997018523A2

WO1997018523A2 - Computer stereo vision system and method

Info

Publication number: WO1997018523A2
Application number: PCT/IL1996/000145
Authority: WO
Inventors: Moshe Razon
Original assignee: Moshe Razon
Priority date: 1995-11-14
Filing date: 1996-11-12
Publication date: 1997-05-22
Also published as: JP2000500236A; CN1202239A; AU738534B2; IL115971A0; KR19990067273A; WO1997018523A3; AU7331696A; BR9611710A; EP0861415A4; IL115971A; EP0861415A2

Abstract

The system and the method are intended to enable a robot (20) to see the environment in which it operates, by means of a pair of identical cameras (3). The pictures from the cameras (3) are processed by a coordinator/translator (59). The system includes a memory register (51) for movement identification at time intervals.A basic shapes registry (52) identifies shapes.

Description

COMPUTER STEREO VISION SYSTEM AND METHOD

Field of Invention

This invention relates to computer vision and identification seen in real time by means of cameras, a computer, etc. in general, and to 3-D computer vision by means of 2 or more "stereo" cameras. Background of the Invention

The need for computer vision is an every day necessity for resolving problems and for use by man practically in all fields: From research, astronomy, robotics, industry, agriculture, manufacture, services, driving, security, assistance to the blind, detection of phenomena, etc. Today, as both internal and external computer memory is growing larger, the physical dimension of computer components is being reduced continuously, and various types of high speed processors exist. Consequently, if one could create a system and a method for computer vision that would identify anything it sees in real time, everything would become easier and more simple. Present Situation

The computer vision existing today, which makes use of one or two cameras, focuses on the vision of individual, defined, known and mainly static object(s) and their identification is lengthy, comparative, partial and focused only, and does not analyze and identify everything that is seen contemporaneously with the photographing (filming). It requires and uses many devices such as: sensors, lighting, measuring gauges, etc., and is cumbersome, limited, insufficiently efficient and does not provide satisfactory solutions. Objectives of the Invention

The purpose of this invention is to provide a system and a method enabling to analyze and identify all the forms viewed contemporaneously and at the rate of filming, by means of cameras connected to a computer, means for computing of dimension, motion and other perceptible features. Furthermore_, the purpose of this invention is to enable any automated device (tools, computers, robots, etc.) to see by means of varied and appropriate means of vision, any tangible thing and phenomenon, the way man can see and identify them. To enable, help and carry out almost any action, task and work that man does but more accurately, efficiently, faster, better, etc., around the clock, anywhere and in places which are physically difficult to access, or dangerous, inaccessible, boring, etc.

Furthermore, the purpose of this invention is to allow man "see" by means of the invention, from a remote location, the space around a certain area, in a 3-D presentation, while even using multi-media with a multi-media software, by using glasses and/or special devices designed for 3-D vision, or a line for data transmission and a regular monitor screen.

The objective of this invention is to allow for the construction of devices according to it, while using equipment, systems, circuits and electronic components, basic software etc., existing on the market in one form or another, so that by their connection, combination, adaptation, extension, etc., devices can be created and/or assembled according to this invention, including the ability of adaptation to circumstances. Any average electronics engineer, computer engineer, system analyzer, etc., will be able to design, assemble and construct devices according to this invention.

The Invention

The invention consists of an innovative system and method for preferred stereo computer vision, including and operating as follows:

Foreword

1. Any material written as explanation for the software and which does not form part of the invention process as a whole, is a sample proposal only. Its purpose is to illustrate the process of operation of computer vision, and does not derogate from the essence of the invention.

2. The explanation given below is general, since there will be various types, models and sizes of computer vision systems (e.g., large, small, medium, Phillips screwdriver, etc.), in accordance with the data that the computer vision will have to transmit to the customer it serves, but the basic operation principle in all of them is identical.

3. Computer vision is a means of offering a service of "viewing". Its task is to see a section in space, to decipher it as necessary, to store in a register whatever needed and to transmit further the object it has seen or relevant data.

If required, it is possible to design data transmission in 3-D form as well, or "multimedia" (including data regarding the viewed object's shape, location, position, size, color, distance, etc.), all in accordance with the computer vision system's purpose.

4. Since computer vision is supposed to serve a customer, and the customer is supposed to perform limited operations, his work space is usually confined. Therefore, the computer vision system adapted for that customer will have to provide him with the data he needs for fulfilling his task. With the aid of appropriate data and software, it will be able to get "familiar" with the environment in which the customer operates, will recognize forms of objects which may be found in the area, the perception of which is important for that customer. Furthermore, the system will recognize the equipment and the tools in all fields of work encountered by the customer, which he has to deal with and use within the framework of his duty, also by grades of recurrence.

5. Computer vision can help with and use additional aids as well, such as of the memory system, of customer's measurements, or which may be external to the system or the customer. Between the customer's computer vision system and the aids there will be mutual relations, compatibility, reference, consideration, reciprocity, data transmission, etc.

List of Drawings (Figures)

Drawing (Fig.) no. 1 is a schematic sketch of a preferred system including possible connections according to the invention with two processors, in the form of a block diagram;

Drawing (Fig.) no. 2 is a schematic sketch of the parts composing a system constructed and operating according to the invention;

Drawing (Fig.) no. 3 is a horizontal section schematically illustrating the central viewing axes, the fields of sight and the parallel confining lines of the fields of sight and the optic images of the system described in drawing no. 1 ;

Drawing (Fig.) no. A/3 (X4 enlarg.) is the parallel optic images of drawing no. 3;

Drawing (Fig.) no. 4 is a horizontal section schematically showing the fields of sight and the shapes viewed (of the system described in drawing no. 1 ) in various sizes and at various distances;

Drawing (Fig.) no. A/4 (X4 enlargement) represents pictures of the shapes viewed in drawing no. 4, seen simultaneously by both the right and the left cameras of the system described in drawing no. 1. Photographing (Filming) and the Cameras

1. A pair of identical cameras 3 (drawing 1 and 2), aligned, coordinated in shooting angle including variable and enlargement/reduction, creating for the photographing cameras optical parallel fields of sight (drawing 3) concurrent on a common plane hereinafter referred to as the horizontal plane, in other words, the distance with regard to each field of sight line is identical in both cameras from (0:0) and up to (Y:X), and is a fixed distance M (drawing 3 and A/3) at any photographing distance. The vertical movement (perpendicular to the horizontal plane) of the cameras is constant and identical.

The photographs taken by the cameras are received in an input-memory A/51 and B/51 (drawing 1 ) in accordance with the rate of photographing and/or other, after being translated into computer language (digital) either by the cameras or by any other equipment. Adjustment by the computer vision system may be physical upon installation and/or at any other given moment.

2. Said two or more cameras, in a single enclosure or in separate packaging, are similar to video cameras, including CCD cameras and including cameras with integrated means for converting data received in the pictures to digital data and including one or more of the following: a) Adaptation for color photographing, at various speeds and at any light such as IR, visible and at any lighting conditions such as poor, by means of light amplification; b) Enlargement or reduction devices, including telescopic or microscopic means; c) Optic image 25 (drawing 3 and A/3) including the desired resolution, whether straight, convex, or concave.

3. The cameras will be provided with the required auxiliary equipment that will work according to instructions given by the computer vision system. a) Adjust lenses (to distance, light, etc. ). b) Move cameras (sideways, around, upward, downward, etc.). c) Set (telescope, microscope, star-light amplifier, etc.). d) Serve the computer vision system and the cameras and bring them to a situation where computer vision can be performed at any given moment and in an adequate, desired and accurate manner. Translator

A 59 and B/59 translator (drawing 1 ) can form an integral part of the cameras, and/or translate both ways to/from the computer vision system to to/from the operation, measurement, connection equipment to any external equipment, and/or any accessory means, it can serve all parts of the system or one part thereof only, or from any of the above, to the computer vision system.

The Process of Picture Input

1. Any picture (including color picture) received from any camera, will enter as it is to the input-memory A/51 and B/51 (drawing 1 ) to its designated place, and will occupy a space proportional to the camera resolution. Pictures of all the cameras will be entered to the input- memory in a separate place, such as an image simulating memory device ("screen cards", for example), with all the colors and at the rate of photographing/filming, furthermore, the picture of the leading camera A for example - will enter into the movement identification register at time intervals 54 (drawing 1) after color filtering 53 (drawing 1 ).

2. Transfer of the picture to its location in the spatial memory A/52 (drawing 1 ), to one camera or to two cameras for 3-D photographing, which would register all the pictures received from fixed, enlargement/reduction and/or other picture, in accordance with the consumer movement 6 (drawing 2) if the computer vision system makes part of the user, and/or with accordance to the movement of the cameras on the horizontal and vertical axes, the pictures will be saved as an outline picture in horizontal and vertical coordinates and will constantly be updated. According to the calculation of the motion, the movement, the horizontal and vertical coordinates of eveiy picture from each and every camera will be updated. They will also be updated in accordance with the movement of the space counters 57 (drawing 1 ).

3. The pictures received from the cameras in accordance with the photographing/filming rate are scanned by an A/55 processor (drawing 1 ), and during the scanning process, with regard to each shape in a picture that has not yet been identified or that has moved/advanced, the distance from the cameras is calculated, as well as any other data that can be calculated and/or that can result from the picture, such as color, and furthermore, the pictures are transferred to the spatial-memory register to the appropriate location, and to any other place, such as the consumer. Method of Scanning and Matching

1 . Scanning of picture data according to the rate of photographing/filming and/or other, assuming that scanning from left to right and from the top downward begins with the picture of the right camera, hereinafter the leading camera towards pixel (0:0), then towards pixel (0:0) in the left camera picture, until point (Y:X) in every camera picture is reached. The scanning data of each and every camera separately will be matched with the spatial-memory pictures, if a 3-D presentation is required - both of them and not only the leading one and in accordance with the coordinates and the space-counters, when there is no matching, they will be updated and matching will be found between the resulting pictures.

2. Pictures that have changed and the dots in them must be matched for purpose of distance measurement and/or contour identification, in addition to the standard matching they will also be matched with one another.

3. Scanning for purpose of matching and calculation of the distance L (drawing 4 and A/4) to that point D in the horizontal line X and in row Y in the pictures of both cameras, with X I being the number of pixels from point (Y:0) in the leading camera and X2 being the number of pixels from point (Y:0) to the same dot in the other camera, then the following difference results: Wr=X2-X l, enabling distance calculation.

4. The pixels representing a point viewed in the right and left cameras, and the above point is located beyond the congruence distance (a pixel represents 2M and 0=Wr<l/2), from this distance and farther a variation of distance cannot be detected. The same point in the pictures received from both cameras is located at the same place (Y:X) and the pixels representing them will be identical.

5. From the congruence distance, the shorter the distance gets, the bigger the difference between the pixels Wr grows. In other words, the passage from a distant shape to a close shape will increase the difference between the pixels Wr, while in the passage from a close shape to a distant shape the difference between the pixels Wr will decrease. Consequently, there will be dots that will not appear in the picture of the leading camera during close-up and in the picture of the other camera when getting far, but which the computer vision system will take into account when performing the matching. These depth dots, when consecutive, also form the contour- line and/or dividing line between the different areas.

6. At first sight the vision computer will hinder the entrance of additional pictures except for the motion-detection register at intervals and will perform a scanning for pixels transmission and matching of first-sight pictures.

The computer vision system will match a pixel (0:0) in a picture of the leading camera with the same pixel in the picture of the other camera; if there is matching then the difference will be O^Wr and the system will pass to the following pixel in both pictures that has to be transferred and matched, and so forth, until an non-matching is found, ln the event that there is no matching, the difference is O≠Wr, the computer vision system will match first the pixels of the picture of the leading camera, the one after the other, with the same pixel in the picture of the second camera which there was no matching with until matching is found, and will continue transmitting and matching accordingly, or until the difference is 0=Wr, and if no matching is found, then the computer will return to the same pixel in the picture of the leading camera which there was no matching with and will match the pixels of the picture of the second camera, one after the other, with the same pixel until matching, and so forth, using the definition of the area limits detected based on the motion- detection register at intervals.

7. Representing pixels that are consecutive in the same line in both pictures at identical distance will be matched with one another according to order, if the two pixels do not match, one should assume that the distance of one of the points has changed, and the process of matching for calculation will act accordingly.

8. In a standard matching screening, when there is no matching between the input picture and a spatial-memory picture, the computer will begin matching between the pictures received from the same place where there was no matching in each picture, and based on previous information, will detect whether any form has moved, has changed or is new, and will calculate the distance for each point in every matching, until matching is found between the received picture and the spatial-memory picture and will proceed as usual. Distance Calculation Process

1 . For purpose of calculation and picture handling, coordinates 57 can be assigned counters (drawing 1 ) (indicating lines), such as from (0,0) to (Y,X) or from (-Y,-X) to (Y,X), that will be a sort of grid on which the spatial-memory pictures are laid. For example, the "0" point will be the central reading point Op (drawing 3 and A/3) (the central point of the picture) for each camera on the horizontal level, with the cameras leveled and the direction being northward in accordance with the compass direction "North".

When the lateral motion of the cameras eastward or upward is positive and westward or downward is negative, the computer is automatically updated (by the size of one pixel angle, for example), according to the motion in the picture by an external factor or by a compass or an accessory system, and will receive the cameras data accordingly. One coordinate indicates a peripheral horizontal and the other indicates a peripheral vertical point.

2. For purpose of orientation in the area there will be, for example, three dimension-counters 57 (drawing 1 ) that will relate to the central line of sight (the central point of the camera) of the leading camera, for example: counter ( 1 ) that will count the camera' s motion advancement "east to west" from the principle "0" point that will be set; counter (2) that will count the camera's motion advancement "north - south" from the principle "0" point that will be set; counter (3) that will count raising and lowering (height), with the sea level, for instance, or any other level established, forming the "0" point.

3. For every picture received into the input-memory, a calculation will be carried out for matching of counters to coordinates in relation to the first point identified in it, and the resulting picture will be transferred to the spatial-memory picture in accordance with the calculation and the relative coordinates.

4. In case of use of video cameras (CCD) or any similar camera, the rotation of the camera around itself on the horizontal or vertical plane enables to map the peripheral picture (in a standard photographing) with coordinates according to the number of pixels in the camera picture. Thus, the space is mapped with coordinates representing angles in the envelope of a sphere whose center is the angular point of the photographing angle of the camera. Upon completion of a whole round turn or arrival to the end of the photographed/viewed space along the horizontal or vertical axis, the computer vision will identify, for example, the number of pixels/coordinates, and will "know" that it must return to the coordinate at the beginning of the course or of the photographed/viewed space in motion.

5. During enlargement, for instance, when the camera zoom is activated in order to decrease the angle of the photographed picture, the coordinates that will be added accordingly will be a decimal fraction, e.g., three figures after the decimal point.

6. The number of pixels occupied by every shape in the space will depend upon the number of pixels in the optic image, the shape's physical dimension, its proximity to the camera and the viewing-angle opening. The higher the number of pixels and the smaller the viewing-angle opening of the optic image, a larger number of pixels will represent a form of the same size in nature (at constant distance from the camera), and the identification of the shapes will be improved. Furthermore, the larger the enlargement is, the larger the number of pixels representing the same dimension of a shape will be.

Method of Calculation

The calculation of the distance L from the cameras to that matching point for which the calculation is made is based on the difference Wr, on the identity, the physical dimension and resolution of the optic-image, the matching between the cameras and the constant distance M (the parallel-deviation).

1. The formula of calculation of the radius r₀ from the angular part of the viewing angle α and up to the optic-image, the width (on the X axis) of which being X₀, and at enlargement/reduction A, is as follows: r₀ = 360*A*X_o/2*π*α (360 is the angle of a full circle turn); and K„ will be all the constant data in the formula and will be: K₀ = 360*X„/2*π

2. The radius r from the viewing-angle α and up to any point viewed, X being the number of horizontal pixels (on the X axis) in the optic-image and the formula for calculation will be : r = 360*A*X*M/2*π*Wr*α and K will be all the constant data in the formula and will be: K = 360*X*M/2*π

3. Ki is a constant at any rate, directly proportionate to the enlargement/reduction A and there is an inverse ratio with the sight-angle α and its formula is: Kι =A/α

4. The formula for calculation of the distance L will be: L = Kι(K/Wr-K„) (factor of the variable Wr).

5. The calculation of any indirect distance L_t, where Qj is the size of a pixel representation at the location of any form with standard sizes such as the height of a known man.

The formula for calculation will be:

L, = 360*A*Q, *X/2*π*α - r₀ = K, (K*Q, - K₀)

Thus it results that: the calculation of the distance to the points enables to calculate the size of the pixel representation for the point it represents, helps/allows to detect frames of areas the points belong to, and any datum requiring any calculation such as width, height, depth, size, angle, characteristics and any other definition such as status (fluttering, hung, etc.).

The Process of Area Separation and Data Collection

Each of the areas 1 1 and 12 (drawing A/4) will be attributed a code (provisional name) for identification. An area may embody in itself other areas of various dimensions. Each area must usually have a contour (except for a dot and a line), any characteristics such as a fixed distance of the area and difference from the environment or different color or contour-line and/or separation between the area and surrounding area, area movement and area condition (fluctuating, hung, etc.). These, as well as other data will be identified and defined by the computer vision system by means of a B/55 processor (drawing 1 ).

1. The color separation 58 (drawing 1 ) software 56 (drawing 1 ) will process the colors of the received picture and will separate the colors, assist in area definition using other previous area definitions, and for each dominant color of an area it will detect an additional definition of the rate of that color out of all the colors in that area. For example, 4 forms of division are possible: a. The amount of color is: 100% - 90% of the colors in the area; b. The amount of color is: 90% - 70% of the colors in the area; c. The amount of color is: 70% - 50% of the colors in the area; d. The amount of color is 50% and below of the colors in the area. Furthermore I would also like to put an emphasis on the manner of dispersion of the color (sporadic dots, spots of a certain size, etc.).

2. Concurrently with the reception and in addition to the normal reception, the picture of the leading camera undergoes color filtering 53 (drawing 1 ) (saving of resources), and is saved in the motion-detection register at time intervals 54 (drawing 1 ), is duplicated and examined at regular time intervals, in photography in fixed enlargement/reduction and in desired and known enlargement/reduction (there may be several). Based on previous data taken from the received pictures (such as color, shade, calculations, etc.), data such as motion, movement, speed, angle (angle in relation to the viewed object and angle of the viewed object in relation to the cameras and others in space), as well as movement within the area (such as eye movement), in the area envelope (such as a position, hand movement, etc.) can be detected and/or calculated for area/form 1 1 and 12 (drawing 4), to check whether the basis is stable and the upper part moves (e.g., a tree), whether the area movement flows (e.g., a river), direction (north, south, east, west, upward, downward, forward, backward, right and left), the speed, as well as the type of motion and/or movement and condition of the form, and. based upon the space-counters, also the location of the computer vision and the areas and forms, and also help in defining area-frames that have not yet been detected.

3. A register for fundamental and necessary basic-shapes C/52 (drawing 1 ) (e.g., geometrical) which will be made of contour-lines - in black/white - with lines here and there in them, and/or few dots, in the simplest form of the shapes compatible with the computer vision system, its task and objective. The basic-shapes will be saved in a certain order allowing immediate access for comparing the input shapes after they undergo appropriate frame treatment, in size matching against them in order to obtain (approximately) comparative data regarding the treated shape (a shape can match several basic-shapes). For purpose of setting definitions for the area-contour such as: rectangular, grade, dome, pointed, etc., the memory for the basic-shapes (in principle no larger than 256), will depend on the number of saved shapes B/52 (drawing 1 ) (there may be hundreds of thousands).

4. Table(s) 58 (drawing 1) such as the "true" table are compatible with said computer vision, one after/inside the other for the area/shape data included in them and/or obtained from one table and these tables use said data in a particular order, in part with regard to the present table, the following table, etc.. for detection of features, definitions and conclusions. All or any of the features, definitions and conclusions gathered are added and/or join the key-element data for detection, matching and identification. Additional features detection, definition setting and drawing of conclusions will be enabled, such as: inanimate object, a car, a hovering object, a living thing, for example: if the area is between a given size and another size, if it moves and there is motion and/or movement in the envelope and/or inside it, etc., where each table deals with a certain subject such as handling of possible motion and/or movement, handling of possible colors, handling of the change of distance from the surrounding environment, etc.

5. A data table 58 (drawing 1 ) for recurrent forms found at a photography distance adjusted to the stored shapes and the presence of which in the picture is highly reasonable. In these cases, immediately upon receiving the picture and calculating the size and additional individual data such as color, the computer vision system will check out their recurrence and they will constitute key-elements for detection and identification as against the register of stored shapes.

6. If required, necessary and possible, identification data 57 (drawing 1 ) such as heat, radiation, voice, taste and smell will also be added to the existing identification data.

7. Auxiliary data 57 (drawing 1 ) can be obtained from an internal factor (such as space-counters, a compass, etc. ), an external factor (such as from the cameras, enlargement/reduction for example, from the user, for example speed of movement, etc.), a printer, a speedometer and/or any accessory to the system, such as a compass, a telescope, a distance meter (hereinafter - auxiliary means), as well as other associated data, in accordance with the computer vision requirements, the desired rate and the computer language (digital). 8. The memorized and recognized forms B/52 (drawing 1 ) of pictures, maps, signs and/or text data near/above any form that can be defined and/or given a name and/or identified independently (such as an inanimate object, a plant, a living thing, background, sign, phenomenon and any other thing), will be memorized in a particular and known order matched with the key-elements, in one of the following four forms: a. As received by the camera including the colors.

B. Black/white pictures.

C. In a simplified form - (sketch).

D. In form of a table with a name and data, or in a worded form such as a card-index. The pictures and maps will be in a photography standard that depends upon the size of the shape and the photographing distance. A shape memorized in the form of a picture can be stored in several places - and in each place it will be saved from the angle of sight of a different side, so that it can be identified from every direction. For example, for a 3-D picture 6 various pictures may be stored - one of each Cartesian direction.

The Process of Area Identification in a Picture and Matching With Stored Shapes

1. Since the stored shapes cannot express every possible size of that shape at various distances in the space, a "logarithmic" system and/or any other form of processing 56 and B/52 (drawing 1 ) will create size matching, between the distance of photographing of the real shape received in the picture at a given moment in space, and the appropriate distance of photography of the stored shape.

2. An area can contain additional areas within itself. Matching and identification will start from the largest area and will proceed towards the smaller areas. Area identification may save the need for identification of internal areas if they are part of the data of an identified shape within the area more generally speaking, yet, if they are indispensable for completing the identification (e.g., position) they will be identified.

3. The key-elements 58 (drawing 1) from data, features, definitions and conclusions, arranged in a special order adjusted to the same order in the data base of the known stored shapes, allow fast classification and arrangement of the key elements that have been gathered and which allow detection, matching and identification of unidentified areas. As, at normal sight, the unidentified areas are very few at any photographing/filming rate and therefore identification is almost immediate, and the detection is carried out like the detection of a word in a dictionary, the category being language and the order of the key-elements being alphabetical and the entire word being a key, a component can be absolute, between and between, or possible. Identification is made as against the register of recognized stored shapes that includes data on shapes such as in form of pictures, maps, signs and/or text data (in form of table, near, above) saved in the register, and when pictures and maps are concerned - also by size and photographing/filming angle in compliance with the stored shapes photographing standard depending upon the shape's physical dimension, the photographing distance and the purpose of computer vision.

4. Shapes requiring a deeper analysis - of secondary areas, parts, secondary details, age, type, position, etc., will undergo treatment accordingly, providing more identifying and detailed data which include any data the vision is requested to provide to the consumer it serves.

Sign Detection and Identification

A software for detection and identification of significant signs in the picture.

For example:

1. The software program will detect, match and identify in the picture signs (such as marking, writing, street sign, traffic light, etc. ). According to the identification data the software will know for example, where within the shape there is writing or a certain sign (e.g., location of a vehicle registration number) and will know to access that place for purpose of matching, identifying and matching of size to the memorized writing or sign.

2. In order to help in the process of detection, there will be catalog(s) or a dictionary [signs, writing (words, names in the local language or in English, etc.), forms, street signs, traffic lights, etc.].

Partial Vision

1. Partial vision occurs when part of the shape is hidden, and based upon the visible part the whole shape must be detected and identified.

2. In these cases, the computer vision detects those parts that appear in the picture and will indicate with regard to them all the details concerning that specific shape and the visible part thereof. llnidentified Shape

During the matching and identification stage, if no shape matching the shape in the picture is found.

1. That shape will be saved (area, part, detail, etc.) in the register and at the place designated for it with all the relevant data.

2. The computer vision system will count how many times it encounters that shape. If it does not encounter it anymore for a reasonable period of time, the system will use that place in the register for other similar cases.

3. The computer vision system may include an option for transmitting and receiving message to or from an external factor, such as facsimile interface, a monitor and a printer 31 and 32 (drawing 2), a voice system, computer communications, etc. and use it for identifying a shape which has not been identified according to the system memory.

Map Analysis

A computer vision system that serves a device moving in a building, a city, a field, a country, etc., will be able to analyze a viewed picture or course and accurately compare the data with the same place in a map memorized in the system register.

1. The computer vision system will know its initial place in the map according to a space-counter or from an external factor, or will find its location on the map by identifying data from the real picture. For example, the system will perform a reading of the name of a street and the name of another street that crosses it and two house numbers on the same side of the street and will identify the place by means of an appropriate plan, including the city where it is found.

2. The map will provide data on course(s) and alongside the course(s): a. Data on obstacles, pedestrian crossing, rail road, junction, intersection, slope, traffic signs, road marks both on an outside the course, etc. The data appear on the drawing, the map, etc., some of them contiguous with the course, while others in a different manner. B. Data may be registered in form of writing, codes that refer to shapes or data table(s) (e.g. writing, codes, signs, traffic signs, etc.). Rotary Movement

The computer vision system is supposed to see a space that varies as a result of the movement of the viewed shapes and/or as a result of the tilting of the ca eras upward, downward and sidewise and/or as a result of the user's turning around any axis and/or of change in the movement of the user, whom the cameras are connected to.

The spatial memory pictures will cover all the angles of the circle and the desired field of sight of said computer vision system. Every time the camera receives a picture at any angle, the system detects a change in the matching between the last picture and the picture memorized in the spatial memory with the appropriate coordinates, and replaces the section of the last picture that is different in that place, with the section in the picture saved in the spatial memory.

Saving and Presentation of Space in 3-D and Multimedia

From each pair of computers, the computer vision system will preserve full spatial memory pictures.

1. The pictures will be served as a whole like in a circular movement and every picture will be saved separately, creating a complex space-picture around each camera.

2. The space-pictures will allow the computer vision system transmit, display or present to any other factor the data of the space it moves or is found in, by means of a 3-D display software program (which must not necessarily form part of the computer vision system), including display of identification details and data.

3. Possible presentation by connecting an external display or any other type of presentation, which will enable to see in 3-D the space around the computer vision system, as in real life.

4. It will be possible to transmit data to a screen or to any other type of display by means of a 33 interface (drawing 2), either by physical transfer or by transmission, this not making part of the computer vision system, but part of the user or a supplement to the system or the user.

Area/Shpae Follow-up

1. For purpose of follow-up, unidentified areas will be stored under a temporary name and the identified forms will be stored under their name, with reference to the coordinates of the first point scanned in the area/shapes in the picture of the leading camera and additional points in the envelope and/or the shape they will be indicated, in order to follow the change in the location of the area/shape, and in the position and/or movement and/or motion, and/or in parts of the shape in the envelope and/or within the inside (e.g., animate thing, vegetation, etc.), at any time. 2. All data collected from the picture, from the calculations, from movement identification at time-intervals and from the saved forms data base, and from any means of identification with regard to the shape, are preserved in case of need until the shape leaves the spatial memory picture of the camera(s). Saving in the Register

If required, the computer vision system will transfer for storage pictures (full, sample, partial, etc.), or shapes (which are pictures, graphs, signs, codes, etc.) to a register for use at a later date. The computer vision system will have to store or memorize pictures in order to be able to restore them whenever needed at a later date, and will perform this in the following manner:

1. Will transfer full and consecutive pictures and/or data that define the pictures from a certain moment and up until another moment to the appropriate register (for example, the memorization of a certain event that one wants to see before the event, during and after the event in order to understand a certain process or evidence, etc.).

2. Picture or map update and replacement in the register with the new and more updated one.

3. Process and/or data on anything viewed and/or any shape, with any method.

External Connections

1. The computer vision system will have means of connection 50 (drawing 1 ) to which a computer, a robot, a user, a device 20 (drawing 1), a printer, a fax machine, controlled machines, a microphone, loudspeakers, a communications line, etc., can be connected.

Of course, every computer vision system will only have the necessary means of connection, for purpose of transfer of vision data, data exchange, receiving of directions, instructions, etc. according to need (there may be one input/output and a first code for opening or tuning).

2. Coordinators A/59 and B/59 (drawing 1) will coordinate the connection between the computer vision system and the cameras, user, etc., and from any of them to the computer vision system. - 18 -

Transfer

After having identified the details and their data in accordance with the objective and requirements of the user it intends to serve, the system will transfer the required and needed data to the user by means of appropriate means of transfer or interface.

1. Based on the picture received, calculations, information stored in its memories, processing, decoding, additional data and various processing, the computer vision system will provide data to the user it serves and will submit him with all the direct and computerized data with regard to each and every object (such as: measurements, distance, position). Reporting will be continuous or upon requirement.

2. The computer vision system will be able to identify the picture as a whole and in detail, as well as the location of each and every detail in the picture and will be able to transfer the picture as required, in form of instructions, writing, speech, fax, etc. or as a whole, with analysis of details of he various shapes, all according to needs.

3. The user who receives the report from the system should be prepared for reception and processing of the received data, in order to be able to act accordingly. If the user uses computer vision for purpose of performance of certain operations, he can make use of the system for accurate motion, direction, etc.

4. The spatial memory pictures will be transmitted to any user as they are, directly or from the spatial memory to a monitor screen or an appropriate device, as required, including for purpose of 3-D presentation and/or along with the identification data.

Epilogue

1. The computer vision system shall be appropriately protected as far as possible against blinding light, light flashes of any kind whatsoever including laser and at any location and against any other possible physical injury.

2. The computer vision system will be compatible and will operate in accordance with the user's needs and requirements.

3. The computer vision system will comprise software programs 56 (drawing 1 ), electronic and general circles by which the system will function, and a software program which will adjust the size and photographing/filming enlargement/reduction data and in accordance with the standards of size and filming angle of the stored shape, software for 3-D presentation and multimedia, as well as any other software program that should be required.

4. The received pictures, data, the calculation of the pictures in the spatial- memory and any other information concerning the received pictures or maps or data stored in any of the system memories may be sent out as are, including the data, the data only, and/or input or spatial stereo pictures, to any user such as a robot, according to requirements, design and with any method, and the user will be able to draw data as he wishes and in accordance with the system design.

5. All the above mentioned operations will be carried out by A/55 and B/55 (drawing 1 ) processors, additional processors if required, as well as means of processing and computing. They will use computer devices, components, one or more electronic circles and alike, and any combination thereof required for and adapted to the purpose of computer vision.

The work distribution among them will be such, that each component, part, electronic circle, system, etc., will perform its task without interfering with one another and there will be total coordination and compatibility among all of them, for example, processor no. 1 informs processor no. 2 by means of a component, electronic circle, etc., that it has finished its part in operation A and that processor no. 2 may continue carrying out its part in complex operation A, etc.

Commercial Implementation

1 . Computer vision may be used: a. As a viewer (watcher, analyzer, decoder, reporter, etc.). b. As a viewer that collects data and preserves them in any data base or register, in any way (fully, partially, by any form of classification or sorting, etc.).

2. Due to the multiplicity of possibilities of use of computer vision, several standard computer vision systems may be used, where each standard system is adapted so as to provide certain services and perform certain tasks, and thus it will be available "on the shelter", while for special tasks, and thus it will be available "on the shelter", while for special requirements, a computer vision system adapted to specific needs will be designed.

3. Each computer vision system will be assigned standard and specific abilities.

4. It is possible to adapt the computer vision system to the specific needs and requirements of any user who is a designer or a constructor during the stage of design, and to integrate it as part of the user, or as a separate unit that serves the user.

Claims

1. A stereo computer vision system including: a. A pair of identical, aligned, coordinated cameras at fixed distance M, with coordinated photographing/filming angle and enlargement/ reduction, creating for the photographing cameras optical parallel fields of sight, coordinated, aligned, with identical field of sight line in both cameras from (0:0) and up to (Y:X), at a fixed distance M at any photographing distance, and the pictures of said cameras are received by means of input-memory devices in the computer, translated to computer language; b. Based on step A, the pixels of the received pictures are matched by the available means, at photographing rate and/or at any other rate to each of the previous picture(s) existing in the system's spatial memory registry according to the coordinates and space-counters, and when no matching is found there, the system performs updating and proceeds with matching of two pictures that have been input; c. Based on steps A and B, with regard to unidentified areas or areas in which any movement takes place and which are seen simultaneously in both cameras, the system looks for various features of the picture, such as color, and based on the difference in the count of pixels in each picture taken by a camera from the beginning of a line up to that matching point, calculation of the distance is performed, and these calculated distances enable to calculate data such as the size of a point in space represented by a pixel, and based on this, the size of the area and the distances; d. Based on steps B and C, in matching points of unidentified areas with stored pictures, there will be points around the area that will not match due to a change in the distance between the area and the surrounding environment and the distance between the two cameras, these depth dots creating a contour-line and/or dividing line in addition to the approximately similar distance to the area dots, and with regard to areas at similar distance, changes in color or shade between areas and/or area motion/movement, will allow, by means of the available means, to detect and define a frame or countour for each and every area; e. For purpose of collection of additional data enabling to perform with the means available calculations, matching, setting of definitions and drawing of conclusions, the picture received from one of the cameras undergoes color filtering and is entered into the movement- identification register at time-intervals, after being duplicated and is then examined at fixed time intervals and based on steps B to D, the frame or countour of the area that has been detected is matched in size with the basic-shapes register, said data from these registers enable to calculate, for example, the size of an area/shape and speed of movement/motion in time and additional data, part of said data that have been input and part of the data that have been calculated/ detected previously allow matching as against a register of table(s) such as "true" for setting definitions and drawing conclusions with regard to other key-elements, such as: inanimate object, fluttering, animate, for example, if the size of the area goes from size to size and there is movement and/or motion in the envelope and/or inside it, etc.; f. Said key-elements drawn from data, features, definitions and conclusions, ordered in a particular order, allow, through the means provided, to detect, match and identify areas as against a register of stored shapes; g. The unidentified areas will be stored under temporary names and the identified shapes will be stored under their own name and with reference to the coordinates data in each scanning of the first dot and additional dots in accordance with that type of shape/area, in order to follow and observe any change in location, position and/or motion and/or movement in the envelope and/or inside the shape, at all times.

2. The computer vision system claimed in claim 1, wherein said pair and/or more cameras can be packaged in a single casing or separately, are similar to video cameras, including CCD cameras and including cameras in which there are combined means for converting data received from pictures taken by the cameras into digital data, and includes one or more of the following: a) Adaptation for color photographing, at various speeds and at any light such as IR, visible and at any lighting conditions such as meager, by means of light amplification; b) Enlargement or reduction devices, including telescopic or microscopic means; c) Optic image including the desired resolution, whether straight, convex, or concave.

3. A computer vision system as in claims 1 and 2, wherein the main said means for gathering detected and/or calculated data, features, defined, drawn and/or viewed, allow for immediate access to detecting said data by means of said means, as the case might be, according to classiflcation by key-elements arranged in one or more hierarchic order established in advance and absolutely, between and between and possibly, either internal or external, they include one or more of the following: a. Input-memory for receiving the photographed/viewed pictures; b. Spatial-memory for one camera or two cameras, for 3-D representation, that collects all the pictures received by enlargement/reduction, fixed and/or other photographing, the pictures will be saved as an outline picture expressed in horizontal and vertical peripheral coordinates and they will be constantly updated; c. A registry in the form of a table(s) for known stored shapes, that will included pictures, maps, signs and/or text data close above each shape that can be defined and/or given a name to and/or identified separately (such as an object, a plant, a living thing, a background, a sign, a phenomenon, or anything), stored in a special order and known, adjusted to the key-elements, the pictures and maps will be at photographing standard depending upon the size of the shape and the distance of shooting; d. A register for movement detection at time intervals for one of the pictures obtained after undergoing color filtering and checking at regular intervals for purpose of detecting and calculating, such as movement, motion and speed; e. Basic shapes register, preferably in black/white such as known geometrical shapes, for example, rectangular, stair, etc.; f. A data base containing the following registers: 1. A register for color separation; 2. A register for storing contours of areas and areas prior to identification; 3. A register for table(s) such as "true" table, arranged the one after/inside the other for setting definitions and drawing conclusions, for data, definitions and conclusions; 4. A register for data table for recurrent shapes; 5. A register for key- elements; g. Registers for accessory data: 1. Internal, such as from space-counters, from compass, etc.; 2. External, such as from the cameras, for example, enlargement/reduction, from the user, for example, speed of movement, etc.; 3. Additional accessory data.

4. A computer vision system as in claim 1 to 3, wherein the said data are anything that can be defined as a datum, that can be received from a picture, from a calculation, detection, characterization, from definitions and conclusions, from space-counters, from an external element such a printer, a speedometer and/or an accessory to the system such as a compass, a telescope, a distance meter (hereinafter, auxiliary means), in accordance with the computer vision requirements and the desired rate and the computer language (digital), and any of the following, or any combination thereof: a. Measurements (e.g., distance, width, height, depth, size, ratio and angle), color (e.g., level of color percentage, color dispersion, sporadic dots, spots), frame (contour), motion, movement, speed, location and angle (e.g., with area, shape, cameras) with relation to direction (e.g., north, south, east, west, upward, downward, forward, backward, right, left) and the system of coordinates, or to cameras or to areas or to shapes, of the area or the shape or parts of the area, or the shape; b. Features, phenomena, attributes, detection of heat, radiation, voice, smell and taste; c. Definitions and conclusions (e.g., temporary name, rectangular, stair, object, car, fluttering, flame, flowing of water, living thing, walking on four legs, growth, private type vehicle right side, change in time).

5. A computer vision system as in claim 4, wherein are included means for identification and calculation of the said data of areas, features, defmitions and conclusions with regard to unidentified shapes and/or in which some motion occurs (e.g., motion, movement) in the entire area/shape and/or in the envelope and/or inside, and which are seen in the picture or part thereof, and all of them, a part thereof, a combination thereof are joined with the key-element data and is performed by means of: software programs, electronic circles and rules by which the system will operate, and will match size, distance, enlargement/reduction and photography angle of the viewed shape, in accordance with the size, the distance and the shooting angle of the stored shape, and a software program for the transfer, 3-D presentation and in form of multimedia.

6. A computer vision system as in claims 4 and 5, wherein the data, features, definitions and conclusions will be matched with a register/memory with all the data or a part thereof, with the said means and software programs for comparing and matching, and wherein identification by key-elements according to classification in one or more pre-determined hierarchic order(s) will be performed, either absolutely, or between and between, or possible, said key-elements will allow identification of areas, shapes or parts thereof, in relation to infoπnation known in advance or from previous photographs, with regard to said data of known areas, shapes or part thereof, as the case may be.

7. A computer vision system as in claims 1 to 6, wherein a computer, computer components, components, electronic circles, a processor, a computerized data processing system and alike, one or more of them, form one or more of the said means or any combination thereof; the said means known software programs, new and compatible; said means for converting data obtained from photographed pictures into digital data; said means for matching and definition of color identity with regards to the same viewed point and/or points generated in the photographed pictures; said means for calculation of distance and dimensions, for identification of features, definitions and conclusions; said means for matching of size and matching of photographing angle; said means for movement detection at time intervals, type of movement, motion; said means for handling of "true" table(s); said means for handling recurrent shapes; said means for data collection for key composition; said means for comparison and matching; said means for storing various types of information; said means for receiving, transmitting and drawing of data; said means including power supply, data protection, protection of said means, the system and alike.

8. A computer vision system as in any of claims 1 to 7, wherein the received and stored information that has been collected before, during and after identification, including calculations, known data, features, definitions that have been collected such as with regard to areas, shapes, is immediately or after a short while transmitted forward in form of information, in a regular manner and/or by stereo in form of multimedia and/or in 3D form, and which the system preserves and/or provides and/or which are accessible upon request and/or automatically to the user, such as a robot, a device, a blind person, by means of adequate interfacing means.

9. A stereo computer vision method including: a. A pair of identical, aligned, coordinated cameras at fixed distance M, with coordinated photographing filming angle and enlargement/ reduction, creating for the photographing cameras optical parallel fields of sight, coordinated, aligned, with identical field of sight line in both cameras from (0:0) and up to (Y:X), at a fixed distance M at any photographing distance, and the pictures of said cameras are received by means of input-memory devices in the computer, translated to computer language; b. Based on step A, the pixels of the received pictures are matched by the available means, at photographing rate and/or at any other rate to each of the previous picture(s) existing in the system's spatial memory registry according to the coordinates and space-counters, and when no matching is found there, the system performs updating and proceeds with matching of two pictures that have been input; c. Based on steps A and B, with regard to unidentified areas or areas in which any movement takes place and which are seen simultaneously in both cameras, the system looks for various features of the picture, such as color, and based on the difference in the count of pixels in each picture taken by a camera from the beginning of a line up to that matching point, calculation of the distance is performed, and these calculated distances enable to calculate data such as the size of a point in space represented by a pixel, and based on this, the size of the area and the distances; d. Based on steps B and C, in matching points of unidentified areas with stored pictures, there will be points around the area that will not match due to a change in the distance between the area and the surrounding environment and the distance between the two cameras, these depth dots creating a contour-line and/or dividing line in addition to the approximately similar distance to the area dots, and with regard to areas at similar distance, changes in color or shade between areas and/or area motion/movement, will allow, by means of the available means, to detect and define a frame or countour for each and every area; e. For purpose of collection of additional data enabling to perform with the means available calculations, matching, setting of definitions and drawing of conclusions, the picture received from one of the cameras undergoes color filtering and is entered into the movement- identification register at time-intervals, after being duplicated and is then examined at fixed time intervals and based on steps B to D, the frame or countour of the area that has been detected is matched in size with the basic-shapes register, said data from these registers enable to calculate, for example, the size of an area/shape and speed of movement/motion in time and additional data, part of said data that have been input and part of the data that have been calculated/ detected previously allow matching as against a register of table(s) such as "true" for setting definitions and drawing conclusions with regard to other key-elements, such as: inanimate object, fluttering, animate, for example, if the size of the area goes from size to size and there is movement and/or motion in the envelope and/or inside it, etc.; f. Said key-elements drawn from data, features, definitions and conclusions, ordered in a particular order, allow, through the means provided, to detect, match and identify areas as against a register of stored shapes; g. The unidentified areas will be stored under temporary names and the identified shapes will be stored under their own name and with reference to the coordinates data in each scanning of the first dot and additional dots in accordance with that type of shape/area, in order to follow and observe any change in location, position and/or motion and/or movement in the envelope and/or inside the shape, at all times.

10. A computer vision method as in claim 9, wherein said pair and/or more cameras can be packaged in a single casing or separately, are similar to video cameras, including CCD cameras and including cameras in which there are combined means for converting data received from pictures taken by the cameras into digital data, and includes one or more of the following: a) Adaptation for color photographing, at various speeds and at any light such as IR, visible and at any lighting conditions such as meager, by means of light amplification; b) Enlargement or reduction devices, including telescopic or microscopic means; c) Optic image including the desired resolution, whether straight, convex, or concave.

11. A computer vision method as in claims 9 and 10, wherein the main said means for gathering detected and/or calculated data, features, defined, drawn and/or viewed, allow for immediate access to detecting said data by means of said means, as the case might be, according to classification by key-elements arranged in one or more hierarchic order established in advance and absolutely, between and between and possibly, either internal or external, they include one or more of the following: a. Input-memory for receiving the photographed/viewed pictures; b. Spatial-memory for one camera or two cameras, for 3-D representation, that collects all the pictures received by enlargement/reduction, fixed and/or other photographing, the pictures will be saved as an outline picture expressed in horizontal and vertical peripheral coordinates and they will be constantly updated; c. A registry in the form of a table(s) for known stored shapes, that will included pictures, maps, signs and/or text data close above each shape that can be defined and/or given a name to and/or identified separately (such as an object, a plant, a living thing, a background, a sign, a phenomenon, or anything), stored in a special order and known, adjusted to the key-elements, the pictures and maps will be at photographing standard depending upon the size of the shape and the distance of shooting; d. A register for movement detection at time intervals for one of the pictures obtained after undergoing color filtering and checking at regular intervals for purpose of detecting and calculating, such as movement, motion and speed; e. Basic shapes register, preferably in black/white such as Icnown geometrical shapes, for example, rectangular, stair, etc.; f. A data base containmg the following registers: 1. A register for color separation; 2. A register for storing contours of areas and areas prior to identification; 3. A register for table(s) such as "true" table, arranged the one after/inside the other for setting definitions and drawing conclusions, for data, definitions and conclusions; 4. A register for data table for recurrent shapes; 5. A register for key- elements; g. Registers for accessory data: 1. Internal, such as from space-counters, from compass, etc.; 2. External, such as from the cameras, for example, enlargement/reduction, from the user, for example, speed of movement, etc.; 3. Additional accessory data.

12. A computer vision method as in claims 9 to 10, wherein the said data are anything that can be defined as a datum, that can be received from a picture, from a calculation, detection, characterization, from definitions and conclusions, from space-counters, from an external element such a printer, a speedometer and/or an accessory to the system such as a compass, a telescope, a distance meter (hereinafter, auxiliary means), in accordance with the computer vision requirements and the desired rate and the computer language (digital), and any of the following, or any combination thereof: a. Measurements (e.g., distance, width, height, depth, size, ratio and angle), color (e.g., level of color percentage, color dispersion, sporadic dots, spots), frame (contour), motion, movement, speed, location and angle (e.g., with area, shape, cameras) with relation to direction (e.g., north, south, east, west, upward, downward, forward, backward, right, left) and the system of coordinates, or to cameras or to areas or to shapes, of the area or the shape or parts of the area, or the shape; b. Features, phenomena, attributes, detection of heat, radiation, voice, smell and taste; c. Definitions and conclusions (e.g., temporary name, rectangular, stair, object, car, fluttering, flame, flowing of water, living thing, walking on four legs, growth, private type vehicle right side, change in time).

13. A computer vision method as in claim 12, wherein is included an additional stage of means for identification and calculation of the said data of areas, features, definitions and conclusions with regard to unidentified shapes and/or in which some motion occurs (e.g., motion, movement) in the entire area/shape and/or in the envelope and/or inside, and which are seen in the picture or part thereof, and all of them, a part thereof, a combination thereof are joined with the key-element data and is performed by means of: software programs, electronic circles and rules by which the system will operate, and will match size, distance, enlargement/reduction and photography angle of the viewed shape, in accordance with the size, the distance and the shooting angle of the stored shape, and a software program for the transfer, 3-D presentation and in form of multimedia.

14. A computer vision method as in claims 12 and 13, wherein the data, features, definitions and conclusions will be matched with a register/memory with all the data or a part thereof, with the said means and software programs for comparing and matching, and wherein identification by key-elements according to classification in one or more pre-determined hierarchic order(s) will be performed, either absolutely, or between and between, or possible, said key-elements will allow identification of areas, shapes or parts thereof, in relation to information known in advance or from previous photographs, with regard to said data of known areas, shapes or part thereof, as the case may be.

15. A computer vision method as in claims 9 to 14, wherein a computer, computer components, components, electronic circles, a processor, a computerized data processing system and alike, one or more of them, form one or more of the said means or any combination thereof; the said means known software programs, new and compatible; said means for converting data obtained from photographed pictures into digital data; said means for matching and definition of color identity with regards to the same viewed point and/or points generated in the photographed pictures; said means for calculation of distance and dimensions, for identification of features,^" definitions and conclusions; said means for matching of size and matching of photographing angle; said means for movement detection at time intervals, type of movement, motion; said means for handling of "true" table(s); said means for handling recurrent shapes; said means for data collection for key composition; said means for comparison and matching; said means for storing various types of information; said means for receiving, transmitting and drawing of data; said means including power supply, data protection, protection of said means, the method and alike.

16. A computer vision method as in any of claims 9 to 15, wherein the received and stored information that has been collected before, during and after identification, including calculations, known data, features, definitions that have been collected such as with regard to areas, shapes, is immediately or after a short while transmitted forward in form of information, in a regular manner and/or by stereo in form of multimedia and/or in 3D form, and which the system preserves and/or provides and/or which are accessible upon request and/or automatically to the user, such as a robot, a device, a blind person, by means of adequate interfacing means.

17. A computer vision system and method essentially including any of the innovations herein or any combinations thereof, as described, referred to, explained, illustrated, shown or implied, in detail and in the above claims, or in the drawings attached hereto.