WO2004068865A1 - Steroscopic panoramic image capture device - Google Patents

Steroscopic panoramic image capture device Download PDF

Info

Publication number
WO2004068865A1
WO2004068865A1 PCT/US2003/002285 US0302285W WO2004068865A1 WO 2004068865 A1 WO2004068865 A1 WO 2004068865A1 US 0302285 W US0302285 W US 0302285W WO 2004068865 A1 WO2004068865 A1 WO 2004068865A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
combined
panoramic
imaging system
images
Prior art date
Application number
PCT/US2003/002285
Other languages
French (fr)
Inventor
Don Pierce
Steven Herrnstadt
Trent Grover
Fred Good
Original Assignee
Micoy Corporation
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Micoy Corporation filed Critical Micoy Corporation
Priority to JP2004567640A priority Critical patent/JP2006515128A/en
Priority to PCT/US2003/002285 priority patent/WO2004068865A1/en
Priority to CNA038261855A priority patent/CN1771740A/en
Priority to AU2003214899A priority patent/AU2003214899A1/en
Priority to CA002513874A priority patent/CA2513874A1/en
Priority to EP03710742A priority patent/EP1586204A1/en
Publication of WO2004068865A1 publication Critical patent/WO2004068865A1/en
Priority to IL169827A priority patent/IL169827A0/en

Links

Classifications

    • GPHYSICS
    • G02OPTICS
    • G02BOPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B13/00Optical objectives specially designed for the purposes specified below
    • G02B13/06Panoramic objectives; So-called "sky lenses" including panoramic objectives having reflecting surfaces
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00Image analysis
    • G06T7/30Determination of transform parameters for the alignment of images, i.e. image registration
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20Image signal generators
    • H04N13/204Image signal generators using stereoscopic image cameras
    • H04N13/243Image signal generators using stereoscopic image cameras using three or more 2D image sensors
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60Control of cameras or camera modules
    • H04N23/698Control of cameras or camera modules for achieving an enlarged field of view, e.g. panoramic image capture
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/194Transmission of image signals
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20Image signal generators
    • H04N13/204Image signal generators using stereoscopic image cameras
    • H04N13/207Image signal generators using stereoscopic image cameras using a single 2D image sensor
    • H04N13/221Image signal generators using stereoscopic image cameras using a single 2D image sensor using the relative movement between cameras and objects
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20Image signal generators
    • H04N13/204Image signal generators using stereoscopic image cameras
    • H04N13/239Image signal generators using stereoscopic image cameras using two 2D image sensors having a relative position equal to or related to the interocular distance
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20Image signal generators
    • H04N13/286Image signal generators having separate monoscopic and stereoscopic modes
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/30Image reproducers
    • H04N13/332Displays for viewing with the aid of special glasses or head-mounted displays [HMD]
    • H04N13/344Displays for viewing with the aid of special glasses or head-mounted displays [HMD] with head-mounted left-right displays
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N2013/0074Stereoscopic image analysis
    • H04N2013/0081Depth or disparity estimation from stereoscopic image signals
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N2013/0074Stereoscopic image analysis
    • H04N2013/0088Synthesising a monoscopic image signal from stereoscopic images, e.g. synthesising a panoramic or high resolution monoscopic image

Definitions

  • Embodiments of the invention relate in general to a panoramic image capture device and, more specifically, to a panoramic image capture device for producing a stereoscopic panoramic image.
  • Panoramic cameras are known in the art. Such cameras often use a single, rotatable camera. Although such devices are suitable for stationary images, such devices typically produce blurred or distorted images when used to capture non-stationary objects. It is also known in the art to utilize an image capture system having a plurality of image capture devices. In this manner, a plurality of images can be captured, substantially simultaneously, and stitched together using processes known in the art. Although such systems substantially eliminate the problem associated with capturing objects in motion, such systems do not provide means for producing a stereoscopic image. It is also known in the art to use a "fish-eye" lens to capture a panoramic image. Such images, however, import a large amount of distortion into the resulting image, and capture images of relatively low quality.
  • an image capture system produces a stereoscopic pair of panoramic images.
  • this invention provides an image capture system for producing a seamless panoramic image.
  • this invention provides an image capture system for producing a stereoscopic panoramic image in motion.
  • this invention provides a stereoscopic panoramic image of minimal distortion.
  • this invention provides an imaging system for full- motion, real time, panoramic stereoscopic imaging.
  • an imaging system comprising a first image capture device, a second image capture device, and a third image capture device.
  • Means are also provided for combining at least a first portion of a first image captured using the first image capture device with a portion of a second image captured using the second image capture device to produce a first combined image.
  • Means are also provided for combining at least a second portion of the first image with at least a portion of a third image captured using the third image capture device, to produce a second combined image.
  • a plurality of image capture devices are utilized to produce a plurality of images, a portion of each of which is combined with a portion of adjacent images to produce a first combined panoramic image. Similarly, a second portion of each image is combined with separate portions of adjacent images to produce a second combined panoramic image.
  • the first combined panoramic image, and second combined panoramic image are displayed in a stereoscopic orientation to produce a stereoscopic panoramic image.
  • the imaging system of the present invention may be utilized to capture a plurality of images, to produce a full-motion stereoscopic panoramic image.
  • Fig. 1 illustrates a front elevation of the imaging system of the present invention
  • Fig. 2 illustrates a graphic depiction of image capture regions of adjacent image capture devices of the imaging system of the present invention
  • Fig. 3 illustrates a bottom perspective view of the final panoramic stereoscopic image displayed on a screen, and the polarized glasses used to view the image;
  • Figs. 4A-4B illustrate a left panoramic image and right panoramic image
  • Figs. 5A-5B illustrates images associated with first image buffer and second image buffer
  • Figs. 6A-6C are a flowchart of the transformation process utilized in association with the imaging system of the present device, to transform the plurality of images captured with the imaging devices into a first panoramic image and a second panoramic image to produce a stereoscopic panoramic image;
  • Fig. 7 illustrates a perspective view of a 360 degree unobstructed panoramic stereoscopic image, created using the camera of Fig. 1.
  • a camera (10) is shown having a body (12) constructed of plastic or other similar lightweight material.
  • the body (12) is substantially spherical, having a diameter preferably between about 0.001 and about 500 centimeters, and more preferably, between about 10 and about 50 centimeters.
  • a plurality of lenses (14) are provided substantially equally spaced across the surface of the body (12).
  • the lenses (14) are preferably circular lenses having a diameter of preferably between about 5 angstroms and about 10 centimeters, and more preferably, between about 0.5 and about 5 centimeters, hi the preferred embodiment, the lenses are model number BCL38C 3.8 millimeter micro lenses, manufactured by CBC America, located at 55 Mall Drive, Commack, NY 11725. As shown in Fig. 2, the lenses (14) are each associated with a charge coupled device (CCD) assembly (16), such as those well known in the art. Although in the preferred embodiment a GP-CX171/LM CCD color board camera, manufactured by CCD (CCD) assembly (16), such as those well known in the art. Although in the preferred embodiment a GP-CX171/LM CCD color board camera, manufactured by CBC America, located at 55 Mall Drive, Commack, NY 11725. Although in the preferred embodiment a GP-CX171/LM CCD color board camera, manufactured by
  • the lenses (14) and/or CCD assemblies (16) can also be described as "image capture units” (26), (40), and (54).
  • image capture units 26), (40), and (54).
  • all of the image capture units (26), (40), and (54) are operationally coupled to a processing unit (CPU)(22), which can thereby receive images from the image capture devices (14) and/or (16).
  • the CPU (22) is a 900 MHz, Pentium®4 class personal computer provided with an Oxygen GVX210 graphics card manufactured by 3Dlabs of 480 Pontrero, Sunnyvale, CA 94086.
  • the CPU may be of any type known in the art, it is preferably capable of quad buffering and utilizing page flipping software in a manner such as that known in the art.
  • the CPU (22) may be coupled to a head mounted display (24) which, in the preferred embodiment, is a VFX3D, manufactured by Interactive Imaging Systems, Inc., located at 2166 Brighton Henrietta Townline Road, Rochester, NY 14623. As shown in Fig. 2, the lenses (14) are divergent from one another, and offset about twenty degrees from one another along the substantially arcuate path or line defined by the body (12).
  • each lens (14) and/or image capture unit (26), (40), (54) allows each lens (14) and/or image capture unit (26), (40), (54) to have a substantially similar focal point.
  • Each lens (14) has a about a fifty-three degree field of view, which overlaps the field of view of a laterally adjoining lens by between about ten and about ninety percent, and, preferably, between about fifty and about sixty-five percent.
  • a first image capture unit (26) is associated with an optical axis (28), bisecting the image bordered on one side by a left side plane (30) and on the other side by a right side plane (32).
  • the lens (14) of the image capture unit (26) is focused on a defined image plane (34), divided into a left image plane (36) and a right image plane (38).
  • a second image capture unit (40) is also provided with an optical axis (42), a left side plane (44), a right side plane (46), a defined image plane (48), a left image plane (50), and a right image plane (52).
  • a third image capture unit (54), to the right of the first image capture unit (26), is provided with optical axis (56), a left side plane (58), a right side plane (60), a defined image plane (62), a left image plane (64), and a right image plane (66).
  • every point associated with a final panoramic image is within the defined image plane of at least two adjacent image capture units (26), (40) and/or (54).
  • the defined image planes of adjacent image capture units overlap vertically, preferably about 1-20 percent, more preferably about 5- 10 percent, and about 7 percent in the preferred embodiment.
  • the defined image planes of adjacent image capture units overlap horizontally, preferably about 20 percent, more preferably about 5-10 percent, and about 6 percent in the preferred embodiment.
  • a first panoramic image (72) and second panoramic image (74) are created.
  • Each of the panoramic images (72), (74) may display about 90 degrees of a scene (e.g., about half of a hemisphere).
  • an image (76), associated with the left image plane (36) of the first image capture unit (26) is combined with an image (78), associated with the left image plane (50) of the second image capture unit (40) and an image (80), associated with the left image plane (64) of the third image capture unit (54).
  • an image (76), associated with the left image plane (36) of the first image capture unit (26) is combined with an image (78), associated with the left image plane (50) of the second image capture unit (40) and an image (80), associated with the left image plane (64) of the third image capture unit (54).
  • the associated image planes (36), (50) and (64), preferably overlap by about 0.5-30 percent, more preferably by about 10-20 percent, and about 13 percent in the preferred embodiment, but are not parallel to one another, and are not necessarily tangent to a curve defining the first panoramic image (72).
  • the images (76), (78), (80), (82), (84) and (86), associated with planes (36), (50) and (64) must be transformed, to remove distortion associated with their non-parallel orientation, before they can be stitched together as described below to form the final panoramic image (68).
  • Figs. 2, 3, 4A-4B and 5A-5B are examples of the images (76), (78), (80), (82), (84) and (86), associated with planes (36), (50) and (64)
  • the images may be transmitted via hardwired, wireless, or any desired connection to the CPU (22).
  • the CPU (22) then may operate to transform the images in accordance with the process described in Figs. 6A-6C.
  • source images which in the preferred embodiment are substantially rectilinear images, but which may, of course, be any type of image, are obtained or received from the image capture units (26), (40) and (54) by the CPU (22).
  • Figs. 2, 4A, 4B and 6A-6C As shown in block (94), the CPU (22) may then define registration pixel pairs for the untransformed source images.
  • the CPU(22) creates an input file.
  • the input file includes the height and width of the final panoramic image (68), the source information, and registration point information.
  • the source information includes the file name and path of the source image, the height and width of the source image in pixels, and the yaw, pitch and roll angles of the source of the associated image capture unit.
  • the horizontal field of view of the image capture unit which is preferably between about 1 and about 80 degrees, more preferably between about 30 and about 60 degrees, and about 53 degrees in the preferred embodiment, is defined by the associated left side plane and right side plane, the X Shift, Y Shift and zoom values of the source image.
  • the information associated with the registration points includes information regarding the source image associated with the first pixel position of the registration point, a horizontal and vertical pixel position in the first source image, the list index of info ⁇ nation regarding the source image associated with the second pixel position, and a horizontal and vertical pixel position in the second source image.
  • the images (76-86) associated with the image planes (36, 38, 50, 52, 64 and 66) are substantially rectilinear, normal flat-field images, and the panoramic images (72 and 74) are at least partially equirectangular, representing pixel mapping on a spherical surface in a manner such as that shown in Figs. 4A-4B. Accordingly, once the CPU (22) creates the input file, as shown in block (98), the registration pixel pairs can be transformed to locate their position in the final panoramic image (68).
  • a vector is defined that represents a first pixel of a given registration pixel pair in three- dimensional space, locating it on the final panoramic image (68). This is accomplished by applying the following matrix transformation to each pixel:
  • segment x Alter the horizontal position of the pixel in the source image so that it is relative to the image's center. Then compensate for the
  • segment y Alter the vertical position of the pixel in the source image so that it is relative to the image's center. Then compensate for the Y Shift and zoom variables of the source image.
  • segment z Using various source image variables, determine the z segment that corresponds to the scale provided by the image's size in pixels.
  • the pixel vector represents the global globX, globY, and globZ positions of that point in three-dimensional space.
  • the CPU (22) then converts these positions into spherical coordinates and applies them directly to the final panoramic coordinates.
  • the vector's yaw angle represents its horizontal panorama position, newX, and its pitch angle represents its vertical panorama position, newY.
  • the CPU (22) calculates the distance between the registration pixel pairs. If the average distance of the registration pixel pairs for a given source image are not yet minimized, as would be the case upon the initial transformation, shown in block (102), the yaw of the source image is altered slightly, whereafter the process returns to block (98) and the registration pixel pairs are again transformed to pixel points in the final panoramic image (68). This process continues, altering the yaw, until the average distance of the source image registration pixel pairs is minimized.
  • the average distance of all source image registration pixel pairs is calculated and, if they are not yet minimized, the yaw, pitch, roll, XShift, YShift and zoom of the source images are altered as shown in block (108), and the process returns to block (98), where the process continues, until the distance between the registration pixel pairs is minimized across all of the source images.
  • an output file is created, identifying the height and width of the first panoramic image (72), the yaw, pitch, roll, XShift, YShift and zoom transformation image information relative to each particular source image.
  • a vector is defined representing the particular pixel's position in three dimensional space, using the vector transformation described above.
  • the vector is transformed to reflect the yaw, pitch, roll, XShift, YShift and zoom information associated with the source image as defined in the output file.
  • the transformed vector is associated with a pixel in the final panoramic image (68). As shown in block (118), this process is repeated until all of the pixels in a particular source image have been transformed into vectors, and their position located on the final panoramic image (68)
  • two image buffers (90) and (92) are created, each having a height and width approximately equal to that of the final panoramic image (68).
  • FIGs. 3, 5A-5B and 6B As shown in block (122), once the image buffers (90) and (92) have been created, vector transformation information associated with a quadrilateral of four adjacent pixels of a particular source image is utilized to draw the quadrilateral of pixels onto the appropriate image buffer (90) or (92). (Figs. 5A-5B and 6C). If the pixel is in the left image planes (38), (52) or (66), the pixel is written to the left image buffer (90).
  • the pixel is in the right image buffer (92).
  • FIGs. 2 and 5A-5B since the transformation is likely to spread out the quadrilateral of pixels on the image buffer, there will likely be gaps between pixels as they are converted from their rectilinear location to their equirectangular location in the associated image buffer (90) or (92).
  • FIGs. 5A- 5B and 6C When the quadrilateral of pixels is located on the associated image buffer (90) or (92), the gaps thereby created are filled as smoothly as possible, perhaps using a linear gradient of the corner pixel colors, in a manner such as that known in the art.
  • Known linear gradient fill techniques may also be used to diminish visible seams between images.
  • additional source image information is applied to the image buffers (90) and (92), for areas of the image buffers (90) and (92) that have already been filled with pixel data, the alpha transparency of the new overlapping pixels is linearly degraded, smoothing the resulting seam as described below.
  • the CPU (22) may eliminate the gaps by interpolating the internal pixel values, using any method known in the art, which may include comparing the gap to the adjacent pixels, or "white boxing" the gap by utilizing the immediately preceding frame and immediately succeeding frame in a motion capture system, to extrapolate the most appropriate pixel value for the gap.
  • blocks (122) and (124) are repeated until all of the source image pixels have been mapped to the appropriate image buffer (90) or (92). Once all of the pixels have been mapped, as shown in block (128), the first image buffer pixels are compared to the first panoramic image pixels.
  • the CPU (22) sets the pixel in the first image buffer (90) to maximum visibility. If the first panoramic image (72) already has a pixel associated with a pixel in the first image buffer (90), the existing pixel is compared to the corresponding pixel in the first image buffer (90). If the pixel in the first image buffer (90) has a shorter distance to the center of its respective source image than does the existing pixel, the pixel in the first image buffer (90) is set to maximum visibility.
  • the pixel in the first image buffer (90) is set to minimum visibility.
  • the process is repeated to merge pixels of the second image buffer (92) into the second panoramic image (74).
  • the overlapping edges of the image in the image buffers (90) and (92) are feathered by degrading visibility of the pixels over the area of overlap. This feathering smoothes the overlapping areas of the image once it is merged from the image buffers (90) and (92) into the panoramic images (72) and (74).
  • blocks (122) through (132) are repeated until all of the source images have been merged into the first and second panoramic images (72) and (74).
  • the images (76), (78) and (80) associated with the left image planes (36), (50) and (64) of the image capture units (26), (40) and (54) are used to create the first panoramic image (72), and the images (82), (84) and (86) associated with the right image planes (38), (52) and (66) of the image capture units (26), (40) and (54) are used to create the second panoramic image (74).
  • the panoramic images (72) and (74) can be displayed together as the final panoramic image (68). (Figs. 3, 4A and 4B).
  • the panoramic images (72) and (74) may be reverse polarized, shown on a standard panoramic screen (140), such as that shown in Fig. 3, and viewed using glasses (142) having lenses of reversed polarity.
  • the panoramic images (72) and (74) may be conveyed to the head mounted display (24) shown in Fig. 1.
  • the CPU (22) can send multiple panoramic images to the head mounted display (24) using known "page-flipping" techniques to send and separately display the images in the left and right displays of the head mounted display (24), as appropriate. This process can be used to animate the display as a full 24 frame per second motion picture, or to display multiple visual images to each display.
  • an imaging system see Fig.
  • first image capture unit (26) to capture a first image (36)
  • second image capture unit (40) to capture a second image (50)
  • third image capture unit (54) to capture a third image (64)
  • the first portion of the first image can be any amount up to the entire first image, or may be between about 20 percent and about 80 percent of the first image.
  • the portion of the second image can be any amount up to the entire second image, or may be between about 20 percent and about 80 percent of the second image.
  • the first and second combined images can be repeatedly produced and displayed to convey stereoscopic motion.
  • An imaging system may also comprise an image capture unit (26) for providing an image, a means (22) for using a first portion of said image to provide a first stereoscopic image, and means (22) for using a second portion of said image to provide a second stereoscopic image.
  • the head mounted display unit (24) may also be provided with an orientation sensor (144), such as those well known in the art, to change the image provided to the head mounted display (24) as the sensor (144) moves.
  • an orientation sensor 144
  • a user may look up, down, and in either direction, to see that portion of the final panoramic image (68) associated with the vector of the user's line of sight, and have the sensation of actually looking around a three- dimensional space.
  • the camera (10) may be provided with a plurality of image capture pairs having substantially rectilinear capture systems oriented substantially parallel to one another.
  • the pairs may be offset by a predetermined factor to obtain a desired stereoscopic image.
  • the transformation process associated with this embodiment is identical to that described above, albeit instead of dividing the images in half, and sending pixels from each half to a separate image buffer, all of the pixels associated with the images from the "left" image capture devices of each image capture device pair are sent to one image buffer, and all of the pixels associated with images from the "right” image capture devices are sent to another image buffer.
  • the camera (10) and CPU (22) may be utilized to capture twenty-four or more frames per second, and display the final stereoscopic, panoramic image (68), in real time, as a motion picture.
  • computer generated graphical information produced in a manner such as that well known in the art, may be combined with the final panoramic images (72) and (74) in the CPU (22) to provide a seamless integration of actual images captured with the camera (10), and digitized virtual reality images (146). (Fig. 3) This combination produces a seamless display of real and virtual panoramic stereographic virtual images.
  • the images captured by the camera (10) may be transformed, utilizing the above transformation procedures, to produce a seamless 360-degree panoramic monographic image.
  • the camera (10) is provided with a support post (148) and a transportation unit such as a remote control carriage (150), similar to or identical to those used in association with remote control cars and the like. Images associated with the left image planes of the image capture units may be used to produce a combined image and the images associated with the right image planes of the image capture units may be used to overwrite and fill the combined image to hide the support post (148), carriage (150) and any other camera equipment otherwise visible in the combined image.
  • the foregoing transformation procedures may be utilized for such overwriting and filling.
  • a final panoramic image (152), which may be displayed on a spherical screen (154) or on the head mounted display (24). (Figs. 1 and 7).
  • the interpolation and feathering procedures detailed above may be utilized to approximate the image lying beneath the carriage (150), to produce the appearance of a completely unobstructed 360-degree image.

Abstract

An imaging system comprising a plurality of first image capture devices. Overlapping rectilinear images are captured and halved, with left halves being stiched and transformed into a first equirectangular image and the right halves being stiched and transformed into a second equirectangular image. The first equirectangular image, and second equirectangular image are displayed in a stereoscopic orientation to produce a stereoscopic equirectangular image. The imaging system may be utilized to capture a plurality of sequential images, to produce a full-motion stereoscopic equirectangular image.

Description

STEREOSCOPIC PANORAMIC IMAGE CAPTURE DEVICE
Technical Field
Embodiments of the invention relate in general to a panoramic image capture device and, more specifically, to a panoramic image capture device for producing a stereoscopic panoramic image.
Background
Panoramic cameras are known in the art. Such cameras often use a single, rotatable camera. Although such devices are suitable for stationary images, such devices typically produce blurred or distorted images when used to capture non-stationary objects. It is also known in the art to utilize an image capture system having a plurality of image capture devices. In this manner, a plurality of images can be captured, substantially simultaneously, and stitched together using processes known in the art. Although such systems substantially eliminate the problem associated with capturing objects in motion, such systems do not provide means for producing a stereoscopic image. It is also known in the art to use a "fish-eye" lens to capture a panoramic image. Such images, however, import a large amount of distortion into the resulting image, and capture images of relatively low quality. Accordingly, it would be desirable to produce a panoramic image of lower distortion and higher quality. Typically, to capture a stereoscopic image, two imaging systems are positioned next to one another to capture a particular image. Unfortunately, this method cannot be extrapolated to producing a stereoscopic panoramic image, as one image capture device would necessarily fall into the field of view of the adjacent image capture device. It would be desirable, therefore, to provide a panoramic image capture system which could be utilized to produce a stereoscopic pair of panoramic images for a stereoscopic display of a particular image.
Summary of the Invention In an advantage provided by this invention, an image capture system produces a stereoscopic pair of panoramic images.
Advantageously, this invention provides an image capture system for producing a seamless panoramic image.
Advantageously, this invention provides an image capture system for producing a stereoscopic panoramic image in motion.
Advantageously, this invention provides a stereoscopic panoramic image of minimal distortion.
Advantageously, this invention provides an imaging system for full- motion, real time, panoramic stereoscopic imaging. Advantageously, in a preferred example of this invention, an imaging system is provided comprising a first image capture device, a second image capture device, and a third image capture device. Means are also provided for combining at least a first portion of a first image captured using the first image capture device with a portion of a second image captured using the second image capture device to produce a first combined image. Means are also provided for combining at least a second portion of the first image with at least a portion of a third image captured using the third image capture device, to produce a second combined image.
In the preferred embodiment, a plurality of image capture devices are utilized to produce a plurality of images, a portion of each of which is combined with a portion of adjacent images to produce a first combined panoramic image. Similarly, a second portion of each image is combined with separate portions of adjacent images to produce a second combined panoramic image. Preferably, the first combined panoramic image, and second combined panoramic image are displayed in a stereoscopic orientation to produce a stereoscopic panoramic image. The imaging system of the present invention may be utilized to capture a plurality of images, to produce a full-motion stereoscopic panoramic image.
Brief Description of the Drawings The present invention will now be described, by way of example, with reference to the accompanying drawings in which:
Fig. 1 illustrates a front elevation of the imaging system of the present invention;
Fig. 2 illustrates a graphic depiction of image capture regions of adjacent image capture devices of the imaging system of the present invention;
Fig. 3 illustrates a bottom perspective view of the final panoramic stereoscopic image displayed on a screen, and the polarized glasses used to view the image;
Figs. 4A-4B illustrate a left panoramic image and right panoramic image; and
Figs. 5A-5B illustrates images associated with first image buffer and second image buffer;
Figs. 6A-6C are a flowchart of the transformation process utilized in association with the imaging system of the present device, to transform the plurality of images captured with the imaging devices into a first panoramic image and a second panoramic image to produce a stereoscopic panoramic image; and
Fig. 7 illustrates a perspective view of a 360 degree unobstructed panoramic stereoscopic image, created using the camera of Fig. 1.
Detailed. Description of the Preferred Embodiment Referring to Fig. 1, a camera (10) is shown having a body (12) constructed of plastic or other similar lightweight material. In the preferred embodiment, the body (12) is substantially spherical, having a diameter preferably between about 0.001 and about 500 centimeters, and more preferably, between about 10 and about 50 centimeters. Provided substantially equally spaced across the surface of the body (12) are a plurality of lenses (14). The lenses (14) are preferably circular lenses having a diameter of preferably between about 5 angstroms and about 10 centimeters, and more preferably, between about 0.5 and about 5 centimeters, hi the preferred embodiment, the lenses are model number BCL38C 3.8 millimeter micro lenses, manufactured by CBC America, located at 55 Mall Drive, Commack, NY 11725. As shown in Fig. 2, the lenses (14) are each associated with a charge coupled device (CCD) assembly (16), such as those well known in the art. Although in the preferred embodiment a GP-CX171/LM CCD color board camera, manufactured by
Panasonic and available from Rock2000.com is used, any known image capture system may be used. Thus, for the purposes of this disclosure, the lenses (14) and/or CCD assemblies (16) can also be described as "image capture units" (26), (40), and (54). As shown in Fig. 2, all of the image capture units (26), (40), and (54) are operationally coupled to a processing unit (CPU)(22), which can thereby receive images from the image capture devices (14) and/or (16). In the preferred embodiment, the CPU (22) is a 900 MHz, Pentium®4 class personal computer provided with an Oxygen GVX210 graphics card manufactured by 3Dlabs of 480 Pontrero, Sunnyvale, CA 94086. Although the CPU may be of any type known in the art, it is preferably capable of quad buffering and utilizing page flipping software in a manner such as that known in the art. The CPU (22) may be coupled to a head mounted display (24) which, in the preferred embodiment, is a VFX3D, manufactured by Interactive Imaging Systems, Inc., located at 2166 Brighton Henrietta Townline Road, Rochester, NY 14623. As shown in Fig. 2, the lenses (14) are divergent from one another, and offset about twenty degrees from one another along the substantially arcuate path or line defined by the body (12). This offset, which may be from about 5 to about 45 degrees along the direction of the substantially arcuate path, allows each lens (14) and/or image capture unit (26), (40), (54) to have a substantially similar focal point. Each lens (14) has a about a fifty-three degree field of view, which overlaps the field of view of a laterally adjoining lens by between about ten and about ninety percent, and, preferably, between about fifty and about sixty-five percent. As shown in Fig. 2, a first image capture unit (26) is associated with an optical axis (28), bisecting the image bordered on one side by a left side plane (30) and on the other side by a right side plane (32). The lens (14) of the image capture unit (26) is focused on a defined image plane (34), divided into a left image plane (36) and a right image plane (38).
Similarly, a second image capture unit (40) is also provided with an optical axis (42), a left side plane (44), a right side plane (46), a defined image plane (48), a left image plane (50), and a right image plane (52). A third image capture unit (54), to the right of the first image capture unit (26), is provided with optical axis (56), a left side plane (58), a right side plane (60), a defined image plane (62), a left image plane (64), and a right image plane (66).
By providing a plurality of image capture units (26), (40) and (54), dividing the defined image planes (34), (48) and (62) into portions, such as halves, and orienting the image capture units (26), (40) and (54) as shown in Fig. 2, every point associated with a final panoramic image is within the defined image plane of at least two adjacent image capture units (26), (40) and/or (54). As shown in Fig. 5A-5B, the defined image planes of adjacent image capture units overlap vertically, preferably about 1-20 percent, more preferably about 5- 10 percent, and about 7 percent in the preferred embodiment. Similarly, the defined image planes of adjacent image capture units overlap horizontally, preferably about 20 percent, more preferably about 5-10 percent, and about 6 percent in the preferred embodiment. These overlaps aid in the "stitching" process described below.
To produce the final panoramic image (68) of the present invention, which may display about 180 degrees of a scene (e.g., a hemisphere), a first panoramic image (72) and second panoramic image (74) are created. (Figs. 3 and 4A-4B). Each of the panoramic images (72), (74) may display about 90 degrees of a scene (e.g., about half of a hemisphere). To create the first panoramic image (72), an image (76), associated with the left image plane (36) of the first image capture unit (26) is combined with an image (78), associated with the left image plane (50) of the second image capture unit (40) and an image (80), associated with the left image plane (64) of the third image capture unit (54). (Figs. 2, 4A and 5A). As shown in Fig. 2, the associated image planes (36), (50) and (64), preferably overlap by about 0.5-30 percent, more preferably by about 10-20 percent, and about 13 percent in the preferred embodiment, but are not parallel to one another, and are not necessarily tangent to a curve defining the first panoramic image (72). (Fig. 2 and 4A). Accordingly, the images (76), (78), (80), (82), (84) and (86), associated with planes (36), (50) and (64) must be transformed, to remove distortion associated with their non-parallel orientation, before they can be stitched together as described below to form the final panoramic image (68). (Figs. 2, 3, 4A-4B and 5A-5B).
Once the images (76), (78) and (80), associated with the left image planes (38), (52), (66), and the images (82), (84) and (86), associated with the right image planes (36), (50) and (64) have been collected and received from all of the image capture units (26), (40) and (54), the images may be transmitted via hardwired, wireless, or any desired connection to the CPU (22). The CPU (22) then may operate to transform the images in accordance with the process described in Figs. 6A-6C. As shown in block (88), source images, which in the preferred embodiment are substantially rectilinear images, but which may, of course, be any type of image, are obtained or received from the image capture units (26), (40) and (54) by the CPU (22). (Figs. 2, 4A, 4B and 6A-6C). As shown in block (94), the CPU (22) may then define registration pixel pairs for the untransformed source images.
Thereafter, as shown in block (96), the CPU(22) creates an input file. The input file includes the height and width of the final panoramic image (68), the source information, and registration point information. The source information includes the file name and path of the source image, the height and width of the source image in pixels, and the yaw, pitch and roll angles of the source of the associated image capture unit. The horizontal field of view of the image capture unit, which is preferably between about 1 and about 80 degrees, more preferably between about 30 and about 60 degrees, and about 53 degrees in the preferred embodiment, is defined by the associated left side plane and right side plane, the X Shift, Y Shift and zoom values of the source image. The information associated with the registration points includes information regarding the source image associated with the first pixel position of the registration point, a horizontal and vertical pixel position in the first source image, the list index of infoπnation regarding the source image associated with the second pixel position, and a horizontal and vertical pixel position in the second source image.
The images (76-86) associated with the image planes (36, 38, 50, 52, 64 and 66) are substantially rectilinear, normal flat-field images, and the panoramic images (72 and 74) are at least partially equirectangular, representing pixel mapping on a spherical surface in a manner such as that shown in Figs. 4A-4B. Accordingly, once the CPU (22) creates the input file, as shown in block (98), the registration pixel pairs can be transformed to locate their position in the final panoramic image (68).
Starting with an arbitrary source image, a vector is defined that represents a first pixel of a given registration pixel pair in three- dimensional space, locating it on the final panoramic image (68). This is accomplished by applying the following matrix transformation to each pixel:
Define segment x: Alter the horizontal position of the pixel in the source image so that it is relative to the image's center. Then compensate for the
X Shift and zoom variables of the source image.
Define segment y: Alter the vertical position of the pixel in the source image so that it is relative to the image's center. Then compensate for the Y Shift and zoom variables of the source image. Define segment z: Using various source image variables, determine the z segment that corresponds to the scale provided by the image's size in pixels.
Transform the vector so that it corresponds to the rotation angles of the source camera. This is calculated by transforming the vector by each of three rotational matrices:
Rotation about x-axis:
Figure imgf000010_0001
Rotation about y-axis:
Xos(yaw)
Figure imgf000010_0003
Figure imgf000010_0002
Rotation about z-axis: f cos(roll) s iroll) 0 0s - sin(rotV) cos(røtV) 0 0
0 0 1 0 0 0 0 1
Upon matrix transformation, the pixel vector represents the global globX, globY, and globZ positions of that point in three-dimensional space. The CPU (22) then converts these positions into spherical coordinates and applies them directly to the final panoramic coordinates. The vector's yaw angle represents its horizontal panorama position, newX, and its pitch angle represents its vertical panorama position, newY.
As shown in block (100), once the registration pixel pairs have been mapped to the final panoramic image (68) in this manner, the CPU (22) calculates the distance between the registration pixel pairs. If the average distance of the registration pixel pairs for a given source image are not yet minimized, as would be the case upon the initial transformation, shown in block (102), the yaw of the source image is altered slightly, whereafter the process returns to block (98) and the registration pixel pairs are again transformed to pixel points in the final panoramic image (68). This process continues, altering the yaw, until the average distance of the source image registration pixel pairs is minimized. Thereafter, the pitch, roll, X Shift, Y Shift and zoom are altered, until the average distance of the associated registration pixel pairs is minimized. Once the yaw, pitch, roll, X Shift, Y Shift and zoom of a particular source image is optimized, as shown in block (104), the transformation procedure is repeated for all source images until they are all thus optimized.
As shown in block (106), once all of the source images have been thus optimized, the average distance of all source image registration pixel pairs is calculated and, if they are not yet minimized, the yaw, pitch, roll, XShift, YShift and zoom of the source images are altered as shown in block (108), and the process returns to block (98), where the process continues, until the distance between the registration pixel pairs is minimized across all of the source images.
Once the average distance between the registration pixel pairs has been minimized across all source images, as shown in block (110), an output file is created, identifying the height and width of the first panoramic image (72), the yaw, pitch, roll, XShift, YShift and zoom transformation image information relative to each particular source image. Thereafter, as shown in block (112), for each pixel within a given source image, a vector is defined representing the particular pixel's position in three dimensional space, using the vector transformation described above. Once a vector has been defined, as shown in block (114), the vector is transformed to reflect the yaw, pitch, roll, XShift, YShift and zoom information associated with the source image as defined in the output file. After completion of the transformation of the vector, as shown in block (116), the transformed vector is associated with a pixel in the final panoramic image (68). As shown in block (118), this process is repeated until all of the pixels in a particular source image have been transformed into vectors, and their position located on the final panoramic image (68)
As shown in block (120), once all of the pixels of a given source image have been located on the final panoramic image (68), two image buffers (90) and (92) are created, each having a height and width approximately equal to that of the final panoramic image (68). (Figs. 3, 5A-5B and 6B). As shown in block (122), once the image buffers (90) and (92) have been created, vector transformation information associated with a quadrilateral of four adjacent pixels of a particular source image is utilized to draw the quadrilateral of pixels onto the appropriate image buffer (90) or (92). (Figs. 5A-5B and 6C). If the pixel is in the left image planes (38), (52) or (66), the pixel is written to the left image buffer (90). If the pixel is in the right image planes (36), (50) or (64), the pixel is in the right image buffer (92). (Figs. 2 and 5A-5B). As shown in block (124), since the transformation is likely to spread out the quadrilateral of pixels on the image buffer, there will likely be gaps between pixels as they are converted from their rectilinear location to their equirectangular location in the associated image buffer (90) or (92). (Figs. 5A- 5B and 6C). When the quadrilateral of pixels is located on the associated image buffer (90) or (92), the gaps thereby created are filled as smoothly as possible, perhaps using a linear gradient of the corner pixel colors, in a manner such as that known in the art. Known linear gradient fill techniques may also be used to diminish visible seams between images. As additional source image information is applied to the image buffers (90) and (92), for areas of the image buffers (90) and (92) that have already been filled with pixel data, the alpha transparency of the new overlapping pixels is linearly degraded, smoothing the resulting seam as described below.
Accordingly, when mapping the quadrilateral to the associated image buffer (90) or (92), the CPU (22) may eliminate the gaps by interpolating the internal pixel values, using any method known in the art, which may include comparing the gap to the adjacent pixels, or "white boxing" the gap by utilizing the immediately preceding frame and immediately succeeding frame in a motion capture system, to extrapolate the most appropriate pixel value for the gap. As shown in block (126), blocks (122) and (124) are repeated until all of the source image pixels have been mapped to the appropriate image buffer (90) or (92). Once all of the pixels have been mapped, as shown in block (128), the first image buffer pixels are compared to the first panoramic image pixels. Ifthe first panoramic image (72) does not have a pixel associated with a pixel in the first image buffer (90), the CPU (22) sets the pixel in the first image buffer (90) to maximum visibility. Ifthe first panoramic image (72) already has a pixel associated with a pixel in the first image buffer (90), the existing pixel is compared to the corresponding pixel in the first image buffer (90). Ifthe pixel in the first image buffer (90) has a shorter distance to the center of its respective source image than does the existing pixel, the pixel in the first image buffer (90) is set to maximum visibility. Conversely, ifthe pixel in the first image buffer (90) has a greater distance to the center of its respective source image than that of the existing pixel, then the pixel in the first image buffer (90) is set to minimum visibility. The process is repeated to merge pixels of the second image buffer (92) into the second panoramic image (74). As shown in block (130), the overlapping edges of the image in the image buffers (90) and (92) are feathered by degrading visibility of the pixels over the area of overlap. This feathering smoothes the overlapping areas of the image once it is merged from the image buffers (90) and (92) into the panoramic images (72) and (74). As shown in block (132), once the image buffer pixels have been set to the appropriate visibility, and the image of the image buffers feathered, the images in the image buffers are merged into the first and second panoramic images (72) and (74). As shown in block (134), blocks (122) through (132) are repeated until all of the source images have been merged into the first and second panoramic images (72) and (74). As noted above, the images (76), (78) and (80) associated with the left image planes (36), (50) and (64) of the image capture units (26), (40) and (54) are used to create the first panoramic image (72), and the images (82), (84) and (86) associated with the right image planes (38), (52) and (66) of the image capture units (26), (40) and (54) are used to create the second panoramic image (74). Once both of the final panoramic images (72) and (74) have been created, as shown in block (138), the panoramic images (72) and (74) can be displayed together as the final panoramic image (68). (Figs. 3, 4A and 4B). The panoramic images (72) and (74) may be reverse polarized, shown on a standard panoramic screen (140), such as that shown in Fig. 3, and viewed using glasses (142) having lenses of reversed polarity. Alternatively, the panoramic images (72) and (74) may be conveyed to the head mounted display (24) shown in Fig. 1. The CPU (22) can send multiple panoramic images to the head mounted display (24) using known "page-flipping" techniques to send and separately display the images in the left and right displays of the head mounted display (24), as appropriate. This process can be used to animate the display as a full 24 frame per second motion picture, or to display multiple visual images to each display. Thus, an imaging system (see Fig. 2) may comprise a first image capture unit (26) to capture a first image (36), a second image capture unit (40) to capture a second image (50), and a third image capture unit (54) to capture a third image (64), as well as a means (22) for combining a first portion of the first image (36) with a portion of the second image (50) to produce a first combined image, and means (22) for combining a second portion of the first image (36) with a portion of the third image (64) to produce a second combined image. The first portion of the first image can be any amount up to the entire first image, or may be between about 20 percent and about 80 percent of the first image. The portion of the second image can be any amount up to the entire second image, or may be between about 20 percent and about 80 percent of the second image. The first and second combined images can be repeatedly produced and displayed to convey stereoscopic motion. An imaging system may also comprise an image capture unit (26) for providing an image, a means (22) for using a first portion of said image to provide a first stereoscopic image, and means (22) for using a second portion of said image to provide a second stereoscopic image.
The head mounted display unit (24) may also be provided with an orientation sensor (144), such as those well known in the art, to change the image provided to the head mounted display (24) as the sensor (144) moves. In this manner, a user (not shown) may look up, down, and in either direction, to see that portion of the final panoramic image (68) associated with the vector of the user's line of sight, and have the sensation of actually looking around a three- dimensional space.
In an alternative embodiment of the present invention, the camera (10) may be provided with a plurality of image capture pairs having substantially rectilinear capture systems oriented substantially parallel to one another. The pairs may be offset by a predetermined factor to obtain a desired stereoscopic image. Because the images are captured in pairs, the transformation process associated with this embodiment is identical to that described above, albeit instead of dividing the images in half, and sending pixels from each half to a separate image buffer, all of the pixels associated with the images from the "left" image capture devices of each image capture device pair are sent to one image buffer, and all of the pixels associated with images from the "right" image capture devices are sent to another image buffer.
In another alternative embodiment of the present invention, the camera (10) and CPU (22) may be utilized to capture twenty-four or more frames per second, and display the final stereoscopic, panoramic image (68), in real time, as a motion picture. hi yet another alternative embodiment of the present invention, computer generated graphical information, produced in a manner such as that well known in the art, may be combined with the final panoramic images (72) and (74) in the CPU (22) to provide a seamless integration of actual images captured with the camera (10), and digitized virtual reality images (146). (Fig. 3) This combination produces a seamless display of real and virtual panoramic stereographic virtual images.
In yet another alternative embodiment of the present invention, the images captured by the camera (10) may be transformed, utilizing the above transformation procedures, to produce a seamless 360-degree panoramic monographic image. As shown in Fig. 1, the camera (10) is provided with a support post (148) and a transportation unit such as a remote control carriage (150), similar to or identical to those used in association with remote control cars and the like. Images associated with the left image planes of the image capture units may be used to produce a combined image and the images associated with the right image planes of the image capture units may be used to overwrite and fill the combined image to hide the support post (148), carriage (150) and any other camera equipment otherwise visible in the combined image. The foregoing transformation procedures may be utilized for such overwriting and filling. In this manner, only those images which do not include the undesired information are mapped to a final panoramic image (152), which may be displayed on a spherical screen (154) or on the head mounted display (24). (Figs. 1 and 7). For that portion of the image reflecting the area covered by the footprint of the carriage (150), the interpolation and feathering procedures detailed above may be utilized to approximate the image lying beneath the carriage (150), to produce the appearance of a completely unobstructed 360-degree image.
Although the invention has been described with respect to the preferred embodiment thereof, it is to be also understood that it is not to be so limited, since changes and modifications can be made therein which are within the full intended scope of this invention as defined by the appended claims.

Claims

What is claimed is:
1. An imaging system, comprising: a first image capture unit to capture a first image; a second image capture unit to capture a second image; a third image capture unit to capture a third image; means for combining a first portion of the first image with a portion of the second image to produce a first combined image; and means for combining a second portion of the first image with a portion of the third image to produce a second combined image.
2. The imaging system of Claim 1, wherein said first image capture unit, said second image capture unit, and said third image capture unit are located along an arc relative to one another, about five to about forty-five degrees apart.
3. The imaging system of Claim 1 , wherein an image plane associated with said first image overlaps an image plane associated with said second image by about 0.5 to about 30 percent.
4. The imaging system of Claim 1, wherein an image plane associated with said first image vertically overlaps an image plane associated with said second image by about 1 to about 20 percent.
5. The imaging system of Claim 1, wherein said first image and said second image are substantially rectilinear.
The imaging system of Claim 1, wherein said first combined image and said second combined image are at least partially equirectangular.
The imaging system of Claim 1, wherein the first and the second combined images can be displayed to provide a stereoscopic image.
8. The imaging system of Claim 1, further comprising means for displaying said first combined image and said second combined image as a stereoscopic image.
9. The imaging system of Claim 1, further comprising means for sequentially displaying a plurality of the first and the second combined images as a moving stereoscopic image.
10. The imaging system of Claim 1, further comprising means for combining said first combined image with a sufficient plurality of images to produce a first combined panoramic image representing at least about 90 degrees of a scene, and combining said second combined image with a sufficient plurality of other images to produce a second combined panoramic image, representing about 90 degrees of the scene, and means for displaying said first combined panoramic image and said second combined panoramic image to provide a stereoscopic, panoramic image.
11. The imaging system of Claim 10, further comprising a means to display a first set of combined panoramic images and a second set of combined panoramic images in succession to provide a moving stereoscopic, panoramic image.
12. The imaging system of Claim 1, wherein said first combined image and said second combined image are combined with a digital image to produce a stereoscopic image including said digital image.
13. An imaging system, comprising: an image capture unit for providing an image; means for using a first portion of said image to provide a first stereoscopic image; and means for using a second portion of said image to provide a second stereoscopic image.
14. The imaging system of Claim 13, further comprising means for combining said first stereoscopic image and said second stereoscopic image into a panoramic stereoscopic image.
15. The imaging system of Claim 13 , further comprising: a plurality of image capture units to provide a plurality of images; means for combining selected ones of the plurality of images with the image to produce combined images; means for combining said combined images into a first panoramic image and a second panoramic image; and means for displaying said first panoramic image and said second panoramic image to provide a panoramic, stereoscopic image.
16. The imaging system of Claim 15, wherein said first panoramic image displays about 90 degrees of a scene.
17. The imaging system of Claim 15, wherein said panoramic, stereoscopic image displays about 180 degrees of a scene.
18. A method, comprising: obtaining a first image; obtaining a second image; obtaining a third image; combining a first portion of said first image with a portion of said second image to produce a first combined image; combining a second portion of said first image with a portion of said third image to produce a second combined image; and displaying said first combined image and said second combined image as a stereoscopic image.
19. The method of Claim 18, further comprising obtaining said first image, said second image, and said third image from a plurality of image capture units located along an arc.
20. The method of Claim 19, wherein said plurality of image capture units are located about 5 degrees to about 45 degrees apart in a single direction along the arc.
21. The method of Claim 18, further comprising displaying a plurality of first combined images in sequence and a plurality of second combined images in sequence to provide a moving stereoscopic image.
22. The method of Claim 21 , wherein said moving stereoscopic image represents about 180 degrees of a scene.
23. The method of claim 18, further comprising: defining a plurality of registration pixel pairs.
24. The method of claim 23, further comprising: transforming the plurality of registration pixel pairs to minimize a distance of image registration.
25. The method of claim 23 , further comprising: defining a vector to represent a pixel of one of the plurality of registration pixel pairs.
26. The method of claim 25, further comprising: transforming the vector into a transformed vector to correspond to a source rotation angle.
27. The method of claim 26, further comprising: associating the transformed vector with a pixel in a panoramic image.
28. The method of claim 27, further comprising: adjusting a visibility of a pixel in an image buffer by comparing a distance associated with the pixel in the panoramic image with a distance associated with the pixel in the image buffer.
29. The method of claim 21 , further comprising: feathering overlapping edges of an image in an image buffer by degrading visibility of pixels in an overlap area.
30. An imaging system comprising: a first image capture unit to capture a first image; a second image capture unit to capture a second image; a third image capture unit to capture a third image; and a processing unit operationally coupled to the first, second, and third image capture units to receive the first, the second, and the third images, wherein a first portion of the first image can be combined with a portion of the second image to provide a first combined image, wherein a second portion of the first image can be combined with a portion of the third image to provide a second combined image, and wherein the first and second combined images can be displayed to provide a stereoscopic image.
31. The imaging system of Claim 30, wherein said first, the second, and the third image capture units are located approximately equidistant from each other along a substantially arcuate path.
32. The imaging system of Claim 31 , wherein said substantially arcuate path is defined by a substantially spherical body.
33. The imaging system of Claim 30, wherein said first and said second image capture units are separated from each other by an angular distance of about 5 degrees to about
45 degrees along an arc, and wherein said second and said third image capture units are separated from each other by about the angular distance along the arc.
34. The imaging system of Claim 30, wherein a field of view associated with said first image capture unit overlaps a field of view associated with said second image capture unit by an overlap amount, and wherein the field of view associated with said second image capture unit overlaps a field of view associated with said third image capture unit by the overlap amount.
35. The imaging system of Claim 34, wherein the overlap amount is about 10 percent to about 90 percent.
36. The imaging system of Claim 30, wherein a defined image plane associated with said first image capture unit overlaps a defined image plane associated with said second image capture unit by about 1 to about 20 percent.
37. The imaging system of Claim 30, wherein said first portion of said first image is between about 20 percent and about 80 percent of said first image, and wherein said portion of said second image is between about 20 percent and about 80 percent of said second image.
38. The imaging system of Claim 30, wherein a plurality of said first and said second combined images are displayed in sequence to convey motion.
39. The imaging system of Claim 30, wherein the first combined image is combined with a sufficient plurality of images to produce a first combined panoramic image, representing about 90 degrees of a scene, and wherein the second combined image is combined with a sufficient plurality of other images to produce a second combined panoramic image, representing about 90 degrees of the scene, and wherein said first combined panoramic image and said second combined panoramic image are displayed to provide a stereoscopic, panoramic image.
40. The imaging system of Claim 39, wherein a set of first combined panoramic images and a set of second combined panoramic images are displayed in sequence to provide a moving stereoscopic, panoramic image.
41. An imaging system comprising: an image capture unit to provide an image; a processing unit coupled to the image capture unit to receive a first portion of said image to provide a first stereoscopic image, and to receive a second portion of said image to provide a second stereoscopic image.
42. The imaging system of Claim 41, wherein said first stereoscopic image and said second stereoscopic image are combined into a panoramic stereoscopic image.
43. The imaging system of Claim 41 , further comprising: a plurality of image capture units coupled to the processing unit, the plurality of image capture units to provide a plurality of images, wherein selected ones of the plurality of images are combined with at least one other image to produce a plurality of combined images, wherein the plurality of combined images are combined to provide a first panoramic image and a second panoramic image, and wherein the first panoramic image and the second panoramic image are combined to provide a panoramic, stereoscopic image.
44. The imaging system of Claim 43, wherein said first panoramic image displays about 90 degrees of a scene.
45. The imaging system of Claim 43, wherein said panoramic, stereoscopic image displays about 180 degrees of a scene.
46. The imaging system of Claim 41 , further comprising an imaging unit coupled to the processing unit.
PCT/US2003/002285 2003-01-24 2003-01-24 Steroscopic panoramic image capture device WO2004068865A1 (en)

Priority Applications (7)

Application Number Priority Date Filing Date Title
JP2004567640A JP2006515128A (en) 2003-01-24 2003-01-24 Stereo panoramic image capturing device
PCT/US2003/002285 WO2004068865A1 (en) 2003-01-24 2003-01-24 Steroscopic panoramic image capture device
CNA038261855A CN1771740A (en) 2003-01-24 2003-01-24 Steroscopic panoramic image capture device
AU2003214899A AU2003214899A1 (en) 2003-01-24 2003-01-24 Stereoscopic Panoramic Image Capture Device
CA002513874A CA2513874A1 (en) 2003-01-24 2003-01-24 Stereoscopic panoramic image capture device
EP03710742A EP1586204A1 (en) 2003-01-24 2003-01-24 Stereoscopic panoramic image capture device
IL169827A IL169827A0 (en) 2003-01-24 2005-07-21 Steroscopic panoramic image capture device

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/US2003/002285 WO2004068865A1 (en) 2003-01-24 2003-01-24 Steroscopic panoramic image capture device

Publications (1)

Publication Number Publication Date
WO2004068865A1 true WO2004068865A1 (en) 2004-08-12

Family

ID=32823170

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2003/002285 WO2004068865A1 (en) 2003-01-24 2003-01-24 Steroscopic panoramic image capture device

Country Status (7)

Country Link
EP (1) EP1586204A1 (en)
JP (1) JP2006515128A (en)
CN (1) CN1771740A (en)
AU (1) AU2003214899A1 (en)
CA (1) CA2513874A1 (en)
IL (1) IL169827A0 (en)
WO (1) WO2004068865A1 (en)

Cited By (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2008005066A1 (en) * 2006-06-30 2008-01-10 Microsoft Corporation Parametric calibration for panoramic camera systems
WO2012039669A1 (en) * 2010-09-20 2012-03-29 Scalado Ab Method for forming images
US20120243746A1 (en) * 2011-03-22 2012-09-27 Sony Corporation Image processor, image processing method, and program
GB2525170A (en) * 2014-04-07 2015-10-21 Nokia Technologies Oy Stereo viewing
US9196069B2 (en) 2010-02-15 2015-11-24 Mobile Imaging In Sweden Ab Digital image manipulation
US9344642B2 (en) 2011-05-31 2016-05-17 Mobile Imaging In Sweden Ab Method and apparatus for capturing a first image using a first configuration of a camera and capturing a second image using a second configuration of a camera
US9432583B2 (en) 2011-07-15 2016-08-30 Mobile Imaging In Sweden Ab Method of providing an adjusted digital image representation of a view, and an apparatus
EP3198865A4 (en) * 2014-09-22 2017-09-27 Samsung Electronics Co., Ltd. Camera system for three-dimensional video
US9792012B2 (en) 2009-10-01 2017-10-17 Mobile Imaging In Sweden Ab Method relating to digital images
US11049218B2 (en) 2017-08-11 2021-06-29 Samsung Electronics Company, Ltd. Seamless image stitching
GB2591278A (en) * 2020-01-24 2021-07-28 Bombardier Transp Gmbh A monitoring system of a rail vehicle, a method for monitoring and a rail vehicle
US11205305B2 (en) 2014-09-22 2021-12-21 Samsung Electronics Company, Ltd. Presentation of three-dimensional video

Families Citing this family (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2011082919A (en) * 2009-10-09 2011-04-21 Sony Corp Image processing device and method, and program
CN102959943B (en) * 2010-06-24 2016-03-30 富士胶片株式会社 Stereoscopic panoramic image synthesizer and method and image capture apparatus
TWI506595B (en) * 2011-01-11 2015-11-01 Altek Corp Method and apparatus for generating panorama
JP5754312B2 (en) * 2011-09-08 2015-07-29 カシオ計算機株式会社 Image processing apparatus, image processing method, and program
KR101804199B1 (en) * 2011-10-12 2017-12-05 삼성전자주식회사 Apparatus and method of creating 3 dimension panorama image
CN103096101B (en) * 2011-11-07 2016-03-30 联想(北京)有限公司 Image synthesizing method, device and electronic equipment
KR20150068298A (en) * 2013-12-09 2015-06-19 씨제이씨지브이 주식회사 Method and system of generating images for multi-surface display
CN106535806B (en) * 2014-03-28 2019-06-18 直观外科手术操作公司 The quantitative three-dimensional imaging of surgical scene from multiport visual angle
US10210660B2 (en) * 2016-04-06 2019-02-19 Facebook, Inc. Removing occlusion in camera views
CN106851240A (en) * 2016-12-26 2017-06-13 网易(杭州)网络有限公司 The method and device of image real time transfer

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5023725A (en) * 1989-10-23 1991-06-11 Mccutchen David Method and apparatus for dodecahedral imaging system
US6141034A (en) * 1995-12-15 2000-10-31 Immersive Media Co. Immersive imaging method and apparatus

Family Cites Families (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP0605045B1 (en) * 1992-12-29 1999-03-31 Laboratoires D'electronique Philips S.A.S. Image processing method and apparatus for generating one image from adjacent images
JPH0962861A (en) * 1995-08-21 1997-03-07 Matsushita Electric Ind Co Ltd Panoramic video device
US7015954B1 (en) * 1999-08-09 2006-03-21 Fuji Xerox Co., Ltd. Automatic video system using multiple cameras
JP2002094849A (en) * 2000-09-12 2002-03-29 Sanyo Electric Co Ltd Wide view image pickup device
JP2002159019A (en) * 2000-11-16 2002-05-31 Canon Inc Display control device, imaging position estimating device, display system, image pickup system, image positioning method, imaging position estimating method, and recording medium recorded with process program
JP4590754B2 (en) * 2001-02-28 2010-12-01 ソニー株式会社 Image input processing device
JP2003141562A (en) * 2001-10-29 2003-05-16 Sony Corp Image processing apparatus and method for nonplanar image, storage medium, and computer program

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5023725A (en) * 1989-10-23 1991-06-11 Mccutchen David Method and apparatus for dodecahedral imaging system
US6141034A (en) * 1995-12-15 2000-10-31 Immersive Media Co. Immersive imaging method and apparatus

Cited By (23)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR101330373B1 (en) 2006-06-30 2013-11-15 마이크로소프트 코포레이션 Parametric calibration for panoramic camera systems
WO2008005066A1 (en) * 2006-06-30 2008-01-10 Microsoft Corporation Parametric calibration for panoramic camera systems
US9792012B2 (en) 2009-10-01 2017-10-17 Mobile Imaging In Sweden Ab Method relating to digital images
US9196069B2 (en) 2010-02-15 2015-11-24 Mobile Imaging In Sweden Ab Digital image manipulation
US9396569B2 (en) 2010-02-15 2016-07-19 Mobile Imaging In Sweden Ab Digital image manipulation
WO2012039669A1 (en) * 2010-09-20 2012-03-29 Scalado Ab Method for forming images
US9544498B2 (en) 2010-09-20 2017-01-10 Mobile Imaging In Sweden Ab Method for forming images
US20120243746A1 (en) * 2011-03-22 2012-09-27 Sony Corporation Image processor, image processing method, and program
US9071751B2 (en) * 2011-03-22 2015-06-30 Sony Corporation Image processor method and program for correcting distance distortion in panorama images
US9344642B2 (en) 2011-05-31 2016-05-17 Mobile Imaging In Sweden Ab Method and apparatus for capturing a first image using a first configuration of a camera and capturing a second image using a second configuration of a camera
US9432583B2 (en) 2011-07-15 2016-08-30 Mobile Imaging In Sweden Ab Method of providing an adjusted digital image representation of a view, and an apparatus
GB2525170A (en) * 2014-04-07 2015-10-21 Nokia Technologies Oy Stereo viewing
US10455221B2 (en) 2014-04-07 2019-10-22 Nokia Technologies Oy Stereo viewing
US10645369B2 (en) 2014-04-07 2020-05-05 Nokia Technologies Oy Stereo viewing
US11575876B2 (en) 2014-04-07 2023-02-07 Nokia Technologies Oy Stereo viewing
EP3198865A4 (en) * 2014-09-22 2017-09-27 Samsung Electronics Co., Ltd. Camera system for three-dimensional video
US10257494B2 (en) 2014-09-22 2019-04-09 Samsung Electronics Co., Ltd. Reconstruction of three-dimensional video
US10313656B2 (en) 2014-09-22 2019-06-04 Samsung Electronics Company Ltd. Image stitching for three-dimensional video
US10547825B2 (en) 2014-09-22 2020-01-28 Samsung Electronics Company, Ltd. Transmission of three-dimensional video
US10750153B2 (en) 2014-09-22 2020-08-18 Samsung Electronics Company, Ltd. Camera system for three-dimensional video
US11205305B2 (en) 2014-09-22 2021-12-21 Samsung Electronics Company, Ltd. Presentation of three-dimensional video
US11049218B2 (en) 2017-08-11 2021-06-29 Samsung Electronics Company, Ltd. Seamless image stitching
GB2591278A (en) * 2020-01-24 2021-07-28 Bombardier Transp Gmbh A monitoring system of a rail vehicle, a method for monitoring and a rail vehicle

Also Published As

Publication number Publication date
IL169827A0 (en) 2007-07-04
CA2513874A1 (en) 2004-08-12
CN1771740A (en) 2006-05-10
EP1586204A1 (en) 2005-10-19
JP2006515128A (en) 2006-05-18
AU2003214899A1 (en) 2004-08-23

Similar Documents

Publication Publication Date Title
US6947059B2 (en) Stereoscopic panoramic image capture device
US11528468B2 (en) System and method for creating a navigable, three-dimensional virtual reality environment having ultra-wide field of view
WO2004068865A1 (en) Steroscopic panoramic image capture device
US7429997B2 (en) System and method for spherical stereoscopic photographing
CN107637060B (en) Camera rig and stereoscopic image capture
CN109076172B (en) Method and system for generating an efficient canvas view from an intermediate view
JP4642723B2 (en) Image generating apparatus and image generating method
US20120249730A1 (en) Stereoscopic panoramic video capture system using surface identification and distance registration technique
WO2012166593A2 (en) System and method for creating a navigable, panoramic three-dimensional virtual reality environment having ultra-wide field of view
CN101938599A (en) Method for generating interactive dynamic panoramic image
JP2002532795A (en) Image correction method for compensating viewpoint image distortion
CA2677884A1 (en) Banana codec
US11812009B2 (en) Generating virtual reality content via light fields
US6836286B1 (en) Method and apparatus for producing images in a virtual space, and image pickup system for use therein
WO2018035347A1 (en) Multi-tier camera rig for stereoscopic image capture
JP4406824B2 (en) Image display device, pixel data acquisition method, and program for executing the method
CN109769111A (en) Image display method, device, system, storage medium and processor
US20130286255A1 (en) Image capturing device and image presentation system
CN108450031A (en) Image capture apparatus
KR20060015460A (en) Stereoscopic panoramic image capture device
CN115174805A (en) Panoramic stereo image generation method and device and electronic equipment
CN108122283B (en) Method for editing VR image by coordinate transformation
JPH11220758A (en) Method and device for stereoscopic image display
EP3229470B1 (en) Efficient canvas view generation from intermediate views
Zaitseva et al. The development of mobile applications for the capturing and visualization of stereo and spherical panoramas

Legal Events

Date Code Title Description
AK Designated states

Kind code of ref document: A1

Designated state(s): AE AG AL AM AT AU AZ BA BB BG BR BY BZ CA CH CN CO CR CU CZ DE DK DM DZ EC EE ES FI GB GD GE GH GM HR HU ID IL IN IS JP KE KG KP KR KZ LC LK LR LS LT LU LV MA MD MG MK MN MW MX MZ NO NZ OM PH PL PT RO RU SC SD SE SG SK SL TJ TM TN TR TT TZ UA UG US UZ VC VN YU ZA ZM ZW

AL Designated countries for regional patents

Kind code of ref document: A1

Designated state(s): GH GM KE LS MW MZ SD SL SZ TZ UG ZM ZW AM AZ BY KG KZ MD RU TJ TM AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HU IE IT LU MC NL PT SE SI SK TR BF BJ CF CG CI CM GA GN GQ GW ML MR NE SN TD TG

121 Ep: the epo has been informed by wipo that ep was designated in this application
DFPE Request for preliminary examination filed prior to expiration of 19th month from priority date (pct application filed before 20040101)
WWE Wipo information: entry into national phase

Ref document number: 2513874

Country of ref document: CA

WWE Wipo information: entry into national phase

Ref document number: 169827

Country of ref document: IL

WWE Wipo information: entry into national phase

Ref document number: 1020057013716

Country of ref document: KR

Ref document number: 2004567640

Country of ref document: JP

WWE Wipo information: entry into national phase

Ref document number: 2003710742

Country of ref document: EP

WWE Wipo information: entry into national phase

Ref document number: 2003214899

Country of ref document: AU

WWE Wipo information: entry into national phase

Ref document number: 1496/KOLNP/2005

Country of ref document: IN

ENP Entry into the national phase

Ref document number: 2005126718

Country of ref document: RU

Kind code of ref document: A

WWE Wipo information: entry into national phase

Ref document number: 20038261855

Country of ref document: CN

WWP Wipo information: published in national office

Ref document number: 2003710742

Country of ref document: EP

WWP Wipo information: published in national office

Ref document number: 1020057013716

Country of ref document: KR