WO1999065235A1

WO1999065235A1 - Image sensor with cropping

Info

Publication number: WO1999065235A1
Application number: PCT/US1999/013243
Authority: WO
Inventors: Mark Hsu; Mitchell Norcross; Georges Auberger; Remy Zimmerman
Original assignee: Logitech, Inc.
Priority date: 1998-06-09
Filing date: 1999-06-09
Publication date: 1999-12-16
Also published as: WO1999065235A9; CN1183743C; AU4437499A; CN1311952A

Abstract

An image sensor (102) which can output only desired portions of an image. Thus, the image is essentially cropped at the time it is produced, eliminating the generation of data from the image sensor (102) corresponding to undesired portions of the data. This reduces the storage requirements for data being digitally processed, reducing or eliminating the need for a frame buffer memory. In a preferred embodiment, a single silicon chip (100) including a CCD array (102) includes a number of registers (112) which can be programmed for each frame to indicate which lines should be read out. Within each line, the chip (100) can be controlled by timing pulses to quickly read out data before or after the desired cropped portion of the image, with the quickly read data simply being discarded. The actual data desired can then be read out at a slower clock rate to allow processing on the fly.

Description

IMAGE SENSOR WITH CROPPING

BACKGROUND OF THE INVENTION The present invention relates to image sensors, and in particular to image sensors in video cameras connected to a bus, such as the universal serial bus (USB) .

The ability to take and store both still and video digital pictures is becoming more important . Digital images can be manipulated on a computer, and transferred over a network or the Internet. Widespread use is limited by the high cost of digital video cameras and still cameras, and accordingly it would be desirable to be able to reduce their cost.

Fig. 1 is a high-level block diagram of a typical video camera. Light is received through a lens 10 and provided to a sensor array 12 , such as a charged coupled device (CCD) array. The detected signals are then sent to an analog front end circuit 14, which amplifies and processes the signals and converts them into digital form. The digitized signals are then stored in a frame buffer memory 16.

A processing circuit 18 digitally manipulates the images, and may be a programmed processor or an application specific integrated circuit (ASIC) . The processing circuit provides the image data to a bus 20 for transmission to a computer.

One bus standard being adopted for peripheral devices connecting to a computer is the USB. Due to the USB being a serial bus, and the fact that it must be time shared with multiple peripherals, there is limited bandwidth for the transferring of data to the computer. Thus, processing circuit 18 will typically do compression operations to reduce the amount of digital bits that need to be sent, and also do some preprocessing, such as scaling, cropping and filtering. A typical image format is the common interchange format (CIF) which provides 352 pixels by 288 lines. Each pixel is typically represented by 24 bits, 8 bits for each of red, green and blue (RGB) . A typical frame rate may be 30 frames 5 per second. As can be seen, this is a large amount of data to be sent over a bus. Typically, the compression techniques may need to reduce the data to around 3 or 4 bits per pixel from 24.

Fig. 2 is a block diagram of a typical CCD array 12. ^•■10 As can be seen, array 12 includes -an array of photodiodes 22 arranged in rows and columns. A number of vertical shift registers 24 are provided for receiving the value from a photodiode once a capacitance associated with each photodiode has been charged up with an image. The vertical shift

15 registers download into a horizontal shift register 26, which serially shifts data out through an output circuit 28.

Fig. 3 illustrates at a high level the timing used to control CCD array 12. A number of regularly spaced horizontal transfer pulses 30 are used to clock the lines into

20 horizontal shift register 26. An enlarge example of a single horizontal transfer pulse 35 is shown. For each such pulse, the data in an entire line are serially clocked out of the chip using clock pulses 37. Once all the data has been clocked out, the next line is shifted down and clocked out of

25 the vertical shift registers into the horizontal shift register 26 by another horizontal transfer pulse. After the data has been transferred, a new series of clock signals 37 clock this new data serially out of output circuit 28, and the process repeats itself until the entire image has been

30 transferred. In the meantime, a new image has been charging the photodiode capacitors, and can then be stored in the vertical shift registers all at once using a vertical transfer pulse 32. Then the process repeats, with the horizontal transfer pulses shifting each line into horizontal shift

35 register 26 for subsequent shifting out of all the bits in a line.

Once a line has been loaded into the horizontal shift register 26, the entire line must be off-loaded before the next horizontal transfer pulse. Otherwise, the data in the shift register would be overwritten and that line of the image would be lost. Also, precise timing needs to be maintained with the horizontal transfer pulses to output all the lines before the next image is read. If this isn't done, an image can be lost if all its lines are read in the allotted time. Accordingly, referring back to Fig. 1, frame buffer memory 16 is used to store at least an entire frame (all the lines of an image) from the CCD array. Typically, the frame buffer memory can be asynchronously accessed by processing circuit 18 to perform compression, cropping, etc. In addition, processing circuit 18 will access the data from frame buffer memory when it has access to bus 20.

Some applications will not require a frame buffer, such as the NTSC standard, which covers transfer from a camcorder to a TV. Since a stream of data can be sent over a dedicated channel, no frame buffer is needed. There is no shared bus to contend with.

For the USB, there are several modes of bus access. There is a control mode, a bulk transfer mode, and an isochronous mode. The isochronous mode is used for transferring video data, and involves negotiating by the peripheral with the bus to obtain a certain bandwidth. Since the bus is shared with other peripherals, the bandwidth may be limited. The maximum guaranteed bandwidth is 1 Kbyte/ms. If the bus is crowded, less than this bandwidth may be provided. In addition, due to bus jitter, the timing of when the Kbyte or less can be sent over the bus may vary. Accordingly, the frame buffer memory also provides a buffering function allowing the processing circuit to adapt to the timing of the bus and have data available.

SUMMARY OF THE INVENTION The present invention provides an image sensor which can output only desired portions of an image. Thus, the image is essentially cropped at the time it is produced, eliminating the generation of data from the image sensor corresponding to undesired portions of the data. This reduces the storage requirements for data being digitally processed, reducing or eliminating the need for a frame buffer memory.

In a preferred embodiment, a single silicon chip including a CCD array includes a number of registers which can be programmed for each frame to indicate which lines should be read out. A first register will store the number of lines to be skipped until reading begins, with a second register storing the number of lines to be read. Within each line, the chip can be controlled by timing pulses to quickly read out data before or after the desired cropped portion of the image, with the quickly read data simply being discarded. The actual data desired can then be read out at a slower clock rate to allow processing on the fly.

The present invention can be used in a video camera which eliminates the frame buffer memory. This elimination of memory chips reduces the cost and size of the video camera circuitry. The present invention accomplishes this by having the bus controller provide signals to the timing generator to control when horizontal transfer pulses are provided. Thus, instead of regularly spaced transfer pulses as set forth in the specifications of the CCD chips, they are irregularly spaced to coincide with the timing of the USB. The timing of the transfer pulses to the CCD also takes into account the availability of buffer space for a full horizontal line in the bus controller.

The timing of the horizontal transfer pulses to the CCD is also affected by the frame rate and compression rate, etc., which affect the amount of data stored in the buffer of the bus controller, and thus the amount of data that can be stored prior to the next bus access. The availability of a line of storage in the bus controller buffer insures that the entire line can be transferred out of the CCD prior to the next horizontal transfer pulse.

By eliminating the frame buffer, the present invention enables the CCD and analog front end circuitry to be integrated onto a single silicon chip. In a preferred embodiment, the timing generator is also integrated onto the same chip. For a further understanding of the nature and advantages of the invention, reference should be made to the following description taken in conjunction with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS Fig. 1 is block diagram of a prior art video camera system.

Fig. 2 is a block diagram of a prior art CCD array. Fig. 3 is a timing diagram of a prior art CCD array chip.

Fig. 4 is a block diagram of a video camera system according to one embodiment of the present invention.

Fig. 5 is timing diagram of the pulses provided to a CCD array in accordance with one embodiment of the present invention.

Fig. 6 is a block diagram of one embodiment of a DSP of Fig. 4.

Fig. 7 is a diagram of an image sensor array illustrating cropping according to the invention.

Fig. 8 is a timing diagram of different timing events during a frame readout.

Fig. 9 is a block diagram illustrating the provision of two different pipelines or channels of image data corresponding to the cropped portion in the background.

Fig. 10 is a block diagram of an image sensor chip according to the invention.

DETAILED DESCRIPTION OF THE INVENTION Fig. 4 is a block diagram of a video camera according to the present invention. This system includes a lens 10 and a sensor array, such as a CCD array 12, as in Fig. 1, and is connected to a USB 20, as in Fig. 1.

The signals from the CCD chip are provided to an analog front end 40 which includes a differential amplifier or correlated double sampler (CDS) 42, which provides the analog signals to an automatic gain control (AGC) circuit 44. The data is then provided to an analog-to-digital converter (ADC) 46.

The digitized signals, rather than being stored in a frame buffer memory as in the prior ar , are then provided 5 directly to a digital processing circuit 48. They are first provided to a video digital signal processor 50 which performs the compression, cropping, scaling and other functions on the data, as well as digital filtering. Once processed, the digital data is provided to a line buffer 52. Line buffer 52 "10 stores a single line of data from "the horizontal shift register of CCD chip 12.

Data from line buffer 52 is provided to a bus interface 54 which includes a bus controller buffer 56 and a bus controller 58. Preferably, bus controller buffer 56 is 15 capable of storing two USB frames at the maximum possible rate, or 2 Kbytes. Depending upon the frame rate, compression rate, etc., this may typically vary up to 10 lines, or even to 15-20 lines if decimation is done.

Bus controller 58 provides a control signal on a 20 line 60 to a timing generator 62. Timing generator 62 provides clocking signals on line 64 to CCD chip 12. Clocking signals 64 include the horizontal and vertical transfer pulses described in Fig. 3. The vertical transfer pulses are provided as in the prior art, periodically loading the charged 25 values from the CCD array. However, the horizontal transfer pulses are varied to match the reading of the data out of the CCD array with the processing by the following circuitry and the transferring of data to the USB.

In one embodiment, the analog front end 40 and the 30 timing generator 62 are integrated onto the same semiconductor chip substrate 41 as CCD array 12. The elimination of the frame buffer allows this single chip to connect directly to the digital processing circuitry. This allows a more compact, less expensive video camera to be built . 35 Fig. 5 illustrates the timing of the horizontal transfer pulses provided on line 64. Rather than a continuous stream of pulses, the bits are clocked out as needed by the processing circuit in groups as illustrated. A first group of three lines 66 is issued, and then there is a delay while these pixels are processed and transferred to the bus. Then, a next of group of two lines 68 may be provided. There might then be a delay depending upon the availability in the line buffer and the timing of transfer of data onto the USB. A single line 69 is shown being transferred next, followed by a subsequent group of four lines transferred by a number of pulses 70. After an entire image has been transferred, a vertical transfer pulse 72 is provided. The example of Fig. 5 is intended to be simply illustrative. Note that because the horizontal pulses are provided between the same vertical pulses, in order to allow the gaps shown, the pulses are provided much closer together. Thus, bursts of small groups of pulses are provided to quickly send pixel data as fast as the circuitry can handle it. In one embodiment, an entire image may be clocked out in a series of closely spaced pulses, allowing the processing to complete before the next vertical transfer pulse. If the processing still is not completed at the time of the next vertical transfer pulse, there can be a delay, which conforms to the timing of the processing and the availability of the USB. Thus, the horizontal pulses may all be sequentially provided, but at a faster rate than in the prior art and the timing of this grouping of pulses after the vertical pulse can be varied to be right after, or just before the next vertical pulse, in order to accommodate the processing and bus timings.

Fig. 6 illustrates the main components of one embodiment of video DSP 50 of Fig. 4. A first block 74 is a circuit which extracts YUV information from the mosaic presented by the CCD. This is followed by a sealer 76, which transforms the image to fit the desired X and Y scales. Finally, a compression circuit 78 compresses the data. Each of these elements include local memory to store the data while it is being processed. Typically, 2 lines of buffer are used for circuit 74, while a single line of buffer is used for sealer 76 and another single line buffer is needed for compression circuit 76. The present invention includes these internal dedicated buffers as well, but eliminates the need for the frame buffer external to the DSP. The internal DSP line buffers are small enough to be integrated onto the same chip as the logic they support.

In operation, when the video camera is activated, bus controller 58 will negotiate with the USB to obtain a guaranteed amount of data per millisecond in bandwidth. _Once this amount of bandwidth has been determined, the frame rate, frame size and compression rate are programmed by processing circuit 48 to accommodate the bandwidth available. This will determine the number of lines which will be read out of the bus controller buffer 56 on each transfer over the USB and thus the rate at which lines can be read out of CCD array 12.

In a preferred embodiment, timing generator 62 automatically generates the horizontal transfer pulses automatically, unless inhibited by the bus controller. The timing generator is programmed to generate the pulses at a faster rate than the prior art, allowing compression by not inhibiting pulses, and spreading out pulses by inhibiting an appropriate number of pulses. Fig. 7 illustrates an image sensor array 80 with a cropped area 82. In the prior art, the entire image 80 would be sent to signal processing circuitry, which would then pick out the pixels corresponding to area 82 for processing in a cropping operation. In this invention, rather than reading out the entire array 80 from the image sensor, the reading starts at a line 84, skipping the lines up until then. The reading will stop at a line 86. As can be seen, this will greatly reduce the amount of data to be processed, and thus reduce the amount of storage space necessary. Additionally, for each line, it is most economical to simply read out the entire line, rather than varying the starting position, which would require additional circuitry. Using this method, all of the pixels up to a pixel 88 could be read at a very high clock speed, and discarded. Pixels in the target area from pixel 88 to pixel 90 could then be clocked out to be processed. Pixels between pixel 90 and the end of the image would then be clocked out at a fast rate again, and again discarded. Fig. 8 is a timing diagram illustrating a typical frame duration, with different time periods tl-t6 illustrated (tl is the beginning of a frame) . A timing generator can generate these timings to the array, and can be either located off chip, or could be integrated onto the image sensor chip itself, with registers on the chip storing certain parameters described below to control the timing.

Five major parameters are stored in registers to control the sequencing of the sensor readout: 1. Ti eToFirstLine: number of clock cycles from the end of exposure (or transfer from photosites to array) to start of first line readout. [t2]

2. SkippedLines : number of lines to skip before active reading from lines. This can be used in two different way: as a start address for line or as a counter.

3. TargetLines: number of lines read out for further processing. The combination of SkippedLines and TargetLines defines the target band (area of interest) , eliminating time overhead in the cropped zone.

4. TimeToNextLine: number of clock cycles from the last pixel of one target line to the first black pixel of the next target line. This parameter is used to spread out the reading of target lines over time. [t5] 5. TimeToNextFrame : number of clock cycles from the last line sequence to the beginning of the next frame (photosite transfer) . [t6]

The following fixed parameters are defined for a given sensor: PixelsPerLine being the total number of pixels in a line (black and active) , [t4]

LinesPerFrame being the total number of lines in a frame,

SkipTime being the time required to skip a line (should be as small as possible) , [t3] (*)

TransferTime being the time overhead for the transfer of charged image data from photosites into the vertical registers, [tl] (*) PixelClock being the sensor (pixel) clock frequency, preferably 12MHz. The frame rate is then expressed as :

1/FrameRate = PixelClock * [TransferTime + TimeToFirstLine ₊

(LinesPerFrame - AvtiveLines) *SkipTime + TargetLines* (PixelsPerLines + TimeToNextLine) ₊ TimeToNextFrame] In one embodiment, t2 and t6 have a resolution of a PixelClock period to ensure maximum flexibility. At a minimum, either t2 or t6 have such a resolution, allowing a clock cycle resolution for the whole frame time with a coarser resolution for the other variables. This could save a few bits on some counters . To reach frame rates between 10 fps and 60 fps, and to accommodate some processing/software backend requirements, the programmable parameters have the following range in one embodiment :

1. TimeToFirstLine: up to 5 ms (*) 2. SkippedLines: 0 to LinesPerFrame - 1

3. TargetLines : 1 to LinesPerFrame

4. TimeToNextLine: up to 5 ms (*)

5. TimetoNextFrame: up to 5 ms (*) [larger for low frame rate] (*) : minimum values to allow frame rate as high as possible.

Programming Interface

Registers are programmed through an I2C interface. Registers are doubled buffered, allowing an active register change only at frame boundaries, keeping register values consistent over the whole frame time.

In one embodiment, the I2C interface is characterized to run at 1 Mhz to ensure timely transfer of all parameters during the duration of a frame.

In one embodiment, instead of sending only cropped area 82 of Fig. 7, area 82 can be sent more often than the entire array 80. For example, where the cropped area includes the portion of the image which has movement, this may be updated more frequently than the rest of the image, which provides the background. For example, area 82 might be sent at 100 frames per second, while the entire area might be sent at 25 frames per second. This would result in the entire image only being sent once for every four transmissions of area 82.

Fig. 9 illustrates a block diagram of a system taking advantage of this aspect of the invention. Image sensor 92 provides data to a digital signal processing (DSP) circuit 94, which separately processes the background and cropped areas. The background can be transmitted intermittently through a transmission pipeline 96 as a first channel, while the movement area 82 can be transmitted through a second pipeline 98. These could be transmitted over a USB as two separate channels of information. Thus, the invention switches the frame size and frame rate on the fly between two different versions.

Fig. 10 illustrates one embodiment of a semiconductor chip 100. A sensor array 102 provides data to a shift register 104, where it is clocked out serially by an output circuit 106. This circuitry is that found in a standard CCD array. The present invention adds onto the same chip a timing generator 108, rather than having the timing externally generated. Also provided is an array control circuit 110 for providing control signals to not just the sensor array, but the other elements of chip 100 (the control lines are not shown to avoid complicating the drawing) . A group of registers 112 store the parameters needed for sequencing of the sensor readout, as described above. Each register is two registers, providing double buffering. Thus, data can be read in on the first register, while the previous data in the second register is being used to process the current frame. Once the current frame readout is completed, the data from the first register can be transferred to the second, or simply directly used for the next frame, while the other half of the double buffer register can then be loaded for the subsequent frame timing. Preferably, an I2C bus 114 is used, using the standard I2C specification calling for one data and one control line.

In another aspect of the invention, the other analog circuitry for processing the sensor data is provided on chip 100. In particular, a correlated double sampler (CDS) 116 connects to output 106. An automatic gain control (AGC) circuit 118 is connected to CDS 116. Finally, an analog-to- digital converter (ADC) 120 converts the pixel data into digital form, for transmission over a serial output line 122 for digital processing, such as by DSP 94 of Fig. 9.

As will be understood by those of skill in the art, the present invention may be embodied in other specific forms without departing from the essential characteristics thereof. For example, instead of quickly reading unwanted pixels in a line, multiple outputs of the horizontal shift register of the CCD could be provided. Although having an output at every pixel location would add significant circuiry and line routing, a coarse resolution of an output every 10 or 20 pixels, for example, could be added. The fine resolution could then be achieved by the varying clock read out rates, as described above. Accordingly, the foregoing description is intended to be illustrative, but not limiting, of the scope of the invention which is set forth in the following claims.

Claims

WHAT IS CLAIMED IS:

1. A method for operating an image sensor comprising the steps of: starting reading of said sensor at a first selected line, which may be other than a first line; and stopping reading of said sensor at a second selected line, which may be other than a last line of said image sensor.

2. The method of claim 1 further comprising the steps of : selecting a starting and stopping location in said first selected line; and reading data between said starting and stopping locations at a different rate than data outside said starting and stopping locations.

3. The method of claim 2 wherein said data outside said starting and stopping locations is read for each read of said data between said starting and stopping locations, but at a different clock rate.

4. The method of claim 2 wherein said data outside said starting and stopping locations is not read for some reads of said data between said starting and stopping locations .

5. The method of claim 1 further comprising the steps of: converting said data into digital form; and transferring said data in digital form directly to a digital signal processing circuit without intermediate storage in a frame buffer.

6. The method of claim 1 further comprising the steps of: processing image data, read from said image sensor, in a video digital signal processor (DSP) , said processing including performing image compression; providing processed image data to a bus; and providing a clock signal to said image sensor to transfer image data from said image sensor array at a rate corresponding to a data transfer rate over said bus, such that a frame buffer memory for storing an entire frame of said image is not needed.

7. A method for operating an image sensor comprising the steps of: starting reading of said sensor at a first selected line, which may be other than a first line; stopping reading of said sensor at a second selected line, which may be other than a last line of said image sensor; selecting a starting and stopping location in said first selected line; reading data between said starting and stopping locations at a different rate than data outside said starting and stopping locations, wherein said data outside said starting and stopping locations is read for each read of said data between said starting and stopping locations, but at a different clock rate; converting said data into digital form; and transferring said data in digital form directly to a digital signal processing circuit without intermediate storage in a frame buffer.

8. A image sensor comprising: an array of image sensing elements; a first register configured to store information to identify a start reading line of said array, which may be other than a first line; and a second register configured to store information indicating a number of lines of said array to be read.

9. The image sensor of claim 8 wherein said registers are double buffered registers .

10. The image sensor of claim 8 further comprising an I2C interface to said registers.

11. The sensor of claim 8 further comprising: a third register configured to store a number of clock cycles from the end of an exposure of said image sensing elements to a start of a first line readout; a fourth register configured to store a number of clock cycles from a last pixel of one target line to a first pixel of a next target line; a fifth register configured to store a number of clock cycles from a last line read in a frame to the beginning of a next frame; and a control circuit configured to read data between said at locations and at a rate specified by said registers.

12. The image sensor of claim 8 further comprising: a timing generator configured to provide a clock signal to said image sensor array to transfer image data from said image sensor array at a rate corresponding to a data transfer rate over a bus, such that a frame buffer memory for storing an entire frame of said image is not needed.

13. A image sensor comprising: an array of image sensing elements on a silicon substrate; a first register on said silicon substrate configured to store information to identify a start reading line of said array, which may be other than a first line; a second register on said silicon substrate configured to store information indicating a number of lines of said array to be read; and a timing generator on said silicon substrate configured to provide a clock signal to said image sensor array to transfer image data from said image sensor array at a rate corresponding to a data transfer rate over a universal serial bus (USB) , such that a frame buffer memory for storing an entire frame of said image is not needed.