WO2002045004A1

WO2002045004A1 - Method of image processing utilizing a pipelined inverse wavelet transform

Info

Publication number: WO2002045004A1
Application number: PCT/US2001/046734
Authority: WO
Inventors: Vance Faber; Roland A. Sweet; Jonathan N. Bradley; Randall L. Dougherty
Original assignee: Lizardtech, Inc.
Priority date: 2000-12-01
Filing date: 2001-12-03
Publication date: 2002-06-06
Also published as: AU2002225945A1

Abstract

A method for computing a discrete wavelet transform of an image using an amount of memory that is proportional to only one dimension of the image is provided. The method is specifically independent of the other dimension of the image, and is generally performed in a nonrecursive fashion. In one case, a multi-level forward discrete wavelet transform (FDWT) may be computed. In this case, the image may be read into the main memory by horizontal strips, while the algorithm proceeds by processing rows (e.g., lines) within the strips (106). In another case, a multi-level inverse discrete wavelet transform (IDWT) may be computed in order to reconstruct an arbitrary scene of any resolution of a wavelet-encoded image. In this case, the scene may be produced in the primary memory by rows.

Description

METHOD OF IMAGE PROCESSING UTILIZING A PIPELINED INVERSE

WAVELET TRANSFORM

CROSS-REFERENCE(S) TO RELATED APPLICATION(S) This application claims the benefit of prior provisional U.S. Application Serial

No. 60/250,855, filed December 1, 2000, under 35 U.S.C. § 119.

FIELD OF THE INVENTION This invention relates generally to methods for image processing and compression, and more particularly, to methods for image processing and compression utilizing discrete wavelet transforms.

BACKGROUND OF THE INVENTION As more and more images are being stored on an access by computers, the need for better processing and compression techniques is increasing. Discrete wavelet transforms (DWT) are used in a number of highly efficient image processing techniques. An exemplary method of image processing utilizing discrete wavelet transforms is taught in U.S. Patent No. 5,710,835 to Bradley. Another method is taught in U.S. Patent No. 6,201,897, to Nixon.

The '897 patent is directed to an image processing system and method that perform discrete wavelet transformation (DWT) on very large images. The invention described in the '897 patent application transforms the data of an image using a DWT process by acquiring the image line-by-line and, while acquiring the image, providing the succession of image lines to low-pass and high-pass digital filters for DWT filtering at various levels using a recursive procedure. In the initial stage of the DWT transformation, each input line is individually filtered (in a horizontal direction) by low- pass and high-pass filters to produce "intermediate results" that are stored in pre-allocated memory buffers. Each intermediate result buffer contains several lines of intermediate results. The columns of the intermediate results in the buffers are then filtered (in a vertical direction) by low-pass and high-pass filters to produce four subbands of the DWT. The subbands are identified as LL (low-pass/low-pass). LH (low-pass/high-pass),

HL (high-pass/low-pass), and HH (high-pass/high-pass). The LL, LH, HL, and HH designations indicate the type of filtration performed on the input lines and the type of filtration performed on the columns ofthe intermediate results. The LL subband is thus a reduced-resolution version of the original image formed by low-pass filtration of the input lines (resulting in low-pass intermediate results stored in the buffers) and low-pass filtration of the columns of the intermediate results buffers. The DWT is repeated through multiple levels, with the LL subband of the previous level being the input data for DWT processing by the next level. According to the described method, the DWT process is applied at each level using recursion to obtain the LL subband from a previous level, until the LL subband resolution is no longer than the filter being applied. The invention described in the '897 patent application operates by determining a number N of levels into which an image will be transformed, the levels being ordered from a highest level (closest to the original image) to a lowest level. Buffers for the

N levels are allocated in the memory. The image is then transformed by recursively performing the following steps while the image is being acquired: (1) requesting at least one line of the image for the highest level of transformation; (2) producing transformation i results in response to provision of the at least one line of the image data to the highest level, the transformation results including, at each level, intermediate results and subband results derived from the intermediate results; and (3) during production of the transformation results, storing the recently-produced intermediate results in the allocated buffers, while discarding earlier-produced intermediate results from the buffers.

As explained in the '897 patent, the transform method ofthe invention operates by sequentially requesting a single line of LL output for the lowest level of the DWT (i.e., the smallest level DWT). The smallest DWT, in turn, requests two LL lines of output from the next higher level (i.e., next larger level) to be used as input for the smallest level DWT. This process proceeds recursively until the largest level requests two image lines from the source image. Once a level DWT has obtained two larger level LL subband lines, the level DWT then produces the LL, LH, HL, and HH subband lines for the level. The LH, HL, and HH subbands are compressed and transmitted or stored. The level DWT immediately returns the generated LL subband line to the next smaller level (as input to the smaller level). The DWT process proceeds by making recursive program calls through higher levels to obtain input lines for the lower levels, with the higher levels returning their generated LL output lines as input for the lower levels. At each level, the DWT discards the two earliest-produced lines from the intermediate results buffer and adds the two LL subband lines most recently obtained from the larger (previous) level.

The recursive DWT procedure performed by the method of the '897 patent is shown in the process of a flow diagram of a Figure 5 and in a pseudo code table. As shown, the DWT operation begins by allocating computer memory for the buffers for each level. In the recursive routine, the particular level determines whether another output line is to be generated at that level. If so, the level requests two input LL lines from the next larger level via a recursive program call. The recursion begins at the lowest level and cascades upwardly through each level, triggering recursive reads of LL lines at the higher levels, up to the point where the original image lines are being input into the highest level DWT for processing. As long as lines are being input into the transform process, an exit will be taken, and two LL lines from the next larger level will be obtained, generating low-pass and high-pass intermediate results that are stored in the level's buffer as the newest intermediate results. The LL, LH, HL, and HH subband lines are generated from the intermediate results, with the LH, HL, and HH lines being compressed and transmitted or stored. The LL line is returned to the next lower level DWT that made the recursive program call to the higher level.

The pseudo code provided in the '897 patent depicts the recursion techniques. For each of the smallest level DWT lines, the main program invokes a routine called "dwt_level_line". Each time dwt_level_line is called, the routine determines whether it is processing the largest level DWT line. If not, the routine invokes another instance of itself by calling itself for the next higher level. The routine dwt_level_line recursively calls itself up through the levels until it finds that it is processing the original image data at the largest level. At that point, with that condition met, the particular instance of dwt_level_line at the highest level computes the LL, LH, HL, and HH subbands. The LL subband produced by that instance of dwt_level_line is then returned to the previous lower level instance of dwt_level_line and processing at the lower level continues, either by recursively calling itself again (to obtain another LL line from the next higher level) or by computing the LL, LH, HL, and HH subbands for the level. The LL subband is always immediately returned to the previous lower level instance of dwt_level_line unless the processing is already at the lowest level (at which point, all four subbands are compressed, transmitted, and/or stored). The present invention is directed to an improved method of image processing utilizing discrete wavelet transforms. More specifically, the present invention is directed to a method of image processing utilizing discrete wavelet transforms wherein the amount of memory that is used is proportional to only one dimension of the image and is independent ofthe other dimension.

SUMMARY OF THE INVENTION A method of image processing that computes a multi-level discrete wavelet transform using an amount of memory that is proportional to only one dimension of the image is provided. In accordance with one aspect ofthe invention, in one embodiment a method is provided for computing a multi-level forward discrete wavelet transform (FDWT) using an amount of primary memory that is proportional to only one dimension ofthe image, and specifically independent ofthe other dimension, and proportional to the support of the FDWT. The method is generally performed in a nonrecursive fashion. The image may be read into the main memory by horizontal strips, while the algorithm may proceed by processing rows (e.g., lines) within the strips. The method includes allocating zero or more buffers in main memory to store a strip of image data (e.g., a set of consecutive image rows), a strip of he lower resolution image on each ofthe ni levels, and subband buffers to store certain results of the FDWT. The method also includes initializing the buffers, repeatedly performing a number of processing steps, and flushing the buffers after processing the last row(s) of the image and/or the lower resolution images. The processing steps that are repeatedly performed begin by requesting a strip of the original image. Then, the lines of each strip that are obtained are processed so as to produce partial transformation results at some or all levels, the transformation results including, at each level, intermediate and/or subband data. During production of the transformation results, the recently produced intermediate results are stored in the buffers, while discarding earlier produced intermediate results from the buffers. In addition, some ofthe data in some ofthe subband buffers is forwarded for subsequent processing and/or storage, and the buffers are reset. In addition, the first and last iterations of the processing steps that are repeatedly performed utilize conventional techniques (e.g. boundary conditions) in order to specially handle the beginning and end of the image processing utilizing discrete wavelet transform techniques. In accordance with another aspect of the invention, in one embodiment a method is provided for computing the multi-level inverse discrete wavelet transform (IDWT) to reconstruct an arbitrary scene of any resolution of a wavelet-encoded image using an amount of main memory that is proportional to only one dimension of that scene, and specifically independent of the other dimension, and proportional to the support of the IDWT. The method is generally performed in a nonrecursive fashion. The scene may. be produced in the primary memory by rows. The method includes determining from the location of the scene and the support ofthe IDWT a sufficient subset of each subband of wavelet coefficients to reconstruct the given scene. The method also includes allocating intermediate buffers for the levels in main memory based on the sizes of rows computed above, and the subband buffers for the wavelet coefficients stored in the wavelet-encoded image based on the sizes of rows computed above. The method also includes initializing the buffers, repeatedly performing a number of processing steps, and flushing the buffers. The processing steps that are repeatedly performed begin by requesting rows of the subband buffers containing wavelet coefficients. Then, pixel values are created as a result of partially inverse transforming wavelet coefficients at some or all levels. In addition, rows of image pixels from a given level are moved to the next higher level.

In accordance with another aspect of the invention, in another embodiment, a method is provided where in certain conditions on the vertical boundaries of the original image and all of its lower resolution images are applied to create an encoded image with no apparent vertical boundaries (e.g., ends of horizontal lines). This may give the effect ofthe image appearing to have been affixed to the inside or outside of a vertical cylinder.

In accordance .with another aspect of the invention, in one embodiment, the method operates by creating separate instances of a program object which are referred to as pipe-segments. Each pipe-segment object includes program steps (i.e., functions) for implementing DWT filtering of data that it receives. Each pipe-segment object also includes memory space for holding the data on which it operates. The pipe-segment objects, when created, are located in computer memory at different memory addresses. The main control program invokes a pipe-segment object by mvoking one of the functions defined for a pipe-segment object that performs a specific task.

In accordance with another aspect of the invention, each pipe-segment object is assigned to and performs one level of DWT transformation. A multi-level DWT is achieved by directing the pipe-segment object assigned to one level to extract a line of LL subband data from the pipe-segment object assigned to the next higher level in the DWT. The pipe-segment objects may thus be conceptually viewed as ordered in a row, or a pipeline, extending from the highest level (i.e., the largest in size) DWT to the lowest level (i.e., the smallest in size) DWT. In accordance with another aspect ofthe invention, each pipe-segment object may employ a wavelet (e.g., a "5,3" wavelet) in performing the DWT. It will be appreciated that other types of wavelets may also be used. In the 5, 3 embodiment the vertical (column-based) low-pass filtering for each image line at each level is ultimately supported by data in five input lines, and the vertical high-pass filtering for each image line at each level is supported by data in three input lines. Similarly, the horizontal (row- based) low-pass and high-pass filtering for each image line at each level is supported by five row values and three row values, respectively. (Alternatively, each pipe-segment object may employ a "9,7" wavelet which may be described similarly.)

In accordance with another aspect of the invention, the main control program controls which ofthe pipe-segment objects is invoked at any instance of time. The main control program operates by "looking down the pipeline," beginning at the highest level DWT, to determine which level pipe-segment object is the first to have a "hole" requiring data from the next higher level pipe-segment object. A hole indicates that LL subband data is needed from the next higher level pipe-segment object in order to proceed with processing the DWT. For instance, if the highest level pipe-segment object (referred to herein as "PSI") is the first pipe-segment object in the pipe to need data for DWT processing, the main control program invokes a function of PSI that pulls (i.e., reads in) two lines of original image data for DWT processing. The LL, LH, HL, and HH subband lines resulting from the DWT processing in PSI are retained in PSl's memory space, and PSI is considered "filled." Continuing with this example, the main control program again searches for the first pipe-segment object to have a "hole," which, in this case, is not PSI since the DWT was just completed for PSI. If the main control program finds that the second highest level pipe-segment object ("PS2") is also "filled," the control program proceeds to the third highest level pipe-segment object ("PS3"). Assuming, for example, that PS3 has a hole, the main control program invokes a function of PS3 that pulls the line of LL subband data from the next higher level pipe-segment object, namely, PS2.

In accordance with another aspect of the invention, the DWT processing at any level requires two lines of LL subband data (or two lines of original image data for the highest level DWT). Continuing with the foregoing example, if the LL subband line obtained by PS3 is only the first of the two LL subband lines it needs to perform the DWT, PS3 is still considered to have a "hole," i.e., it is not "filled." In addition, when PS3 pulled the LL subband line from PS2, it created a hole in PS2. Thus, when the main control program again searches for the first pipe-segment object to have a hole, the main control program will first encounter the hole in PS2. At that point, the main control program invokes a function of PS2 which pulls the line of LL subband data from PSI, causing a hole to occur in PSI. The hole in PSI is filled on the next iteration (since PSI is now the first pipe-segment object in the pipe to have a hole) by pulling two lines of original image data into PSI and performing the DWT.

It will be appreciated that the disclosed methods of image processing are advantageous in that they allow for the computation of multi-level discrete wavelet transforms using an amount of memory that is proportional to only one dimension of the image.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing aspects and many ofthe attendant advantages of this invention will become more readily appreciated as the same become better understood by reference to the following detailed description, when taken in conjunction with the accompanying drawings, wherein: FIGURE 1 is a flow diagram illustrating a method for computing a multi-level forward discrete wavelet transform in accordance with the invention;

FIGURE 2 is a flow diagram illustrating a method for computing a multi-level inverse discrete wavelet transform in accordance with the present invention;

FIGURE 3 is a block diagram illustrating a flow of data as an image is processed in accordance with the present invention;

FIGURES 4A-4D are matrices utilized in a forward discrete wavelet transform when the size of a row is even;

FIGURES 5A-5D are matrices utilized in a forward discrete wavelet transform when the size of a row is odd; FIGURES 6A-6F are matrices utilized in an inverse discrete wavelet transform;

• FIGURE 7 is a block diagram illustrating the flow of data through a multi-level discrete wavelet transform; FIGURE 8 is a diagram showing the application of the factors of an inverse discrete wavelet transform;

FIGURE 9 is a diagram illustrating flow of data through a forward multi-level discrete wavelet transform; FIGURE 10 is a diagram illustrating two rows of original image data as cells in an array;

FIGURE 11 is a diagram illustrating the data of FIGURE 10 after being processed by a two-dimensional discrete wavelet transform;

FIGURES 12A-12D are diagrams illustrating the dispersion of the data of FIGURE 11 into four separate arrays;

FIGURE 13 is a block diagram illustrating a pipelined inverse multi-level discrete wavelet transform;

FIGURE 14 is a diagram illustrating one row ofthe pipe segment;

FIGURE 15 is a diagram schematically illustrating the computation of the horizontal component ofthe inverse discrete wavelet transform;

FIGURE 16 is a diagram illustrating the contents of a pipe segment for an interior seam;

FIGURE 17 is a diagram illustrating the form of a pipe segment starting at the boundary ofthe image; FIGURE 18 is a diagram illustrating the form of the pipe segment near but not at the left boundary of the image;

FIGURE 19 is a diagram of a muster array when there is one remaining row in the previous segment wave array and in a first terminal node of a subband tree while no rows remain in a second and third terminal nodes ofthe subband tree; FIGURE 20 is a diagram illustrating a muster array when there is one remaining row in the previous segment wave array and in a first, second and third terminal nodes of the subband tree;

FIGURE 21 is a diagram of a muster array when there is two remaining rows in the previous segment wave array and a first terminal node of the subband tree but only one row in a second and third terminal nodes ofthe subband tree;

FIGURE 22 is a diagram illustrating pipeline stages for a 5-3 wavelet;

FIGURE 23 is a diagram illustrating pipeline stages for a 9-7 wavelet with standard symmetric factors; FIGURE 24 is a diagram illustrating pipeline stages for a 9-7 wavelet with alternative factors;

FIGURE 25 is a diagram of an alternate version for standard factors;

FIGURE 26 is a diagram of an alternate version for alternative factors: FIGURE 27 is a diagram illustrating reduced storage requirements from that shown in FIGURE 26;

FIGURE 28 is a diagram illustrating reduced storage requirements from that shown in FIGURE 27; and

FIGURE 29 is a diagram illustrating pipeline stages and extra information that is needed for processing the termination boundary conditions.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT The present invention provides methods for computing a multi-level discrete wavelet transform using an amount of primary memory that is proportional to only one dimension of an image. As will be described below, in one embodiment, a method is provided for computing a multi-level forward discrete wavelet transform (FDWT). When an FDWT is being computed, the image may be read into the main memory by horizontal strips, while the algorithm proceeds by processing rows (e.g., lines) within the strips. As will also be described in more detail below, in another embodiment, a method is provided for computing a multi-level inverse discrete wavelet transform (IDWT) to reconstruct an arbitrary scene of any resolution of a wavelet encoded image. When an IDWT is being computed, the scene may be produced in the primary memory by rows.

FIGURE 1 is a flow diagram illustrating a method for computing a FDWT. The method is generally performed in a nonrecursive fashion. The method may comprise the following steps: allocating zero or more buffers in main memory to store (a) a strip of image data

(a set of consecutive image rows), (b) a strip of lower resolution image on each ofthe nj levels, and (c) subband buffers to store certain results ofthe FDWT, initializing said buffer(s); repeatedly performing the following steps, although the first and last iterations of these steps may be handled specially in accordance with conventional discrete wavelet transform processing techniques: requesting a strip ofthe original image; processing the lines of each strip obtained above, thereby producing partial transformation results at some or all levels — the transformation results including, at each level, intermediate and or subband data; during production of transformation results, storing recently produced intermediate results in the buffers, while discarding earlier produced intermediate results from the buffers; forwarding some data in some subband buffers for subsequent processing and/or storage, and resetting those buffers; and flushing said buffers, and processing the last row(s) of the image and/or the lower resolution images.

FIGURE 1 is a flow chart of a FDWT routine 100. As shown in FIGURE 1, at a block 102, buffers in main memory are allocated and initialized. At a block 104, a strip of rows ofthe image are read into main memory. At a block 106, the routine transforms a . group of rows of the image in the strip and transmits completed rows of the image between sections of the pipeline.

At a decision block 108, the routine determines whether there are more rows in the strip. If there are more rows in the strip, the routine returns to block 106. If there are not more rows in the strip, then the routine proceeds to a decision block 110.

At decision block 110, the routine determines whether there are more rows to process in the image. If there are more rows to process in the image, the routine returns to block 104. If there are not more rows to process in the image, the routine continues to a block 112. At block 112, the routine flushes the remaining intermediate results through the remainder ofthe pipeline. The routine then ends.

As will be described in more detail below, FIGURE 2 is a flow diagram illustrating a method for computing an IDWT. The method is generally performed in a nonrecursive fashion. The method may be comprised ofthe following steps: determining from the location of the scene and the support of the IDWT a sufficient subset of each subband of wavelet coefficients to reconstruct the given scene; allocating intermediate buffers for (a) the levels in main memory based on the sizes computed above and (b) the subband buffers for the wavelet coefficients stored in the wavelet-encoded image based on the sizes computed above; initializing said buffers; repeatedly performing the following steps [note the first and last steps are special]: requesting rows ofthe subband buffers containing wavelet coefficients; creating pixel values as a result of partially inverse transforming wavelet coefficients at some or all levels, in response to the requesting of rows above; moving rows of image pixels from a given level to the next higher level; and flushing said buffers.

FIGURE 2 shows an IDWT routine 120 that corresponds to the method described above. As shown in a block 122, the routine begins with a given scene request and encoded image information. At a block 124, buffers are allocated and initialized. At a block 126, a row of icon is put into the pipeline. At a block 128, the icon row is bordered with subband data, IDWT is performed, and the completed image rows are moved through the pipeline. At a decision block 130, the routine determines whether more rows of icon are needed. If more rows of icon are needed, the routine returns to block 126. If more rows of icon are not needed, the routine proceeds to a block 132. At block 132, the routine completes the transfer and IDWT on rows in the pipeline and as each segment completes its rows, removes it from the pipeline. The routine then ends. It will be appreciated that when a subset of an image is being processed as described above, the subset may consist of a subrectangle of an image of any resolution, even including a single row, column, pixel, or any constituent part thereof. In addition, for the FDWT routine described above with respect to FIGURE 1, while the image may be read into the main memory by horizontal strips, and the algorithm may proceed by processing rows (e.g., lines) within the strips, other processing techniques can also be utilized. Similarly, for the IDWT routine described above with respect to FIGURE 2, while the scene may be produced in the primary memory by rows, other techniques may also be envisioned.

As will be described in more detail below, FIGURE 3 is a block diagram illustrating a flow of data through pipe segments in accordance with the present invention. In one embodiment of the invention, the described methods operate by creating separate instances of a program object, which are referred to as pipe-segments.

Each pipe-segment object includes program steps for implementing DWT filtering of data that it receives. Each pipe-segment object also includes memory space for holding the data on which it operates. The pipe-segment objects, when created, are located in computer memory at different memory addresses. The main control program invokes a pipe-segment object by invoking one of the functions defined for a pipe-segment object that performs a specific task.

Each pipe-segment object is assigned to and performs one level of DWT transformation. A multi-level DWT is achieved by directing the pipe-segment object assigned to one level to extract a line of LL subband data from the pipe-segment object assigned to the next higher level in the DWT. The pipe-segment objects may thus be conceptually viewed as ordered in a row, or a pipeline, extending from the highest level (i.e., the largest in size) DWT to the lowest level (i.e., the smallest in size) DWT.

In one embodiment, each pipe-segment object employs a "5,3" wavelet in performing the DWT. This means that the vertical (column-based) low-pass filtering for each image line at each level is ultimately supported by data in five input lines, and the vertical high-pass filtering for each image line at each level is supported by data in three input lines. Similarly, the horizontal (row-based) low-pass and high-pass filtering for each image line at each level is supported by five row values and three row values, respectively. (Alternatively, each pipe-segment object may employ a "9,7" wavelet which may be described similarly.) The main control program controls which of the pipe-segment objects is invoked at any instance of time. The main control program operates by "looking down the pipeline," beginning at the highest level DWT, to determine which level pipe-segment object is the first to have a "hole" requiring data from the next higher level pipe-segment object. A hole indicates that LL subband data is needed from the next higher level pipe- segment object in order to proceed with processing the DWT. For instance, if the highest level pipe-segment object (referred to herein as "PSI") is the first pipe-segment object in the pipe to need data for DWT processing, the main control program invokes a function of PSI that pulls (i.e., reads in) two lines of original image data for DWT processing. The LL, LH, HL, and HH subband lines resulting from the DWT processing in PSI are retained in PSl's memory space, and PSI is considered "filled."

Continuing with this example, the main control program again searches for the first pipe-segment object to have a "hole," which, in this case, is not PSI since the DWT was just completed for PSI. If the main control program finds that the second highest level pipe-segment object ("PS2") is also "filled," the control program proceeds to the third highest level pipe-segment object ("PS3"). Assuming, for example, that PS3 has a hole, the main control program invokes a function of PS3 that pulls the line of LL subband data from the next higher level pipe-segment object, namely, PS2. DWT processing at any level requires two lines of LL subband data (or two lines of original image data for the highest level DWT). Continuing with the foregoing example, if the LL subband line obtained by PS3 is only the first of the two LL subband lines it needs to perform the DWT, PS3 is still considered to have a "hole," i.e., it is not "filled." In addition, when PS3 pulled the LL subband line from PS2, it created a hole in PS2. Thus, when the main control program again searches for the first pipe-segment object to have a hole, the main control program will first encounter the hole in PS2. At that point, the main control program invokes a function of PS2 which pulls the line of LL subband data from PSI, causing a hole to occur in PSI. The hole in PSI is filled on the next iteration (since PSI is now the first pipe-segment object in the pipe to have a hole) by pulling two lines of original image data into PS 1 and performing the DWT.

FIGURE 3 illustrates the flow of data through various pipe segments in accordance with the present invention. Blocks representing various resolutions of an image are shown, including a high resolution block 200, a medium resolution block 201, and a low resolution block 202. Block 200 includes a subset 210a, while block 201 includes a subset 211a, and block 202 includes a subset 212a. Subset 212a is related to subsets 210a and 211a.

FIGURE 3 illustrates how pipe segments can be viewed as "windows" on the original image and its various lower resolutions created by the multilevel FDWT. Blocks 200, 201, and 202 represent the original image and two lower resolutions in the case of two levels. Blocks 210a, 211a, and 212a represent windows on these images. Pipe segment 210 receives image data from an image buffer 240. In the order described elsewhere the horizontal FDWT is performed on the rows of the image and certain portions ofthe vertical FDWT are performed on the rows ofthe image in such a way that when the data leaves a pipe segment 210 the rows are completely transformed. When directed subband data is written into subband buffers 220 and the LL subband data is transferred to pipe segment 211. The process is then repeated in pipe segment 211 resulting in subband data 221 and LL subband data is transferred to pipe segment 212, and again in pipe segment 212 resulting in subband data 222 and LL subband data is transferred to icon 230.

FIGURES 4-21 are diagrams providing a description ofthe algorithms used in the pipeline inversed wavelet transform methods of the present invention. The following description of the methods of the present invention discusses a matrix formulation of a FDWT and an IDWT. Also discussed is the multi-level discrete wavelet transform (MLDWT) and its implementation as a pipelined algorithm. These methods are also described in part in provisional U.S. application serial No. 60/250,855, filed December 1, 2000, which is commonly assigned and hereby incorporated by reference in its entirety. In addition, U.S. Patent No. 5,710,835, to Bradley, which describes processes that are related to the present invention, is also incorporated by reference in its entirety. In addition, copending provisional U.S. application entitled "Pipelined Lossless Encoder", filed December 3, 2001, attorney docket number LIZA-1 -18731, is also commonly assigned, and is also hereby incorporated by reference in its entirety. In an example of FDWT, one can use a (9, 7) DWT which describes the support ofthe low frequency and high frequency filter pairs, respectively. The support of a filter is a range of image values needed to apply the filter at a given point in the image so the low frequency filter requires values at points 1, 2, 3, 4, 5, 6, 7, 8 and 9 to compute the filtered value at point 5. The FDWT is applied to vectors (either the rows or columns) of an image and we can represent it as a matrix- vector product: y = T x, where x is a given row (say) of size N. The result ofthe transform is to create the vector y which naturally subdivides into two parts: the low frequency information about x and the high frequency information. When the FDWT is applied to an image (T is applied first to the rows ofthe image and then to the columns - whether the rows are done first or second is immaterial just so long as the transform is applied completely in one direction before the other) the resulting image is subdivided roughly into four parts: low and high frequency components in both directions. We will concentrate on describing the action of T thoroughly before discussing the two-dimensional transform.

Suppose the FDWT is applied to a vector x to yield the vector y: y = T x. We can describe the action of T by the equation:

y = P-¹DU₂L₂U₁L₁Px (1) Where P is a permutation matrix that collects together the group of x components with even indices and the group of components with odd indices, L_j are certain block unit lower triangular matrices, U_j are block unit upper triangular matrices and D is a diagonal scaling matrix. We must distinguish two cases: N (the order of x) even or odd.

Case I: N is even so N = 2 m. Then we can write out the matrices as shown in matrices 301, 302, 303 and 304 of FIGURES 4A, 4B, 4C and 4D, respectively:

Case II: N is odd so N = 2 m -1. Then we can write out the matrices as shown in matrices 311, 312, 313, and 314 of FIGURES 5 A, 5B, 5C, and 5D, respectively. Inverse DWT. It is possible to reverse each of the steps in the FDWT and, thereby, create the inverse DWT (IDWT). Reversing the steps amounts to computing the inverse of each of the matrices in equation (1) and applying them in the reverse order. We can write

x = P-l [L -1 [Ud-l [L₂]-l[U₂]-lD-lPy (2)

The related matrices are shown as matrices 321-326, in FIGURES 6A-6F, respectively.

A multi-level DWT will now be discussed. If we apply the FDWT to an image, the quarter of the transform corresponding to the low frequency information in the two directions appears to be a smoothed version ofthe original image, but of just one-quarter the size. We can store the other three-quarters ofthe transformed image and then apply a FDWT to this smoothed version. We can apply this strategy repeatedly nlev times, creating a multi-leveled hierarchy of successively smoothed and condensed versions of the original image. We call this the multi-level Discrete Wavelet Transform (MLDWT). We can visualize this process in one dimension in the following way: suppose that we are given an initial vector χ(°) of size n⁽°⁾ to which we apply the FDWT T to get y = T x. We break the result vector y into its low and high frequency parts, call the low frequency part χ(*) of size n(*⁾ and store the high frequency part. Then we repeat the process. For nlev = 2 an example is shown in FIGURE 7.

As shown in FIGURE 7, a block 330 includes a vector χ⁽°) of size n⁽°). The FDWT is applied to get y = the transform of χ⁽°⁾. The resulting vector y is broken into its low and high frequency parts, as shown as the low frequency part χ⁽*) of size nC¹) as shown in block 332. The high frequency part 324 goes to storage. The FDWT is applied to χ(!) of block 332 to again get a y = the transform of χ0). The resulting vector y is again broken into its low and high frequency parts. As shown, the low frequency part χ⁽²) of size n(²) is shown in block 336, while the high frequency part is shown in block 338, which is sent to storage.

The FDWT at each stage is accomplished by applying the sequence of matrices given in equation (1). The permutation matrix P is not explicitly applied - we just think of Px as the two groups of x, the even indexed and the odd.

Applying y = L x. In this case the even groups are unchanged so we can write the equations:

Applying y = U x. In this case the odd indexed group is unchanged and we have:

The multiplication of D x is straightforward - the even-indexed values of x are multiplied by alpha and the odd-indexed ones are multiplied by alpha inverse.

These comments on multiplying the various matrices apply also to the IDWT. Multiplying by L or U inverse involves only a change of sign of δ and ε in the above formulas.

A finite support of a Wavelet Stencil will now be discussed. The only data values required to reconstruct a pixel value (by the IDWT) lie within in a distance of 4 pixels. As shown in FIGURE 8, we can illustrate this by a "flow" diagram 340 showing the application of the factors of the IDWT in equation (2). (The "low frequency" portion of the result are those points with even indices and the "high frequency" part are the points with odd indices. They will be labeled L and H, respectively, from now on.) Note that the final value ofthe transform at location 2i - 1 requires only the values ofthe data vector at locations (2i - 5, 2i - 4, ..., 2i + 3) and the value at 2i requires (2i - 3, 2i - 2, ..., 2i + 3). [Note: the point 2i - 1 is an H point and requires 9 points while 2i is an L point and requires 7 points. This is why the wavelet transform is referred to as a (9, 7) wavelet.

From this diagram we make two observations for later use:

1. To compute the transform at a point with an odd index we need the points 4 away in each direction, and

2. To compute the transform at a point with an even index we need the points 3 away in each direction.

The concept of two spatial dimensions will now be discussed. All comments about the one-dimensional (1-D) DWT apply to the 2-D DWT which is just the 1-D algorithm applied completely in one direction and then in the other - the order is immaterial. At each stage of the forward MLDWT we transform in, say, the horizontal direction to create an "L" set and an "H" set. The vertical transform then splits each of these into 2 components yielding four pieces: LL, LH, HL, and HH of roughly the same size. The LL quarter is transformed again, while the remaining three-quarters are encoded for storage.

FIGURE 9 illustrates the process for NLEV = 3. As shown in FIGURE 9, from an image 350 the FDWT produces the set of four pieces 352, of which the pieces LH, HL, and HH go to storage, while the piece LL is again processed by the FDWT. The FDWT produces a set of pieces 354, of which pieces LH, HL, and HH go to storage, while the piece LL is again processed by the FDWT. The FDWT produces the set of pieces 356, which are then all sent to storage. The piece LL is designated as icon.

To show more detail let us follow the first two rows ofthe original image through the first step of the process. Assume that the number of columns in the image is even, say 2m. We represent the original data as cells in the array 360 of FIGURE 10. After the 2D DWT we can label the cells LL, LH, HL, and HH as shown in the array 370 of FIGURE 11. These data will be dispersed into four separate arrays as follows (the ordered pairs in the cells tell what position the data came from). The four separate arrays are shown in FIGURES 12A-12D, as arrays 381-384, respectively.

Processing each pair of rows from the original image yields one row of about half the length in each of the four arrays LL, LH, HL, and HH. So all information about an even-odd pair of rows in the image goes into the same row number in the four smaller arrays. For later use we note that each entry in LL and HL came from even-indexed columns of the original image and each entry in LH and HH came from odd-indexed columns. Note also that the number of columns in the LH and HH arrays will be reduced by one if the width ofthe original image is odd. Also the number of rows in HL and HH will be reduced by one when there are an odd number of rows in the original image.

We have illustrated the first step in the MLDWT. This same process occurs at each level.

The pipeline concept will now be discussed. The idea behind the pipelined inverse MLDWT is that as soon as a few rows of the image on a scale s have been computed, they can be combined with the appropriate high frequency information on scale s-1 to obtain some rows of the image on that scale. When enough of these rows have been completed we can start the IDWT on the next lower scale and so forth. (On a parallel computer once the process was started all steps on all levels could be computed simultaneously in parallel.) We can illustrate this process (again for nlev = 3) as illustrated in FIGURE 13.

As illustrated in FIGURE 13, a pipe segment 2 at a block 392 receives data from an icon at block 390, and the pieces LH, HL and HH at a block 391. A pipe segment 1 at a block 394 receives data from the pipe segment 2 of block 392 and the pieces LH, HL and HH of a block 393. A pipe segment 0 at a block 396 receives data from a pipe segment 1 at block 394, and the pieces LH, HL, and HH at a block 395. Data from the pipe segment 0 at block 396 goes to the image. A pipe segment consists of two arrays: muster and wave each consisting of four rows of length approximately the width of the image on that scale. (Actually there are pairs of muster and wave arrays for each spectral band, too.) The muster array is where all the four pieces of data are reassembled and some of the vertical transform is done. The wave array is where the remaining parts of the vertical transform and all of the horizontal transform are performed. The rows of the wave array are passed to the pipe segment at the next level for processing.

A pipe segment can be thought of as a narrow window that slides down over a scene in the image on a certain scale. The pipe segment must know all the information regarding the width ofthe scene on this level, what rows and columns to extract from the stored three pieces (LH, HL, HH) and how to recombine them properly.

Assume that a pipe segment is processing a scene that extends from x = xr to x = xl and assume that the endpoints are at least four pixels from the boundary (for simplicity - we consider boundaries later). To accurately calculate the IDWT we must add points to each side ofthe image to accommodate the support ofthe DWT. Let

xl'=xl-(3+xl%2), xr'=xr-(3+xr%2).

The term 3 + xl%2 calculates exactly the necessary support to calculate the IDWT at xl. Note that these endpoints xl' and xr' are always odd numbers. Recall that we associate odd-indiced points with H and even ones with L. For exposition assume xl is even and xr is odd. Schematically, we can represent one row of the pipe segment as row 400 of FIGURE 14.

We now describe schematically the computation of the horizontal component of the IDWT. From equation (2) we see that we must perform five matrix multiplications. The first one by D inverse requires that we multiply all L cells by alpha inverse and H cells by alpha - we will not show this. Let us simplify the notation here by representing the remaining four products as H_Q, H^, H₂, and H3, labeled from right to left. (So H₀ = U₂ inverse.) For each product half of the cells are unchanged, the other half are just linear combinations of three adjacent points. We will represent these by the entry Hi at the centerpoint ofthe three adjacent points. We can now write out the entire product 410 as shown in FIGURE 15. This represents the computation ofthe horizontal component of the IDWT on one row of the image. This computation for all rows is done in the wave array. The vertical component of the IDWT is more complicated since we can only do some ofthe parts. Consider the diagram above only now mentally change the H^'s to Vj's and think of the cells labeled X-..3, ..., X_r as being a group of four rows that are currently in the muster array and that have had the Vj operators to the left of the leftmost dashed diagonal applied already. The four rows will be moved to the muster array and the next four rows (here labeled X_r+1, ..., X_r') assembled in the muster array. We can now apply the Vj operators enclosed by both dashed diagonals. This has the effect of completing the vertical transform on those rows currently in the wave array and performing as many of the pieces ofthe vertical transform as there is data for.

The concept of constructing the pipe will now be discussed. We define a Pipe to be nlev pipe segments - one for each scale level ofthe image. For the IMLDWT we must be able to reconstruct any arbitrary scene (subset) of the image on any scale. (We will always discuss only scale 0 - the original image- but the ideas are the same for any positive scale.)

Suppose the requested scene contains columns xr to xl. We must extend the interval [xr = xr(0), xl = xl(0)] to [xr', xl'] (assume this is still within the image). On the next higher scale the interval [xr', xl'] projects up to the interval

xr ®,xflϊ i(xr'+l), (x/'-l)

These formulas are a result ofthe fact that only the points in the image at scale 0 with even row and column indices get projected to the image at scale 1. Since xr' and xl' are odd, xr' + 1 and xl' - 1 are even and they project to the endpoints of the interval on scale 1. To correctly compute the values ofthe image at these endpoints we must extend [xr(l), xlC¹)] to [xr(l)^!, xl(l)']. We continue in this fashion defining the interval needed at each scale up to the icon (scale = nlev).

Let us now look in more detail at pipe segments 0 and 1. At this point we need to consider a 2 x 2 block of the image starting with an even row and an even column as a "fundamental" unit. It corresponds to an L,H pair in rows and columns. After the FDWT has been applied this unit contains one entry from each of LL, LH, HL, and HH. We will now switch notation (to come into agreement with the code) so that

Here "wv" stands for "wave array" and sbl, sb2, and sb3 stand for the three terminal nodes ofthe subband tree at some level (numbered using the Z-pattern ordering). We now illustrate the contents 420 of pipe segment 0 for an interior scene as shown in FIGURE 16.

For completeness we record the layout of a pipe segment in two other cases: the scene abuts the left boundary and the scene lies within 4 pixels ofthe left boundary (to be really complete we should record the variation when the scene abuts or is near the right boundary, but there are no new ideas so we won't). When xr = 0 the scene we are decoding starts at the boundary of the image. To provide adequate data for the support of the DWT at or near the boundary we always reflect the image using grid symmetry which is defined formally by the equation

pixel at x=-j = pixel at x=+j

This equation provides a definition for pixels just outside the boundary. At this point let us define another type of symmetry, namely cell symmetry defined by the equation

pixel at x=-j = pixel at x=j-l

This symmetry will be used later. The contents 430 ofthe pipe segment when xr = 0 is shown in FIGURE 17.

The cells to the left of column 0 are filled with the properly reflected values to the right of column 0.

When xr is not 0 but is within 4 pixels ofthe boundary a slight complication arises in that we must impose the grid symmetry at the column corresponding to the left boundary ofthe image. To simplify this procedure we record the value of xr as the start of the real row, but then reset xr to 0 and treat the scene as though it abutted the left boundary. We can illustrate this when, say, xr = 2, for which the contents 440 are shown in FIGURE 18.

When all possibilities have been considered it is necessary to store seven integers that describe where the data that fills the muster array comes from. The quantities describe the location in the previous segment where the wv entries come from, where they are going, the number of wv elements to transfer, the same three values for sbl, and the source of sb2.

The general flow of rows through the pipe will now be discussed. If the scene at some scale starts at the upper boundary (row 0) we start the transform by filling the muster array with rows 0 through 3. Then we reflect rows (using grid symmetry about row 0) up into the wave array. (Since 0 is even we only need three reflected rows to provide the support for the vertical transform.) We then perform the required vertical transforms, push the muster array to the wave array, perform the horizontal transforms, and then the wave array contains the first four transformed rows of the image at the current scale.

If the scene starts at least four rows away from the upper boundary we just start with the required previous rows to cover the support of the vertical transform and begin the algorithm.

At the bottom of the image at a certain scale the image must be reflected using grid symmetry about the last row. Since all rows are treated in groups of four we must distinguish four cases and determine how to complete the components of the vertical transform at the end of the image. Let the number of rows in the image be N. We distinguish the cases Nmod4. (In what follows we re-label the rows so that the last row is numbered 0, the next-to-last is 1, etc., to make the exposition easier).

Case 0 = Nmod4. Nothing special has to be done here because the last fill of the muster array uses the last ofthe data and fills the muster array completely.

Case 1 = Nmod4. Here there is one remaining row in the previous segment wave array (PSWA) and in sbl. No rows remain in sb2 or sb3. The image at the last fill ofthe muster array looks like the left side ofthe diagram 450 of FIGURE 19.

We can effect the reflection shown above by copying rows r2 and rl from wave to muster. Noting that VO has already been applied to r2 we see we don't actually need r3. Then we only need to compute the components of the vertical transform that are underlined.

Case 2 = Nmod4. In this case there is one remaining row in PSWA, sbl, sb2, and sb3 which when merged fills two rows of muster. So we have the diagram 460 of FIGURE 20.

So in this case we need to copy row 0 of muster to row 2 and row 3 of wave to row 3. (Neither of these rows has had any vertical components applied to it.) We apply all underlined Vi's using a special symmetric formula for V3 on rO.

Case 3 = Nmod4. In this final case there are two remaining rows in PSW and sbl, but only one row in sb2 and sb3. So we have the diagram 470 of FIGURE 21.

We need copy only row 1 of muster to row 3, do all the vertical components that are underlined and use a special symmetric formula for V2 at rO.

Treatment of Case scale < 0. If a negative value of scale is specified, we need only transfer data from PSWA as needed and fill all values from sbl, sb2, and sb3 with zero. FIGURES 22-29 illustrate pipeline stages for various wavelets, including a 5-3 wavelet and alternate versions of a 9-7 wavelet. FIGURE 22 shows a representation of pipeline stages 480 for a 5-3 wavelet. FIGURE 23 shows a representation of pipeline stages 490 for a 9-7 wavelet, utilizing standard symmetric factors. FIGURE 24 shows a representation of pipeline stages 500 for a 9-7 wavelet, utilizing alternative factors. FIGURE 25 shows a representation of an alternate version 510, utilizing standard factors. FIGURE 26 shows a representation of an alternate version 520, utilizing alternative factors. FIGURE 27 shows a representation of pipeline stages 530 for alternative factors, when space is at a premium and the storage has been reduced by 1 (from either 6 to 5 or from 8 to 7). FIGURE 28 shows a representation of pipeline stages 540, similar to that shown in FIGURE 27, except where the space has been reduced one additional unit, at the cost of doubling the computation time.

FIGURE 29 shows a representation of pipeline stages 600, which illustrates the extra information that is needed to handle the termination boundary conditions. The circled rows are the ones which should be saved in order to handle the new incoming rows (or a pipeline termination). In this embodiment, seven saved rows and two new rows are needed. In addition, a tenth row will be needed for pipeline termination when the total number of rows in the input subband is odd.

While the preferred embodiment of the invention has been illustrated and described, it will be appreciated that various changes can be made therein without departing from the spirit and scope ofthe invention.

Claims

The embodiments of the invention in which an exclusive property or privilege is claimed are defined as follows:

1. A method for nonrecursively computing the multilevel forward discrete wavelet transform (FDWT) using an amount of primary memory that is proportional to only one dimension of the image and proportional to the support of the FDWT, comprising:

(a) allocating zero or more buffers in main memory to store a strip of image data, a strip of the lower resolution image on each of the n levels, and subband buffers to store certain results ofthe FDWT,

(b) initializing said buffer,

(c) performing the following steps one or more times: (i) requesting a strip ofthe original image,

(ii) processing the lines of each strip thereby producing partial transformation results for at least some levels, the transformation results including, at each level, at least one of intermediate or subband data,

(iii) during production of transformation results, storing recently produced intermediate results in the buffers, while discarding earlier produced intermediate results from the buffers,

(iv) forwarding at least some data in at least some subband buffers and resetting those buffers,

(d) flushing said buffers, after processing the last rows.

2. The method of Claim 1, wherein each ofthe buffers is associated with one of the levels of the multilevel FDWT and is allocated main memory space in proportion to the width ofthe image on that level and the support ofthe FDWT.

3. A method of Claim 1, wherein certain conditions are applied to extend the image beyond its original limits.

4. A method of Claim 3, wherein certain conditions on the vertical boundaries of the original image and all its lower resolution images are applied to create an encoded image with no apparent vertical boundaries such that the image appears to have been affixed to the inside or outside of a vertical cylinder.

5. A method of Claim 1, wherein the intermediate results are moved to the appropriate locations when it is detected that there are no more rows ofthe image on that level.

6. A method of Claim 1, wherein data comprising entries of subbands are stored in subband buffers until they are processed or stored.

7. A method of Claim 1, wherein intermediate results at one level are identified as pixels representing the image on the next lower level and stored in the buffers for that level.

8. A method of Claim 1, wherein intermediate results at each level are identified as entries of subbands related to that level and are stored in the subband buffers.

9. A method of Claim 1, wherein intermediate results at the lowest level are identified as pixels of the lowest resolution image and are stored in the appropriate subband buffer.

10. A method of Claim 1, wherein certain conditions are applied to extend the original image and all its lower resolution images beyond their original extent and to incorporate said extended data into the FDWT.

11. A method of Claim 1, wherein a factored form of the wavelet transform, such as lifting steps are used to minimize the amount of primary memory required for each level, to reduce arithmetic operations, to make it reversible without error.

12. A method of Claim 1, wherein a nonrecursive algorithm is used to control the flow of data through the series of intermediate buffers.

13. A method of Claim 12, wherein a nonrecursive algorithm is used to control the flow of data through the series of intermediate buffers as though rows of the initial image were pushing data through the buffers.

14. A method of Claim 12, wherein a nonrecursive algorithm is used to control the flow of data through the series of intermediate buffers as though the production of rows ofthe lowest resolution image were pulling data through the buffers.

15. A method for computing the multilevel inverse discrete wavelet transform (IDWT) to reconstruct an arbitrary scene of any resolution of a wavelet-encoded image using an amount of main memory that is proportional to only one dimension of that scene and proportional to the support ofthe IDWT, comprising:

(a) determining from the location of the scene and the support of the IDWT a sufficient subset of each subband of wavelet coefficients to reconstruct the given scene;

(b) allocating intermediate buffers for the levels in main memory based on the subject and the subband buffers for the wavelet coefficients stored in the wavelet-encoded image based on the subset;

(c) initializing said buffers,

(d) performing the following steps one or more times,

(i) requesting rows of the subband buffers containing wavelet coefficients;

(ii) creating pixel values as a result of partially inverse transforming wavelet coefficients at some or all levels:

(iii) moving rows of image pixels from a given level to the next higher level,

(e) flushing said buffers.

16. A method of Claim 15, wherein each buffer is associated with one of the levels ofthe multilevel IDWT and is allocated primary memory space in proportion to the width ofthe scene of image on that level and the support ofthe IDWT.

17. A method of Claim 15, wherein boundary conditions are applied to extend the image beyond its original limits.

18. A method of Claim 17, wherein certain conditions on the vertical boundaries of the original image and all its lower resolution images are applied to create an decoded scene with no apparent vertical boundaries, such that the scene may appear to have been affixed to the inside or outside of a vertical cylinder.

19. A method of Claim 15, wherein the intermediate results are moved to the appropriate locations when it is detected that there are no more rows of wavelet coefficients to inverse transform.

20. A method of Claim 15, wherein intermediate results at one level are identified as pixels representing the reconstructed image on that level and kept in the buffers for use in the processing ofthe image at the next higher level.

21. A method of Claim 15, wherein intermediate results at the highest level are identified as pixels representing the reconstructed requested scene and are stored in an image buffer.

22. A method of Claim 15, wherein certain conditions are applied to extend the original image and all its lower resolution images beyond their original extent and to incorporate said extended data into the IDWT.

23. A method of Claim 15, wherein a factored form of the wavelet transform, called lifting steps is used so as to minimize the amount of primary memory required for each level, to reduce arithmetic operations, and to make it reversible without error.

24. A method of Claim 15, wherein a nonrecursive algorithm is used to control the flow of data through the series of intermediate buffers.

25. A method of Claim 24, wherein a nonrecursive algorithm is used to control the flow of data through the series of intermediate buffers as though rows of the requested scene were pulling data through the buffers.

26. A method of Claim 24, wherein a nonrecursive algorithm is used to control the flow of data through the series of intermediate buffers as though the consumption of rows of the lowest resolution image were pushing data through the buffers.

27. A method of Claim 15, in which the relative location of parts of the subbands ofthe pipelined wavelet encoded image may be determined.

28. A method of Claim 15, in which we determine a minimal subset of each subband of wavelet coefficients necessary to reconstruct the given scene.

29. A method wherein certain conditions on the vertical boundaries of the original image and all its lower resolution images are applied to create an encoded image with no apparent vertical boundaries such that the image may appear to have been affixed to the inside or outside of a vertical cylinder.