US20040008886A1 - Using graph cuts for editing photographs - Google Patents

Using graph cuts for editing photographs Download PDF

Info

Publication number
US20040008886A1
US20040008886A1 US10/413,974 US41397403A US2004008886A1 US 20040008886 A1 US20040008886 A1 US 20040008886A1 US 41397403 A US41397403 A US 41397403A US 2004008886 A1 US2004008886 A1 US 2004008886A1
Authority
US
United States
Prior art keywords
image
graph
background
segmentation
cut
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Abandoned
Application number
US10/413,974
Inventor
Yuri Boykov
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Siemens Corporate Research Inc
Original Assignee
Siemens Corporate Research Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Siemens Corporate Research Inc filed Critical Siemens Corporate Research Inc
Priority to US10/413,974 priority Critical patent/US20040008886A1/en
Assigned to SIEMENS CORPORATE RESEARCH, INC. reassignment SIEMENS CORPORATE RESEARCH, INC. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: BOYKOV, YURI
Publication of US20040008886A1 publication Critical patent/US20040008886A1/en
Priority to US11/772,915 priority patent/US7536050B2/en
Abandoned legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00Image analysis
    • G06T7/10Segmentation; Edge detection
    • G06T7/11Region-based segmentation
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00Pattern recognition
    • G06F18/20Analysing
    • G06F18/23Clustering techniques
    • G06F18/232Non-hierarchical techniques
    • G06F18/2323Non-hierarchical techniques based on graph theory, e.g. minimum spanning trees [MST] or graph cuts
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00Image analysis
    • G06T7/10Segmentation; Edge detection
    • G06T7/162Segmentation; Edge detection involving graph-based methods
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00Arrangements for image or video recognition or understanding
    • G06V10/20Image preprocessing
    • G06V10/26Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
    • G06V10/267Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion by performing operations on regions, e.g. growing, shrinking or watersheds
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00Arrangements for image or video recognition or understanding
    • G06V10/70Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/762Arrangements for image or video recognition or understanding using pattern recognition or machine learning using clustering, e.g. of similar faces in social networks
    • G06V10/7635Arrangements for image or video recognition or understanding using pattern recognition or machine learning using clustering, e.g. of similar faces in social networks based on graphs, e.g. graph cuts or spectral clustering
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/20Special algorithmic details
    • G06T2207/20092Interactive image processing based on input by user
    • G06T2207/20101Interactive definition of point of interest, landmark or seed

Definitions

  • the present invention relates to image processing systems, and, more particularly to a technique for interactive segmentation of images using graph cuts.
  • an image editing system comprises an input device for inputting an image, a graphical user interface for selecting background and object seeds for the image, and an image processor for editing the image.
  • the image processor has various editing routines, including a segmentation routine that builds a graph associated with the image and uses a graph cut algorithm to cut the graph into segments.
  • the user marks certain pixels as “object” or “background” to provide hard constraints for segmentation. Additional soft constraints incorporate both boundary and regional information.
  • Graph cuts are used to find the globally optimal segementation of the image. The obtained solution gives the best balance of boundary and region properties satisfying the constraints.
  • the nodes of the graph represent portions of the image, such as pixels (voxels).
  • the edges of the graph represent neighborhood relationships among the nodes.
  • the graph also includes a background terminal node and an object terminal node. Each of the non-terminal nodes of the graph is connected to both the background terminal node and the object terminal node.
  • the edges of the graph are each assigned a cost value.
  • the cost values are preferably determined according to a cost function that is defined in terms of boundary and region properties of segments.
  • the cost function uses the seed positions to assign cost values.
  • a graph cut is performed using a suitable graph cut algorithm. Inexpensive edges are attractive choices for a minimum graph cut.
  • the image editing system can further be configured with an output device (e.g, computer monitor, printer) for outputting the segmented image.
  • an output device e.g, computer monitor, printer
  • the graphical user interface allows additional background and object seeds to be input. By reviewing the results of initial segmentation, the user will see which areas of the image were incorrectly segmented. Then the user can place additional seeds to correct the problem. This interactive process of adding seeds may be continued until the user obtains satisfactory results.
  • FIG. 1 is a block diagram of a typical photo-editing system
  • FIG. 2 is a block diagram of various routines associated with an embodiment of the present invention.
  • FIG. 3 illustrates segmentation of a photograph using graph cuts
  • FIG. 4 is a workflow diagram showing the use of graph cuts to perform segmentation of a 2D image
  • FIG. 5( a ) illustrates segmentation of a photograph using graph cuts with the regional term included
  • FIG. 5( b ) illustrates segmentation of a photograph using graph cuts without the regional term
  • FIG. 6 illustrates segmentation of a video sequence using graph cuts.
  • a typical photo-editing system 100 includes a scanner 110 (or other digital image source, such as a digital camera), a personal computer 120 (or other digital image processor) with a keyboard 124 and mouse 126 (or other input devices), a printer 130 (or other output devices). Except for certain aspects of the software, to be discussed in detail hereinafter, the photo-editing system may comprise commercially available hardware and software components, assembled in a manner that will be readily apparent to one skilled in the art.
  • the software 200 that controls the personal computer 120 includes an operating system (such as MICROSOFT WINDOWSTM) having a graphical user interface 220 whereby a user may use a pointing device, such as the mouse 126 , via a pointer driver 226 to select and operate various controls (e.g., buttons, menus) appearing on the screen of monitor 122 under control of the display driver 222 thereby generating control signal inputs 125 to an application program such as an image processor 240 .
  • the image processor 240 can include various routines for performing various editing and other imaging functions.
  • the image processor 240 can interact with a cropping routine 242 for trimming an image, a touch-up routine 244 for making small changes and corrections to an image, and a segmentation routine 246 for separating portions of an image.
  • the segmentation routine 246 is described in greater detail hereinafter with respect to FIGS. 3 - 6 .
  • the image processor 240 also includes an image memory 245 for storing in known fashion the original image.
  • the image processor 240 receives image data from the scanner 110 via a scanner driver 228 and outputs image data not only to the monitor 122 via graphics user interface 220 , but also via printer driver 224 to the printer 130 .
  • FIGS. 3 ( a ) and 3 ( b ) the process of using the segmentation routine 246 is illustrated.
  • FIG. 3( a ) shows an original photograph that was inputted into the image processor 240 .
  • FIG. 3( b ) shows a bell and group of people segmented from this photograph.
  • a user can start by entering a few “object” and “background” seeds loosely positioned inside and, correspondingly, outside the object(s) of interest (i.e., the bell and group of people).
  • the user will see which areas are segmented incorrectly. Then the user can put additional seeds into the troubled places. This interactive process of adding seeds may be continued until the user obtains satisfactory results.
  • the present invention provides a general-purpose interactive segmentation technique that divides an image into two segments: “object” and “background”.
  • a user imposes certain hard constraints for segmentation by indicating certain pixels (seeds) that have to be part of the object and certain pixels that have to be part of the background. Intuitively, these hard constraints provide clues on what the user intends to segment.
  • the rest of the image is segmented automatically by computing a global optimum among all segmentations satisfying the hard constraints.
  • a cost function is defined in terms of boundary and region properties of the segments. These properties can be viewed as-soft constraints for segmentation.
  • a globally optimal segmentation can be very efficiently recomputed when the user adds or removes any hard constraints (seeds). This allows the user to get any desired segmentation results quickly via very intuitive interactions. This method applies to all N-D images (volumes).
  • One of the main advantages of the interactive segmentation method employed herein is that it provides a globally optimal solution for the N-dimensional segmentation when the cost function is clearly defined.
  • Some earlier techniques can do that only in 2D applications when a segmentation boundary is a 1D curve.
  • Other techniques either don't have a clear cost function at all (e.g., region growing, split and merger) or compute only an approximate solution (e.g., a local minimum) that can be arbitrarily far from the global optimum (region competition, level set methods, normalized cuts).
  • Global properties of such segmentation may be difficult to analyze or predict. Imperfections in the result might come from deficiencies at the minimization stage.
  • imperfections of a globally optimal solution are directly related to the definition of the cost function.
  • the segmentation can be controlled more reliably.
  • P can contain pixels (or voxels) in a 2D (or 3D) grid and N can contain all unordered pairs of neighboring pixels (voxels) under a standard 8- (or 26-) neighborhood system.
  • A (A 1 , . . . ,A p , . . . ,A
  • Each A p can be either “obj” or “bkg” (abbreviations of “object” and “background”).
  • Vector A defines a segmentation. Then, the soft constraints that we impose on boundary and region properties of A are described by the cost function E(A):
  • R ⁇ ( A ) ⁇ p ⁇ P ⁇ R p ⁇ ( A p ) ( 2 )
  • the coefficient ⁇ 0 in (1) specifies the relative importance of the region properties term R(A) versus the boundary properties term B(A).
  • the regional term R(A) assumes that the individual penalties for assigning pixel p to “object” and “background”, correspondingly R p ( ⁇ ) may reflect on how the intensity of pixel p fits into a known intensity model (e.g., histogram) of the object and background.
  • B(A) comprises the “boundary” properties of segmentation A.
  • Coefficient B ⁇ p,q ⁇ ⁇ 0 should be interpreted as a penalty for discontinuity between p and q. Normally, B ⁇ q,p ⁇ is large when pixels p and q are similar (e.g., in their intensity) and B ⁇ q,p ⁇ is close to zero when the two are very different.
  • the penalty B ⁇ q,p ⁇ can also decrease as a function of distance between p and q.
  • Costs B ⁇ q,p ⁇ may be based on local intensity gradient, Laplacian zero-crossing, gradient direction, and other criteria.
  • a segmentation method decides how to segment unmarked pixels.
  • Some conventional techniques use the same type of hard constraints as the present invention but they do not employ a clear cost function and segment unmarked pixels based on variations of “region growing”. Since the properties of segmentation boundary are not optimized, the results are prone to “leaking” where the boundary between objects is blurry.
  • the present invention combines the hard constraints as above with energy (1) that incorporates region and boundary properties of segments.
  • An example of a graph that we use in this paper is shown in FIG. 4( a ).
  • Each edge e ⁇ E in the graph is assigned a nonnegative weight (cost) ⁇ e .
  • cost cost
  • terminals There are also two special nodes called terminals.
  • Graph cut formalism is well suited for segmentation of images. In fact, it is completely appropriate for N-dimensional volumes.
  • the nodes of the graph can represent pixels (or voxels) and the edges can represent any neighborhood relationship between the pixels.
  • a cut partitions the node in the graph. As illustrated in FIGS. 4 ( c ) and ( d ), this partitioning corresponds to a segmentation of the underlying image or volume.
  • a minimum cost cut generates a segmentation that is optimal in terms of properties that are built into the edge weights.
  • the next step is to compute the globally optimal minimum cut.
  • This cut gives a segmentation 430 of the original image 410 .
  • the image is divided into exactly one “object” and one “background” region.
  • our segmentation method generates binary segmentation with arbitrary topological properties.
  • Other examples will illustrate that object and background segments may comprise several isolated connected blobs in the image.
  • V P ⁇ S,T ⁇ .
  • the set of edges E consists of two types of undirected edges: n-links (neighborhood links) and t-links (terminal links).
  • Each pixel p has two t-links ⁇ p,S ⁇ and ⁇ p,T ⁇ connecting it to each terminal.
  • K 1 + max ⁇ ⁇ q : ⁇ p , q ⁇ ⁇ N ⁇ B ⁇ p , q ⁇ .
  • the graph G is now completely defined. We draw the segmentation boundary between the object and the background by finding the minimum cost cut on the graph G.
  • the minimum cost cut ⁇ on G can be computed exactly in polynomial time via algorithms for two terminal graph cuts assuming that the edge weights specified in the table above are non-negative.
  • R p (“ bkg ”) ⁇ lnPr ( I p
  • This function penalizes a lot for discontinuities between pixels of similar intensities when
  • FIGS. 3 ( a )-( b ) we illustrated the segmentation a bell with a group of people from a photograph.
  • the user can start with a few “object” and “background” seeds loosely positioned inside and, correspondingly, outside the object(s) of interest.
  • By reviewing the results of initial segmentation the user will see what areas are segmented incorrectly. Then (s)he can put additional seeds into the troubled places and efficiently recomputed the optimal segmentation. This process of adding seeds gives more clues to the algorithm and may be continued until the user likes the results.
  • the globally minimum two-terminal cut can be computed on any graph.
  • our technique is valid for segmentation of N-D data.
  • FIG. 6 we segmented moving cars in a video sequence.
  • the sequence of 21 video frames (255 ⁇ 189) was treated as a single 3D volume.
  • the necessary seeds were entered in a simple 3D interface where we could browse through individual 2D slices (frames) of the volume.
  • Initial seeds can be entered in just a few representative frames.
  • the boundary, region, and seed information is automatically propagated between the slices since we compute a globally optimum solution directly on the 3D data set.
  • the whole sequence can be segmented based on seeds placed in just a few frames. For example, entering correcting seeds in one frame can fix imperfections in many adjacent frames.
  • the results in FIG. 6 are obtained by placing seeds in 3 out of 21 frames. Each car was segmented in an independent experiment. We do not show seeds to avoid confusion.
  • the process of placing seeds can be automated for certain applications.
  • the seeds should be positioned with a low probability of “false alarm” while the probability of “right detect” is not required to be high.
  • Even very simple recognition techniques based on filters might be good enough if the corresponding threshold is set high. Such filters would have difficulties near the boundaries of objects but they can confidently place many seeds anywhere else. These seeds give hard constraints. Based on additional soft constraints (1) the minimum cut can complete the segmentation where the recognition method failed.

Abstract

An image editing system comprises an input device for inputting an image, a graphical user interface for selecting background and object seeds for the image, and an image processor for editing the image. The image processor has various editing routines, including a segmentation routine that builds a graph associated with the image and uses a graph cut algorithm to cut the graph into segments. The user marks certain pixels as “object” or “background” to provide hard constraints for segmentation. Additional soft constraints incorporate both boundary and regional information. Graph cuts are used to find the globally optimal segementation of the image. The obtained solution gives the best balance of boundary and region properties satisfying the constraints.

Description

    CROSS REFERENCE TO RELATED APPLICATIONS
  • This application claims the benefit of U.S. Provisional Application Serial No. 60/393,163, filed on Jul. 2, 2002, which is incorporated by reference herein in its entirety.[0001]
  • FIELD OF THE INVENTION
  • The present invention relates to image processing systems, and, more particularly to a technique for interactive segmentation of images using graph cuts. [0002]
  • BACKGROUND OF THE INVENTION
  • Various types of software products for editing digital images exist in the marketplace. Although many of these products perform basic editing tasks such as cutting, cropping, and touching-up reasonably well, it can be difficult to segment images using these products. Segmentation involves isolating a desired portion of an image and separating that portion from the rest of the image. [0003]
  • Conventionally, some photo-editors provide segmentation tools based on user-based seeds that must be placed on desired segmentation boundaries. However, this is often a tedious task since the seeds must be very carefully placed. For instance, it is usually necessary to place numerous seeds directly on the boundary. Although some conventional photo-editors do provide for more relaxed seed placement, these segmentation tools tend to produce inaccurate results. [0004]
  • Accordingly, it would be desirable and highly advantageous to provide improved segmentation techniques for photo-editing that overcome the problems of conventional approaches. [0005]
  • SUMMARY OF THE INVENTION
  • In various embodiments of the present invention, an image editing system comprises an input device for inputting an image, a graphical user interface for selecting background and object seeds for the image, and an image processor for editing the image. The image processor has various editing routines, including a segmentation routine that builds a graph associated with the image and uses a graph cut algorithm to cut the graph into segments. The user marks certain pixels as “object” or “background” to provide hard constraints for segmentation. Additional soft constraints incorporate both boundary and regional information. Graph cuts are used to find the globally optimal segementation of the image. The obtained solution gives the best balance of boundary and region properties satisfying the constraints. [0006]
  • According to various embodiments of the present invention, the nodes of the graph represent portions of the image, such as pixels (voxels). The edges of the graph represent neighborhood relationships among the nodes. The graph also includes a background terminal node and an object terminal node. Each of the non-terminal nodes of the graph is connected to both the background terminal node and the object terminal node. [0007]
  • The edges of the graph are each assigned a cost value. The cost values are preferably determined according to a cost function that is defined in terms of boundary and region properties of segments. In addition, the cost function uses the seed positions to assign cost values. A graph cut is performed using a suitable graph cut algorithm. Inexpensive edges are attractive choices for a minimum graph cut. [0008]
  • The image editing system can further be configured with an output device (e.g, computer monitor, printer) for outputting the segmented image. The graphical user interface allows additional background and object seeds to be input. By reviewing the results of initial segmentation, the user will see which areas of the image were incorrectly segmented. Then the user can place additional seeds to correct the problem. This interactive process of adding seeds may be continued until the user obtains satisfactory results. [0009]
  • These and other aspects, features and advantages of the present invention will become apparent from the following detailed description of preferred embodiments, which is to be read in connection with the accompanying drawings.[0010]
  • BRIEF DESCRIPTION OF THE DRAWINGS
  • FIG. 1 is a block diagram of a typical photo-editing system; [0011]
  • FIG. 2 is a block diagram of various routines associated with an embodiment of the present invention; [0012]
  • FIG. 3 illustrates segmentation of a photograph using graph cuts; [0013]
  • FIG. 4 is a workflow diagram showing the use of graph cuts to perform segmentation of a 2D image; [0014]
  • FIG. 5([0015] a) illustrates segmentation of a photograph using graph cuts with the regional term included;
  • FIG. 5([0016] b) illustrates segmentation of a photograph using graph cuts without the regional term; and
  • FIG. 6 illustrates segmentation of a video sequence using graph cuts.[0017]
  • DESCRIPTION OF PREFERRED EMBODIMENTS
  • Referring to FIG. 1, a typical photo-[0018] editing system 100 includes a scanner 110 (or other digital image source, such as a digital camera), a personal computer 120 (or other digital image processor) with a keyboard 124 and mouse 126 (or other input devices), a printer 130 (or other output devices). Except for certain aspects of the software, to be discussed in detail hereinafter, the photo-editing system may comprise commercially available hardware and software components, assembled in a manner that will be readily apparent to one skilled in the art.
  • In an exemplary embodiment of the present invention, as shown in simplified block diagram form in FIG. 2, the [0019] software 200 that controls the personal computer 120 includes an operating system (such as MICROSOFT WINDOWS™) having a graphical user interface 220 whereby a user may use a pointing device, such as the mouse 126, via a pointer driver 226 to select and operate various controls (e.g., buttons, menus) appearing on the screen of monitor 122 under control of the display driver 222 thereby generating control signal inputs 125 to an application program such as an image processor 240. The image processor 240 can include various routines for performing various editing and other imaging functions. For instance, the image processor 240 can interact with a cropping routine 242 for trimming an image, a touch-up routine 244 for making small changes and corrections to an image, and a segmentation routine 246 for separating portions of an image. The segmentation routine 246 is described in greater detail hereinafter with respect to FIGS. 3-6. The image processor 240 also includes an image memory 245 for storing in known fashion the original image. The image processor 240 receives image data from the scanner 110 via a scanner driver 228 and outputs image data not only to the monitor 122 via graphics user interface 220, but also via printer driver 224 to the printer 130.
  • Referring to FIGS. [0020] 3(a) and 3(b) the process of using the segmentation routine 246 is illustrated. FIG. 3(a) shows an original photograph that was inputted into the image processor 240. FIG. 3(b) shows a bell and group of people segmented from this photograph. A user can start by entering a few “object” and “background” seeds loosely positioned inside and, correspondingly, outside the object(s) of interest (i.e., the bell and group of people). By reviewing the results of initial segmentation the user will see which areas are segmented incorrectly. Then the user can put additional seeds into the troubled places. This interactive process of adding seeds may be continued until the user obtains satisfactory results.
  • The present invention provides a general-purpose interactive segmentation technique that divides an image into two segments: “object” and “background”. A user imposes certain hard constraints for segmentation by indicating certain pixels (seeds) that have to be part of the object and certain pixels that have to be part of the background. Intuitively, these hard constraints provide clues on what the user intends to segment. The rest of the image is segmented automatically by computing a global optimum among all segmentations satisfying the hard constraints. A cost function is defined in terms of boundary and region properties of the segments. These properties can be viewed as-soft constraints for segmentation. A globally optimal segmentation can be very efficiently recomputed when the user adds or removes any hard constraints (seeds). This allows the user to get any desired segmentation results quickly via very intuitive interactions. This method applies to all N-D images (volumes). [0021]
  • One of the main advantages of the interactive segmentation method employed herein is that it provides a globally optimal solution for the N-dimensional segmentation when the cost function is clearly defined. Some earlier techniques can do that only in 2D applications when a segmentation boundary is a 1D curve. Other techniques either don't have a clear cost function at all (e.g., region growing, split and merger) or compute only an approximate solution (e.g., a local minimum) that can be arbitrarily far from the global optimum (region competition, level set methods, normalized cuts). Global properties of such segmentation may be difficult to analyze or predict. Imperfections in the result might come from deficiencies at the minimization stage. In contrast, imperfections of a globally optimal solution are directly related to the definition of the cost function. Thus, the segmentation can be controlled more reliably. [0022]
  • It is also important that the cost function used as a soft constraint for segmentation is general enough to include both region and boundary properties of segments. Consider an arbitrary set of data elements P and some neighborhood system represented by a set N of all unordered pairs {p,q} of neighboring elements in P. For example, P can contain pixels (or voxels) in a 2D (or 3D) grid and N can contain all unordered pairs of neighboring pixels (voxels) under a standard 8- (or 26-) neighborhood system. Let A=(A[0023] 1, . . . ,Ap, . . . ,A|p|) be a binary vector whose components Ap specify assignments to pixels p in P. Each Ap can be either “obj” or “bkg” (abbreviations of “object” and “background”). Vector A defines a segmentation. Then, the soft constraints that we impose on boundary and region properties of A are described by the cost function E(A):
  • E(A)=λ·R(A)+B(A)   (1)
  • [0024] where R ( A ) = p P R p ( A p ) ( 2 ) B ( A ) = { p , q } N B { p , q } · δ ( A p , A q ) and δ ( A p , A q ) = { 1 if A p A q 0 otherwise . ( 3 )
    Figure US20040008886A1-20040115-M00001
  • The coefficient λ≧0 in (1) specifies the relative importance of the region properties term R(A) versus the boundary properties term B(A). The regional term R(A) assumes that the individual penalties for assigning pixel p to “object” and “background”, correspondingly R[0025] p(·) may reflect on how the intensity of pixel p fits into a known intensity model (e.g., histogram) of the object and background.
  • The term B(A) comprises the “boundary” properties of segmentation A. Coefficient B[0026] {p,q}≧0 should be interpreted as a penalty for discontinuity between p and q. Normally, B{q,p} is large when pixels p and q are similar (e.g., in their intensity) and B{q,p{ is close to zero when the two are very different. The penalty B{q,p} can also decrease as a function of distance between p and q. Costs B{q,p} may be based on local intensity gradient, Laplacian zero-crossing, gradient direction, and other criteria.
  • Hard constraints that indicate segmentation regions rather than the boundary are considered. It is assumed that some pixels were marked as internal and some as external for the given object of interest. The subsets marked pixels will be referred to as “object” and “background” seeds. The segmentation boundary can be anywhere but it has to separate the object seeds from the background seeds. Note that the seeds can be loosely positioned inside the object and background regions. The segmentation technique described herein is quite stable and normally produces the same results regardless of particular seed positioning within the same image object. [0027]
  • Obviously, the hard constraints by themselves are not enough to obtain a good segmentation. A segmentation method decides how to segment unmarked pixels. Some conventional techniques use the same type of hard constraints as the present invention but they do not employ a clear cost function and segment unmarked pixels based on variations of “region growing”. Since the properties of segmentation boundary are not optimized, the results are prone to “leaking” where the boundary between objects is blurry. In contrast, the present invention combines the hard constraints as above with energy (1) that incorporates region and boundary properties of segments. [0028]
  • The segmentation technique described herein is based on powerful graph cut algorithms from the field of combinational optimization. The implementation uses a new version of the “max-flow” algorithm. Next we provide some terminology for graph cuts and provide some background information. [0029]
  • Graph Cuts First, we describe the basic terminology that pertains to graph cuts in the context of our segmentation method. An undirected graph G=(V,E) is defined as a set of nodes (vertices V) and a set of undirected edges (E) that connect these nodes. An example of a graph that we use in this paper is shown in FIG. 4([0030] a). Each edge eεE in the graph is assigned a nonnegative weight (cost) ωe. There are also two special nodes called terminals. A cut is a subset of edges C⊂E such that the terminals become separated on the induced graph G(C)=(V,E\C). It is normal in combinatorial optimization to define the cost of a cut as the sum of costs of the edges that it severs C = e C w e .
    Figure US20040008886A1-20040115-M00002
  • Graph cut formalism is well suited for segmentation of images. In fact, it is completely appropriate for N-dimensional volumes. The nodes of the graph can represent pixels (or voxels) and the edges can represent any neighborhood relationship between the pixels. A cut partitions the node in the graph. As illustrated in FIGS. [0031] 4(c) and (d), this partitioning corresponds to a segmentation of the underlying image or volume. A minimum cost cut generates a segmentation that is optimal in terms of properties that are built into the edge weights.
  • Segmentation Technique [0032]
  • In this section we provide algorithmic details about the segmentation technique. Assume that O and B denote the subsets of pixels marked as “object” and “background” seeds. Naturally, the subsets O⊂P and B⊂P are such that O∩B=φ. Remember that our goal is to compute global minimum of (1) among all segmentations A satisfying hard constraints [0033]
  • ∀pεO, Ap=“obj”  (4)
  • ∀pεB, Ap=“bkg”  (5)
  • The general work flow is described in conjunction with FIG. 4. Given an [0034] image 410 we create a graph 420 with two terminals 422, 424. The edge weights reflect the parameters in the regional (2) and the boundary (3) terms of the const function, as well as the known positions of seeds in the image.
  • The next step is to compute the globally optimal minimum cut. This cut gives a [0035] segmentation 430 of the original image 410. In the simplistic examples of FIG. 3 the image is divided into exactly one “object” and one “background” region. In general, our segmentation method generates binary segmentation with arbitrary topological properties. Other examples will illustrate that object and background segments may comprise several isolated connected blobs in the image.
  • Below we describe the details of the graph and prove that the obtained segmentation is optimal. To segment a given image we create a graph G=(V,E) with nodes corresponding to pixels pεP of the image. There are two additional nodes: an “object” terminal (a source S) and a “background” terminal (a sink T). Therefore, [0036]
  • V=P∪{S,T}.
  • The set of edges E consists of two types of undirected edges: n-links (neighborhood links) and t-links (terminal links). Each pixel p has two t-links {p,S} and {p,T} connecting it to each terminal. Each pair of neighboring pixels {p,q} in N is connected by an n-link. Without introducing any ambiguity, an n-link connecting a pair of neighbors p and q will be denoted {p,q}. Therefore, [0037] E = N p P { { p , S } , { p , T } } .
    Figure US20040008886A1-20040115-M00003
  • The following table gives weights of edges in E [0038]
    edge weight (cost) for
    {p, q} B{p,q} {p, q} ε N
    {p, S } λ · Rp (“bkg”) p ε P, p ∉ O ∪ B
    K p ∉ O
    0 p ∉ B
    {p, T} λ · Rp (“obj”) p ε P, p ∉ O ∪ B
    O p ε O
    K p ε B
  • where [0039] K = 1 + max q : { p , q } N B { p , q } .
    Figure US20040008886A1-20040115-M00004
  • The graph G is now completely defined. We draw the segmentation boundary between the object and the background by finding the minimum cost cut on the graph G. The minimum cost cut Ĉ on G can be computed exactly in polynomial time via algorithms for two terminal graph cuts assuming that the edge weights specified in the table above are non-negative. [0040]
  • Below we state exactly how the minimum cut Ĉ defines a segmentation  and prove this segmentation is optimal. We need one technical lemma. Assume that F denotes a set of all feasible cuts C on graph G such that [0041]
  • C severs exactly one t-link at each p [0042]
  • }p,q}εC iff p, q are t-linked to different terminals [0043]
  • if pεO then {p,T}εC [0044]
  • if pεB trhen {p,S}εC. [0045]
  • Lemma 1 The Minimum Cut on G is Feasible, i.e., ĈεF. [0046]
  • Proof: Ĉ severs at least one t-link at each pixel since it is a cut that separates the terminals. On the other hand, it cannot sever both t-links. In such a case it would not be minimal since one of the t-links could be returned. Similarly, a minimum cut should sever an n-link {p,q} if p and q are connected to the opposite terminals just because any cut must separate the terminals. If p and q are connected to the same terminal, then Ĉ should not sever unnecessary n-link {p,q} due to its minimality. The last two properties are true for Ĉ because the constant K is larger than the sun of all n-links costs for any given pixel p. For example, if pεO and Ĉ severs {p,S} (costs K) then we would construct a smaller cost cut by restoring {p,S} and severing all n-links from p (costs less than K) as well as the opposite t-link {p,T} (zero cost). [0047]
  • For any feasible cut CεF we can define a unique corresponding segmentation A(C) such that [0048] { obj , if { p , T } C bkg , if { p , S } C . ( 6 )
    Figure US20040008886A1-20040115-M00005
  • The definition above is coherent since any feasible cut severs exactly one of the two t-links at each pixel p. The lemma showed that a minimum cut Ĉ is feasible. Thus, we can define a corresponding segmentation Â=A(Ĉ). The next theorem completes the description of our algorithm. [0049]
  • Theorem 1 The Segmentation Â=A(Ĉ) Defined by the Minimum Cut Ĉ as in (6) Minimizes (1) Among All Segmentations Satisfying Constraints (4, 5). [0050]
  • Proof: Using the table of edge weights, definition of feasible cuts F, and equation (6) one can show that a cost of any CεF is [0051] C = p O B λ · R p ( A p ( C ) ) + { p , q } N B { p , q } · δ ( A p ( C ) , A q ( C ) ) = E ( A ( C ) ) - p O λ R p ( obj ) - p B λ R p ( bkg ) .
    Figure US20040008886A1-20040115-M00006
  • Therefore, |C|=E(A(C))−const(C). Note that for any CεF assignment A(C) satisfies constraints (4, 6). In fact, equation (6) gives one-to-one correspondence between the set of all feasible cuts in F and the set H of all assignments A that satisfy hard constraints (4, 5). Then, [0052] E ( A ^ ) = C ^ + const = min C F C + const = min C F ( E ( A ( C ) ) = min A H E ( A )
    Figure US20040008886A1-20040115-M00007
  • and the theorem is proved. [0053]
  • To conclude this section we would like to show that the algorithm can efficiently adjust the segmentation to incorporate any additional seeds that the user might interactively add. To be specific, assume that a max-flow algorithm is used to determine the minimum cut G. The max-flow algorithm gradually increases the flow sent from the source S to the sink T along the edges in G given their capacities (weights). Upon termination the maximum flow saturates the graph. The saturated edges correspond to the minimum cost cut on G giving us an optimal segmentation. [0054]
  • Assume now that an optimal segmentation is already computed for some initial set of seeds. A user adds a new “object” seed to pixel p that was not previously assigned any seed. We need to change the costs for two t-links at p [0055]
    t-link initial cost new cost
    {p, S} λRp (“bkg”) K
    {p, T} λRp (“obf”) 0
  • and then compute the maximum flow (minimum cut) on the new graph. In fact, we can start from the flow found at the end of initial computation. The only problem is that reassignment of edge weights as above reduces capacities of some edges. If there is a flow through such an edge then we may break the flow consistency. Increasing an edge capacity, on the other hand, is never a problem. Then, we can solve the problem as follows. [0056]
  • To accommodate the new “object” seed at pixel p we increase the t-links weights according to the table [0057]
    initial
    t-link cost add new cost
    {p, S} λRp (“bkg”) K + λRp (“obj”) K + cp
    {p, T} λRp (“obj”) λRp (“bkg”) cp
  • These new costs are consistent with the edge weight table for pixels in O since the extra constant c[0058] p at both t-links of a pixel does not change the optimal cut. Then, a maximum flow (minimum cut) on a new graph can be efficiently obtained starting from the previous flow without re-computing the whole solution from scratch.
  • Note that the same trick can be done to adjust the segmentation when a new “background” seed is added or when a seed is deleted. One has to figure the right amounts that have to be added to the costs of two t-links at the corresponding pixel. The new costs should be consistent with the edge weight table plus or minus the same constant. [0059]
  • EXAMPLES
  • We demonstrate the general-purpose segmentation method in several examples including photo/video editing. We show original data and segments generated by our technique for a given set of seeds. Our actual interface allows a user to enter seeds via mouse operated brush of red (for object) or blue (for background) color. Due to limitations of the B&W publication we show seeds as strokes of white (object) or black (background) brush. In addition, these strokes are marked by the letters “O” and “B”. For the purpose of clarity, we employ different methods for the presentation of segmentation results in our examples below. [0060]
  • Our current implementation actually makes a double use of the seeds entered by a user. First of all, they provide the hard constraints for the segmentation process as discussed above. In addition, we use intensities of pixels (voxels) marked as seeds to get histograms for “object” and “background” intensity distributions: Pr(I|O) and Pr(I|B). Then, we use these histograms to set the regional penalties R[0061] p(·) as negative log-liklihoods:
  • R p(“obj”)=−lnPr(I p |O)
  • R p(“bkg”)=−lnPr(I p |B).
  • To set the boundary penalties we use an ad-hoc function [0062] B { p , q } exp ( - ( I p - I q ) 2 2 σ 2 ) · 1 dist ( p , q )
    Figure US20040008886A1-20040115-M00008
  • This function penalizes a lot for discontinuities between pixels of similar intensities when |I[0063] p−Iq|<σ. However, if pixels are very different, |Ip−Iq|>σ, then the penalty is small. Intuitively, this function corresponds to the distribution of noise among neighboring pixels of an image. Thus, σ can be estimated as “camera noise”.
  • Note that we use an 8-neighborhood system in 2D examples and 26-neighborhood system in 3D examples. All running times are given for 333 MHz Pentium III. Our implementation uses a new “max-flow” algorithm from [2]. [0064]
  • Photo and Video Editing [0065]
  • In FIGS. [0066] 3(a)-(b) we illustrated the segmentation a bell with a group of people from a photograph. The user can start with a few “object” and “background” seeds loosely positioned inside and, correspondingly, outside the object(s) of interest. By reviewing the results of initial segmentation the user will see what areas are segmented incorrectly. Then (s)he can put additional seeds into the troubled places and efficiently recomputed the optimal segmentation. This process of adding seeds gives more clues to the algorithm and may be continued until the user likes the results.
  • Naturally, the hope is that the method can quickly identify the right object. The user would not like to keep adding new seeds until the whole image is covered in seeds. This is no better than manual segmentation. The performance of the algorithm can be judged by the efforts required from the user. Thus, the results of our segmentation are shown with seeds that we entered to get this segmentation. [0067]
  • Our segmentation algorithm runs in less than a second for most 2D images (up to 512×512) with a fixed set of seeds. When additional seeds are entered the solution is recomputed in the blink of an eye. Thus, the speed evaluation of our method in 2D is mainly concerned with the user efforts. The detailed segmentation in FIG. 3([0068] b) is obtained in approximately a minute. Note that in this example the algorithm created some isolated “background” segments. In fact, the algorithm automatically decided which “background” seeds were grouped together and which were placed into isolated segments. The same is equally true for the “object” seeds. The segments can have any topology.
  • In many cases the regional term of energy (1) helps to get the right results faster. In FIGS. [0069] 5(a) and 5(b) we show some details of segmentation with and without the regional term (λ=0) given the same sets of seeds. In FIG. 5(b) the user would spend more time by placing additional “background” seeds to correct imperfections.
  • The globally minimum two-terminal cut can be computed on any graph. Thus, our technique is valid for segmentation of N-D data. In FIG. 6 we segmented moving cars in a video sequence. The sequence of 21 video frames (255×189) was treated as a single 3D volume. The necessary seeds were entered in a simple 3D interface where we could browse through individual 2D slices (frames) of the volume. Initial seeds can be entered in just a few representative frames. Note that the boundary, region, and seed information is automatically propagated between the slices since we compute a globally optimum solution directly on the 3D data set. Thus, the whole sequence can be segmented based on seeds placed in just a few frames. For example, entering correcting seeds in one frame can fix imperfections in many adjacent frames. The results in FIG. 6 are obtained by placing seeds in 3 out of 21 frames. Each car was segmented in an independent experiment. We do not show seeds to avoid confusion. [0070]
  • The computation of the minimum cut is slower in 3D cases. The initial segmentation might take from 2-3 seconds on smaller volumes (200×200×10) to a few minutes on bigger ones (512×512×50). Thus, efficient re-computing of an optimal solution when new seeds are added is crucial. Most of the time the new seeds are incorporated in a few seconds even for bigger volumes. Therefore, we can still consider our method as “interactive”. The results in FIG. 6 can be obtained in approximately 30 seconds including user interactions. [0071]
  • The process of placing seeds can be automated for certain applications. The seeds should be positioned with a low probability of “false alarm” while the probability of “right detect” is not required to be high. Even very simple recognition techniques based on filters might be good enough if the corresponding threshold is set high. Such filters would have difficulties near the boundaries of objects but they can confidently place many seeds anywhere else. These seeds give hard constraints. Based on additional soft constraints (1) the minimum cut can complete the segmentation where the recognition method failed. [0072]
  • Although illustrative embodiments of the present invention have been described herein with reference to the accompanying drawings, it is to be understood that the invention is not limited to those precise embodiments, and that various other changes and modifications may be affected therein by one skilled in the art without departing from the scope or spirit of the invention. [0073]

Claims (25)

What is claimed is:
1. An image editing system, comprising:
an input device for inputting an image;
a graphical user interface for selecting background and object seeds for the image; and
an image processor for editing the image, the image processor including a segmentation routine that builds a graph associated with the image and uses a graph cut algorithm to cut the graph into segments.
2. The image editing system of claim 1, wherein nodes of the graph represent portions of the image.
3. The image editing system of claim 2, wherein the portions of the image are one of pixels and voxels.
4. The image editing system of claim 2, wherein the edges of the graph represents neighborhood relationships among the nodes.
5. The image editing system of claim 2, wherein the graph includes a background terminal node and an object terminal node.
6. The image editing system of claim 5, wherein each of the non-terminal nodes of the graph is connected to both the background terminal node and the object terminal node.
7. The image editing system of claim 6, wherein the edges of the graph are each assigned a cost value.
8. The image editing system of claim 7, wherein the cost values are determined according to a cost function that is defined in terms of boundary and region properties.
9. The image editing system of claim 8, wherein the cost function uses the background and object seed positions.
10. The image editing system of claim 7, wherein the graph is cut such that the total cost value of severed edges is minimized.
11. The image editing system of claim 1, further comprising an output device for displaying the segmented image.
12. The image editing system of claim 1, wherein the graphical user interface allows additional background and object seeds to be input.
13. A method for editing an-image, comprising the steps of:
selecting background and object seeds for the image; and
segmenting the image by building a graph associated with the image and using a graph cut algorithm to cut the graph into segments.
14. The method of claim 13, wherein nodes of the graph represent portions of the image.
15. The method of claim 14, wherein the portions of the image are one of pixels and voxels.
16. The method of claim 14, wherein the edges of the graph represents neighborhood relationships among the nodes.
17. The method of claim 14, wherein the graph includes a background terminal node and an object terminal node.
18. The method of claim 17 wherein each of the non-terminal nodes of the graph is connected to both the background terminal node and the object terminal node.
19. The method of claim 18, further including assigning a cost value to each of the edges of the graph.
20. The method of claim 19, wherein the cost values are determined according to a cost function that is defined in terms of boundary and region properties.
21. The-method of claim 20, wherein the cost function uses the background and object seed positions.
22. The method of claim 19, wherein the graph is cut such that the total cost value of severed edges is minimized.
23. The method of claim 13, further including the step of displaying the segmented image.
24. The method of claim 13, further including the step of selecting additional background and object seeds.
25. A program storage device readable by a machine, tangibly embodying a program of instructions executable on the machine to perform method steps for editing an image, comprising the method steps of:
selecting background and object seeds for the image; and
segmenting the image by building a graph associated with the image and using a graph cut algorithm to cut the graph into segments.
US10/413,974 2002-07-02 2003-04-15 Using graph cuts for editing photographs Abandoned US20040008886A1 (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
US10/413,974 US20040008886A1 (en) 2002-07-02 2003-04-15 Using graph cuts for editing photographs
US11/772,915 US7536050B2 (en) 2002-07-02 2007-07-03 Using graph cuts for editing photographs

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US39316302P 2002-07-02 2002-07-02
US10/413,974 US20040008886A1 (en) 2002-07-02 2003-04-15 Using graph cuts for editing photographs

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US11/772,915 Continuation US7536050B2 (en) 2002-07-02 2007-07-03 Using graph cuts for editing photographs

Publications (1)

Publication Number Publication Date
US20040008886A1 true US20040008886A1 (en) 2004-01-15

Family

ID=30118368

Family Applications (2)

Application Number Title Priority Date Filing Date
US10/413,974 Abandoned US20040008886A1 (en) 2002-07-02 2003-04-15 Using graph cuts for editing photographs
US11/772,915 Expired - Lifetime US7536050B2 (en) 2002-07-02 2007-07-03 Using graph cuts for editing photographs

Family Applications After (1)

Application Number Title Priority Date Filing Date
US11/772,915 Expired - Lifetime US7536050B2 (en) 2002-07-02 2007-07-03 Using graph cuts for editing photographs

Country Status (1)

Country Link
US (2) US20040008886A1 (en)

Cited By (45)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20050163375A1 (en) * 2004-01-23 2005-07-28 Leo Grady System and method for multi-label image segmentation
US20050226506A1 (en) * 2004-04-09 2005-10-13 Shmuel Aharon GPU multi-label image segmentation
US20060050959A1 (en) * 2004-08-26 2006-03-09 Leo Grady System and method for image segmentation by solving an inhomogenous dirichlet problem
US20060104513A1 (en) * 2004-11-15 2006-05-18 Shmuel Aharon GPU accelerated multi-label digital photo and video editing
US20060104510A1 (en) * 2004-11-15 2006-05-18 Shmuel Aharon GPU accelerated isoperimetric algorithm for image segmentation, digital photo and video editing
US20060147115A1 (en) * 2005-01-06 2006-07-06 Leo Grady System and method for image segmentation by a weighted multigrid solver
US20060147126A1 (en) * 2005-01-06 2006-07-06 Leo Grady System and method for multilabel random walker segmentation using prior models
WO2007005839A2 (en) 2005-07-01 2007-01-11 Microsoft Corporation Video object cut and paste
EP1750226A2 (en) 2005-08-01 2007-02-07 Siemens Medical Solutions USA, Inc. Editing of presegmented images/volumes with the multilabel random walker or graph cut segmentations
US7212201B1 (en) 1999-09-23 2007-05-01 New York University Method and apparatus for segmenting an image in order to locate a part thereof
US20070110310A1 (en) * 2005-10-07 2007-05-17 Siemens Corporate Research Inc Method and Apparatus For Performing Segmentation Based on Partial Differential Equations
US20070237420A1 (en) * 2006-04-10 2007-10-11 Microsoft Corporation Oblique image stitching
WO2008011594A2 (en) * 2006-07-21 2008-01-24 Adobe Systems Incorporated Live coherent image selection
US20080030497A1 (en) * 2005-12-08 2008-02-07 Yangqiu Hu Three dimensional modeling of objects
US20080069444A1 (en) * 2006-09-19 2008-03-20 Adobe Systems Incorporated Image mask generation
US20080080775A1 (en) * 2006-09-29 2008-04-03 Cornell Center For Technology Enterprise & Commercialization Methods and systems for reconstruction of objects
US20080122858A1 (en) * 2006-09-25 2008-05-29 Wilensky Gregg D Image masks
US20080143740A1 (en) * 2006-12-13 2008-06-19 Wilensky Gregg D Stable Mask Combination in Digital Image Editing
US20080247619A1 (en) * 2007-03-29 2008-10-09 Fujifilm Corporation Method, device and computer-readable recording medium containing program for extracting object region of interest
US20080304737A1 (en) * 2007-06-08 2008-12-11 Canon Kabushiki Kaisha Image processing apparatus and image processing method
US20080317342A1 (en) * 2007-06-22 2008-12-25 Matthias Fenchel Method for segmenting structures in image data records and image processing unit for carrying out the method
US20080317351A1 (en) * 2007-06-22 2008-12-25 Matthias Fenchel Method for interactively segmenting structures in image data records and image processing unit for carrying out the method
DE102008013909A1 (en) * 2008-03-12 2009-10-08 Siemens Aktiengesellschaft Method for segmenting image data for separating target structure by regional growth method on basis of graph, involves visualizing image data and graph by graphical user interface
US20100009928A1 (en) * 2004-03-29 2010-01-14 Cheng Jin Q Compositions including triciribine and taxanes and methods of use thereof
US7729537B2 (en) 2005-08-01 2010-06-01 Siemens Medical Solutions Usa, Inc. Editing of presegemented images/volumes with the multilabel random walker or graph cut segmentations
US20100201399A1 (en) * 2009-02-06 2010-08-12 Dieter Metzner Driver Circuit for a Two-Wire Conductor and Method for Generating Two Output Currents for a Two-Wire Conductor
CN101188018B (en) * 2007-12-06 2010-08-25 北大方正集团有限公司 An automatic land return method and device in typeset
US20110164152A1 (en) * 2008-09-24 2011-07-07 Li Hong Image segmentation from focus varied images using graph cuts
US20110164150A1 (en) * 2008-09-24 2011-07-07 Li Hong Automatic illuminant estimation that incorporates apparatus setting and intrinsic color casting information
US20110169979A1 (en) * 2008-09-24 2011-07-14 Li Hong Principal components analysis based illuminant estimation
CN102254302A (en) * 2011-06-07 2011-11-23 盛乐信息技术(上海)有限公司 Picture trimming system and method thereof
CN102270338A (en) * 2011-06-27 2011-12-07 西安交通大学 Method for effectively segmenting repeated object based on image representation improvement
US8533139B2 (en) 2011-09-06 2013-09-10 Czech Technical University In Prague, Faculty Of Electrical Engineering Optimizing computation of minimum cut in graphs with grid topology
US8860838B2 (en) 2008-09-24 2014-10-14 Nikon Corporation Automatic illuminant estimation and white balance adjustment based on color gamut unions
US20140307937A1 (en) * 2011-07-13 2014-10-16 Mckesson Financial Holdings Limited Methods, Apparatuses, and Computer Program Products for Identifying a Region of Interest Within a Mammogram Image
WO2015123792A1 (en) * 2014-02-19 2015-08-27 Qualcomm Incorporated Image editing techniques for a device
US9153031B2 (en) 2011-06-22 2015-10-06 Microsoft Technology Licensing, Llc Modifying video regions using mobile device input
US20150317800A1 (en) * 2014-05-05 2015-11-05 Xiaomi Inc. Method and device for image segmentation
US20160292848A1 (en) * 2015-04-02 2016-10-06 Kabushiki Kaisha Toshiba Medical imaging data processing apparatus and method
US9626476B2 (en) 2014-03-27 2017-04-18 Change Healthcare Llc Apparatus, method and computer-readable storage medium for transforming digital images
US20170140544A1 (en) * 2010-01-20 2017-05-18 Duke University Segmentation and identification of layered structures in images
CN107679467A (en) * 2017-09-19 2018-02-09 浙江师范大学 A kind of pedestrian's weight recognizer implementation method based on HSV and SDALF
US20180068419A1 (en) * 2016-09-08 2018-03-08 Sony Corporation Image processing system and method for object boundary smoothening for image segmentation
CN108111914A (en) * 2016-11-24 2018-06-01 广州市动景计算机科技有限公司 Video playback area recognition methods, device and apparatus for media playing
WO2021136224A1 (en) * 2019-12-31 2021-07-08 华为技术有限公司 Image segmentation method and device

Families Citing this family (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7844113B2 (en) * 2006-04-05 2010-11-30 Siemens Medical Solutions Usa, Inc. Region based push-relabel algorithm for efficient computation of maximum flow
US7889924B2 (en) * 2006-04-10 2011-02-15 Siemens Medical Solutions Usa, Inc. Globally optimal uninitialized graph-based rectilinear shape segmentation
JP4999163B2 (en) * 2006-04-17 2012-08-15 富士フイルム株式会社 Image processing method, apparatus, and program
US8970619B2 (en) 2009-11-24 2015-03-03 Microsoft Technology Licensing, Llc Parallelized generation of substantially seamless image mosaics
EP2706503A3 (en) * 2012-09-11 2017-08-30 Thomson Licensing Method and apparatus for bilayer image segmentation
JP5881625B2 (en) * 2013-01-17 2016-03-09 富士フイルム株式会社 Region dividing apparatus, program, and method
JP6269323B2 (en) * 2014-05-30 2018-01-31 富士ゼロックス株式会社 Image processing apparatus, image processing method, image processing system, and program
US9437008B1 (en) * 2015-07-21 2016-09-06 Sony Corporation Image segmentation using bayes risk estimation of scene foreground and background
US11017536B2 (en) 2018-05-02 2021-05-25 Mako Surgical Corp. Image segmentation

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5455898A (en) * 1993-11-24 1995-10-03 Xerox Corporation Analyzing an image showing a graphical representation of a layout
US5487117A (en) * 1991-12-31 1996-01-23 At&T Corp Graphical system for automated segmentation and recognition for image recognition systems
US5659639A (en) * 1993-11-24 1997-08-19 Xerox Corporation Analyzing an image showing editing marks to obtain category of editing operation
US6230312B1 (en) * 1998-10-02 2001-05-08 Microsoft Corporation Automatic detection of per-unit location constraints
US6556240B2 (en) * 1997-04-24 2003-04-29 Sony Corporation Video camera system having remote commander
US6590584B1 (en) * 1999-05-17 2003-07-08 Fuji Xerox Co., Ltd. Image editing method and apparatus
US6973212B2 (en) * 2000-09-01 2005-12-06 Siemens Corporate Research, Inc. Graph cuts for binary segmentation of n-dimensional images from object and background seeds

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7212201B1 (en) * 1999-09-23 2007-05-01 New York University Method and apparatus for segmenting an image in order to locate a part thereof

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5487117A (en) * 1991-12-31 1996-01-23 At&T Corp Graphical system for automated segmentation and recognition for image recognition systems
US5455898A (en) * 1993-11-24 1995-10-03 Xerox Corporation Analyzing an image showing a graphical representation of a layout
US5659639A (en) * 1993-11-24 1997-08-19 Xerox Corporation Analyzing an image showing editing marks to obtain category of editing operation
US6556240B2 (en) * 1997-04-24 2003-04-29 Sony Corporation Video camera system having remote commander
US6828994B2 (en) * 1997-04-24 2004-12-07 Sony Corporation Video camera system having remote commander
US6230312B1 (en) * 1998-10-02 2001-05-08 Microsoft Corporation Automatic detection of per-unit location constraints
US6590584B1 (en) * 1999-05-17 2003-07-08 Fuji Xerox Co., Ltd. Image editing method and apparatus
US6973212B2 (en) * 2000-09-01 2005-12-06 Siemens Corporate Research, Inc. Graph cuts for binary segmentation of n-dimensional images from object and background seeds

Cited By (88)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7212201B1 (en) 1999-09-23 2007-05-01 New York University Method and apparatus for segmenting an image in order to locate a part thereof
US20110109626A1 (en) * 1999-09-23 2011-05-12 New York University Method and apparatus for segmenting an image in order to locate a part thereof
US7893939B2 (en) 1999-09-23 2011-02-22 New York University Method and apparatus for segmenting an image in order to locate a part thereof
US8441481B2 (en) 1999-09-23 2013-05-14 New York University Method and apparatus for segmenting an image in order to locate a part thereof
US8854363B2 (en) 1999-09-23 2014-10-07 New York University Method and apparatus for segmenting an image in order to locate a part thereof
US9582891B2 (en) 1999-09-23 2017-02-28 New York University Method and apparatus for segmenting an image in order to locate a part thereof
US7460709B2 (en) * 2004-01-23 2008-12-02 Siemens Medical Solutions Usa, Inc. System and method for multi-label image segmentation
US20050163375A1 (en) * 2004-01-23 2005-07-28 Leo Grady System and method for multi-label image segmentation
US20100009928A1 (en) * 2004-03-29 2010-01-14 Cheng Jin Q Compositions including triciribine and taxanes and methods of use thereof
US20050226506A1 (en) * 2004-04-09 2005-10-13 Shmuel Aharon GPU multi-label image segmentation
US7697756B2 (en) * 2004-04-09 2010-04-13 Siemens Medical Solutions Usa, Inc. GPU accelerated multi-label image segmentation (MLS)
US20060050959A1 (en) * 2004-08-26 2006-03-09 Leo Grady System and method for image segmentation by solving an inhomogenous dirichlet problem
US7542604B2 (en) * 2004-08-26 2009-06-02 Siemens Medical Solutions Usa, Inc. System and method for image segmentation by solving an inhomogenous dirichlet problem
US20060104510A1 (en) * 2004-11-15 2006-05-18 Shmuel Aharon GPU accelerated isoperimetric algorithm for image segmentation, digital photo and video editing
US7630549B2 (en) * 2004-11-15 2009-12-08 Siemens Medical Solutions Usa. Inc. GPU accelerated multi-label digital photo and video editing
US20060104513A1 (en) * 2004-11-15 2006-05-18 Shmuel Aharon GPU accelerated multi-label digital photo and video editing
US7519220B2 (en) * 2004-11-15 2009-04-14 Siemens Medical Solutions Usa, Inc. GPU accelerated isoperimetric algorithm for image segmentation, digital photo and video editing
US20060147126A1 (en) * 2005-01-06 2006-07-06 Leo Grady System and method for multilabel random walker segmentation using prior models
US20060147115A1 (en) * 2005-01-06 2006-07-06 Leo Grady System and method for image segmentation by a weighted multigrid solver
US7565010B2 (en) * 2005-01-06 2009-07-21 Siemens Medical Solutions Usa, Inc. System and method for image segmentation by a weighted multigrid solver
US7486820B2 (en) * 2005-01-06 2009-02-03 Siemens Medical Solutions Usa, Inc. System and method for multilabel random walker segmentation using prior models
EP1899897A4 (en) * 2005-07-01 2013-04-03 Microsoft Corp Video object cut and paste
WO2007005839A2 (en) 2005-07-01 2007-01-11 Microsoft Corporation Video object cut and paste
EP1899897A2 (en) * 2005-07-01 2008-03-19 Microsoft Corporation Video object cut and paste
US7729537B2 (en) 2005-08-01 2010-06-01 Siemens Medical Solutions Usa, Inc. Editing of presegemented images/volumes with the multilabel random walker or graph cut segmentations
EP1750226A3 (en) * 2005-08-01 2007-12-26 Siemens Medical Solutions USA, Inc. Editing of presegmented images/volumes with the multilabel random walker or graph cut segmentations
EP1750226A2 (en) 2005-08-01 2007-02-07 Siemens Medical Solutions USA, Inc. Editing of presegmented images/volumes with the multilabel random walker or graph cut segmentations
US7957591B2 (en) * 2005-10-07 2011-06-07 Siemens Medical Solutions Usa, Inc. Method and apparatus for performing segmentation based on partial differential equations
US20070110310A1 (en) * 2005-10-07 2007-05-17 Siemens Corporate Research Inc Method and Apparatus For Performing Segmentation Based on Partial Differential Equations
US20080030497A1 (en) * 2005-12-08 2008-02-07 Yangqiu Hu Three dimensional modeling of objects
US20070237420A1 (en) * 2006-04-10 2007-10-11 Microsoft Corporation Oblique image stitching
US7778491B2 (en) * 2006-04-10 2010-08-17 Microsoft Corporation Oblique image stitching
WO2008011594A2 (en) * 2006-07-21 2008-01-24 Adobe Systems Incorporated Live coherent image selection
US8542923B2 (en) 2006-07-21 2013-09-24 Adobe Systems Incorporated Live coherent image selection
US8050498B2 (en) 2006-07-21 2011-11-01 Adobe Systems Incorporated Live coherent image selection to differentiate foreground and background pixels
WO2008011594A3 (en) * 2006-07-21 2008-08-14 Adobe Systems Inc Live coherent image selection
US8200014B2 (en) 2006-07-21 2012-06-12 Adobe Systems Incorporated Live coherent image selection
US20080069444A1 (en) * 2006-09-19 2008-03-20 Adobe Systems Incorporated Image mask generation
US8508546B2 (en) 2006-09-19 2013-08-13 Adobe Systems Incorporated Image mask generation
US20080122858A1 (en) * 2006-09-25 2008-05-29 Wilensky Gregg D Image masks
US8264500B2 (en) 2006-09-25 2012-09-11 Adobe Systems Incorporated Image masks generated from local color models
US8013870B2 (en) 2006-09-25 2011-09-06 Adobe Systems Incorporated Image masks generated from local color models
US20080080775A1 (en) * 2006-09-29 2008-04-03 Cornell Center For Technology Enterprise & Commercialization Methods and systems for reconstruction of objects
US8103068B2 (en) * 2006-09-29 2012-01-24 Cornell Research Foundation, Inc. Methods and systems for reconstruction of objects
US7701470B2 (en) * 2006-12-13 2010-04-20 Adobe Systems, Incorporated Stable mask combination in digital image editing
US20080143740A1 (en) * 2006-12-13 2008-06-19 Wilensky Gregg D Stable Mask Combination in Digital Image Editing
US20080247619A1 (en) * 2007-03-29 2008-10-09 Fujifilm Corporation Method, device and computer-readable recording medium containing program for extracting object region of interest
US8787642B2 (en) * 2007-03-29 2014-07-22 Fujifilm Corporation Method, device and computer-readable recording medium containing program for extracting object region of interest
US20080304737A1 (en) * 2007-06-08 2008-12-11 Canon Kabushiki Kaisha Image processing apparatus and image processing method
US8165391B2 (en) * 2007-06-08 2012-04-24 Canon Kabushiki Kaisha Image processing apparatus and image processing method to correct color misregistration near a boundary
DE102007028895B4 (en) * 2007-06-22 2010-07-15 Siemens Ag Method for segmenting structures in 3D image data sets
US20080317342A1 (en) * 2007-06-22 2008-12-25 Matthias Fenchel Method for segmenting structures in image data records and image processing unit for carrying out the method
US20080317351A1 (en) * 2007-06-22 2008-12-25 Matthias Fenchel Method for interactively segmenting structures in image data records and image processing unit for carrying out the method
DE102007028895A1 (en) 2007-06-22 2009-01-02 Siemens Ag Method for segmenting structures in image data sets and image processing unit for carrying out the method
US8180151B2 (en) 2007-06-22 2012-05-15 Siemens Aktiengesellschaft Method for segmenting structures in image data records and image processing unit for carrying out the method
US8200015B2 (en) 2007-06-22 2012-06-12 Siemens Aktiengesellschaft Method for interactively segmenting structures in image data records and image processing unit for carrying out the method
CN101188018B (en) * 2007-12-06 2010-08-25 北大方正集团有限公司 An automatic land return method and device in typeset
DE102008013909A1 (en) * 2008-03-12 2009-10-08 Siemens Aktiengesellschaft Method for segmenting image data for separating target structure by regional growth method on basis of graph, involves visualizing image data and graph by graphical user interface
US20110169979A1 (en) * 2008-09-24 2011-07-14 Li Hong Principal components analysis based illuminant estimation
US8860838B2 (en) 2008-09-24 2014-10-14 Nikon Corporation Automatic illuminant estimation and white balance adjustment based on color gamut unions
US9025043B2 (en) * 2008-09-24 2015-05-05 Nikon Corporation Image segmentation from focus varied images using graph cuts
US9013596B2 (en) 2008-09-24 2015-04-21 Nikon Corporation Automatic illuminant estimation that incorporates apparatus setting and intrinsic color casting information
US9118880B2 (en) 2008-09-24 2015-08-25 Nikon Corporation Image apparatus for principal components analysis based illuminant estimation
US20110164152A1 (en) * 2008-09-24 2011-07-07 Li Hong Image segmentation from focus varied images using graph cuts
US20110164150A1 (en) * 2008-09-24 2011-07-07 Li Hong Automatic illuminant estimation that incorporates apparatus setting and intrinsic color casting information
US20100201399A1 (en) * 2009-02-06 2010-08-12 Dieter Metzner Driver Circuit for a Two-Wire Conductor and Method for Generating Two Output Currents for a Two-Wire Conductor
US7888969B2 (en) 2009-02-06 2011-02-15 Infineon Technologies Ag Driver circuit for a two-wire conductor and method for generating two output currents for a two-wire conductor
US20170140544A1 (en) * 2010-01-20 2017-05-18 Duke University Segmentation and identification of layered structures in images
US10366492B2 (en) * 2010-01-20 2019-07-30 Duke University Segmentation and identification of layered structures in images
CN102254302A (en) * 2011-06-07 2011-11-23 盛乐信息技术(上海)有限公司 Picture trimming system and method thereof
US9153031B2 (en) 2011-06-22 2015-10-06 Microsoft Technology Licensing, Llc Modifying video regions using mobile device input
CN102270338A (en) * 2011-06-27 2011-12-07 西安交通大学 Method for effectively segmenting repeated object based on image representation improvement
US9058650B2 (en) * 2011-07-13 2015-06-16 Mckesson Financial Holdings Methods, apparatuses, and computer program products for identifying a region of interest within a mammogram image
US20140307937A1 (en) * 2011-07-13 2014-10-16 Mckesson Financial Holdings Limited Methods, Apparatuses, and Computer Program Products for Identifying a Region of Interest Within a Mammogram Image
US8533139B2 (en) 2011-09-06 2013-09-10 Czech Technical University In Prague, Faculty Of Electrical Engineering Optimizing computation of minimum cut in graphs with grid topology
WO2015123792A1 (en) * 2014-02-19 2015-08-27 Qualcomm Incorporated Image editing techniques for a device
US10026206B2 (en) 2014-02-19 2018-07-17 Qualcomm Incorporated Image editing techniques for a device
US9626476B2 (en) 2014-03-27 2017-04-18 Change Healthcare Llc Apparatus, method and computer-readable storage medium for transforming digital images
US9633444B2 (en) * 2014-05-05 2017-04-25 Xiaomi Inc. Method and device for image segmentation
US20150317800A1 (en) * 2014-05-05 2015-11-05 Xiaomi Inc. Method and device for image segmentation
US9773325B2 (en) * 2015-04-02 2017-09-26 Toshiba Medical Systems Corporation Medical imaging data processing apparatus and method
US20160292848A1 (en) * 2015-04-02 2016-10-06 Kabushiki Kaisha Toshiba Medical imaging data processing apparatus and method
US20180068419A1 (en) * 2016-09-08 2018-03-08 Sony Corporation Image processing system and method for object boundary smoothening for image segmentation
US10089721B2 (en) * 2016-09-08 2018-10-02 Sony Corporation Image processing system and method for object boundary smoothening for image segmentation
CN108111914A (en) * 2016-11-24 2018-06-01 广州市动景计算机科技有限公司 Video playback area recognition methods, device and apparatus for media playing
CN107679467A (en) * 2017-09-19 2018-02-09 浙江师范大学 A kind of pedestrian's weight recognizer implementation method based on HSV and SDALF
WO2021136224A1 (en) * 2019-12-31 2021-07-08 华为技术有限公司 Image segmentation method and device
CN113129307A (en) * 2019-12-31 2021-07-16 华为技术有限公司 Image segmentation method and image segmentation device

Also Published As

Publication number Publication date
US20070292025A1 (en) 2007-12-20
US7536050B2 (en) 2009-05-19

Similar Documents

Publication Publication Date Title
US7536050B2 (en) Using graph cuts for editing photographs
US8913830B2 (en) Multilevel image segmentation
Boykov et al. Interactive graph cuts for optimal boundary & region segmentation of objects in ND images
US8594428B2 (en) Interactive segmentation of images with single scribbles
US11120556B2 (en) Iterative method for salient foreground detection and multi-object segmentation
US7660463B2 (en) Foreground extraction using iterated graph cuts
Preteux On a distance function approach for gray-level mathematical morphology
EP1899897B1 (en) Video object cut and paste
US20060291721A1 (en) Efficiently labelling image pixels using graph cuts
US8787642B2 (en) Method, device and computer-readable recording medium containing program for extracting object region of interest
US20110075927A1 (en) Fast image segmentation using region merging with a k-nearest neighbor graph
US20110013837A1 (en) Hierarchical recursive image segmentation
US20040252886A1 (en) Automatic video object extraction
US20060104542A1 (en) Image tapestry
US20050147298A1 (en) Detection of sky in digital color images
US9501837B2 (en) Method and system for unsupervised image segmentation using a trained quality metric
JP2001236497A (en) Method for automatically forming cropped and zoomed deformation of photographic image
US20060147116A1 (en) Image segmentation method and device
US20040213460A1 (en) Method of human figure contour outlining in images
US7630541B2 (en) Image-wide matting
US20070279650A1 (en) Image editing using image-wide matting
US8135216B2 (en) Systems and methods for unsupervised local boundary or region refinement of figure masks using over and under segmentation of regions
US20220398737A1 (en) Medical image segmentation method based on u-network
JP2003162718A (en) Image processing method and its program
US7388988B2 (en) Systems and methods for processing boundary information of a graphical object

Legal Events

Date Code Title Description
AS Assignment

Owner name: SIEMENS CORPORATE RESEARCH, INC., NEW JERSEY

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:BOYKOV, YURI;REEL/FRAME:014246/0018

Effective date: 20030701

STCB Information on status: application discontinuation

Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION