WO2007120716A3 - Method and apparatus for automatically summarizing video - Google Patents

Method and apparatus for automatically summarizing video Download PDF

Info

Publication number
WO2007120716A3
WO2007120716A3 PCT/US2007/008951 US2007008951W WO2007120716A3 WO 2007120716 A3 WO2007120716 A3 WO 2007120716A3 US 2007008951 W US2007008951 W US 2007008951W WO 2007120716 A3 WO2007120716 A3 WO 2007120716A3
Authority
WO
WIPO (PCT)
Prior art keywords
video
scenes
similarity matrix
shot
frame
Prior art date
Application number
PCT/US2007/008951
Other languages
French (fr)
Other versions
WO2007120716A2 (en
Inventor
Jay N Yagnik
Original Assignee
Google Inc
Jay N Yagnik
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Google Inc, Jay N Yagnik filed Critical Google Inc
Publication of WO2007120716A2 publication Critical patent/WO2007120716A2/en
Publication of WO2007120716A3 publication Critical patent/WO2007120716A3/en

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/40—Scenes; Scene-specific elements in video content
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/70—Information retrieval; Database structures therefor; File system structures therefor of video data
    • G06F16/73—Querying
    • G06F16/738—Presentation of query results
    • G06F16/739—Presentation of query results in form of a video summary, e.g. the video summary being a video sequence, a composite still image or having synthesized frames
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/70—Information retrieval; Database structures therefor; File system structures therefor of video data
    • G06F16/78—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
    • G06F16/783—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
    • G06F16/7834—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content using audio features
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/70—Information retrieval; Database structures therefor; File system structures therefor of video data
    • G06F16/78—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
    • G06F16/783—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
    • G06F16/7847—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content using low-level visual features of the video content
    • G06F16/785—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content using low-level visual features of the video content using colour or luminescence
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2218/00—Aspects of pattern recognition specially adapted for signal processing
    • G06F2218/12—Classification; Matching

Abstract

A method for automatically producing a summary of a video, comprising: receiving the video at a computer system; partitioning the video into scenes; determining similarities between the scenes; selecting representative scenes from the video based on the determined similarities; and combining the selected scenes to produce the summary for the video, wherein partitioning the video into scenes involves: extracting feature vectors for sampled frames in the video; detecting shot boundaries based on distances between feature vectors for successive sampled frames; producing a frame-similarity matrix, wherein each element in the frame-similarity matrix represents a distance between feature vectors for a corresponding pair of sampled frames; using the frame-similarity matrix, the detected shot boundaries and a dynamic-programming technique to compute a shot-similarity matrix, wherein each element in the shot-similarity matrix represents a similarity between a corresponding pair of shots; and determining scene boundaries by selectively merging successive shots together based on the computed similarities between the successive shots and also based on audio breaks in the video.
PCT/US2007/008951 2006-04-12 2007-04-09 Method and apparatus for automatically summarizing video WO2007120716A2 (en)

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
US79186906P 2006-04-12 2006-04-12
US60/791,869 2006-04-12
US11/454,386 US8699806B2 (en) 2006-04-12 2006-06-15 Method and apparatus for automatically summarizing video
US11/454,386 2006-06-15

Publications (2)

Publication Number Publication Date
WO2007120716A2 WO2007120716A2 (en) 2007-10-25
WO2007120716A3 true WO2007120716A3 (en) 2008-04-17

Family

ID=38606286

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2007/008951 WO2007120716A2 (en) 2006-04-12 2007-04-09 Method and apparatus for automatically summarizing video

Country Status (2)

Country Link
US (2) US8699806B2 (en)
WO (1) WO2007120716A2 (en)

Families Citing this family (80)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8316081B2 (en) 2006-04-13 2012-11-20 Domingo Enterprises, Llc Portable media player enabled to obtain previews of a user's media collection
JP2010502085A (en) * 2006-08-25 2010-01-21 コーニンクレッカ フィリップス エレクトロニクス エヌ ヴィ Method and apparatus for automatically generating a summary of multimedia content items
US20080066107A1 (en) 2006-09-12 2008-03-13 Google Inc. Using Viewing Signals in Targeted Video Advertising
US8214374B1 (en) * 2011-09-26 2012-07-03 Limelight Networks, Inc. Methods and systems for abridging video files
US9015172B2 (en) 2006-09-22 2015-04-21 Limelight Networks, Inc. Method and subsystem for searching media content within a content-search service system
US8396878B2 (en) 2006-09-22 2013-03-12 Limelight Networks, Inc. Methods and systems for generating automated tags for video files
US8966389B2 (en) 2006-09-22 2015-02-24 Limelight Networks, Inc. Visual interface for identifying positions of interest within a sequentially ordered information encoding
US8667532B2 (en) * 2007-04-18 2014-03-04 Google Inc. Content recognition for targeting video advertisements
US20080276266A1 (en) * 2007-04-18 2008-11-06 Google Inc. Characterizing content for identification of advertising
US8433611B2 (en) 2007-06-27 2013-04-30 Google Inc. Selection of advertisements for placement with content
US9064024B2 (en) 2007-08-21 2015-06-23 Google Inc. Bundle generation
US20090083790A1 (en) * 2007-09-26 2009-03-26 Tao Wang Video scene segmentation and categorization
US9824372B1 (en) 2008-02-11 2017-11-21 Google Llc Associating advertisements with videos
US20100037149A1 (en) * 2008-08-05 2010-02-11 Google Inc. Annotating Media Content Items
WO2010089383A2 (en) * 2009-02-06 2010-08-12 Thomson Licensing Method for fingerprint-based video registration
US9167189B2 (en) * 2009-10-15 2015-10-20 At&T Intellectual Property I, L.P. Automated content detection, analysis, visual synthesis and repurposing
US9152708B1 (en) 2009-12-14 2015-10-06 Google Inc. Target-video specific co-watched video clusters
US20140033006A1 (en) * 2010-02-18 2014-01-30 Adobe Systems Incorporated System and method for selection preview
JP5553152B2 (en) * 2010-04-09 2014-07-16 ソニー株式会社 Image processing apparatus and method, and program
US20120151343A1 (en) * 2010-12-13 2012-06-14 Deep Tags, LLC Deep tags classification for digital media playback
WO2013023063A1 (en) 2011-08-09 2013-02-14 Path 36 Llc Digital media editing
CN103226586B (en) * 2013-04-10 2016-06-22 中国科学院自动化研究所 Video summarization method based on Energy distribution optimal strategy
CN105612554B (en) 2013-10-11 2019-05-10 冒纳凯阿技术公司 Method for characterizing the image obtained by video-medical equipment
US9225879B2 (en) * 2013-12-27 2015-12-29 TCL Research America Inc. Method and apparatus for video sequential alignment
US9760768B2 (en) 2014-03-04 2017-09-12 Gopro, Inc. Generation of video from spherical content using edit maps
US9685194B2 (en) 2014-07-23 2017-06-20 Gopro, Inc. Voice-based video tagging
US9792502B2 (en) 2014-07-23 2017-10-17 Gopro, Inc. Generating video summaries for a video using video summary templates
US9639762B2 (en) * 2014-09-04 2017-05-02 Intel Corporation Real time video summarization
CN105530554B (en) * 2014-10-23 2020-08-07 南京中兴新软件有限责任公司 Video abstract generation method and device
CN104394422B (en) * 2014-11-12 2017-11-17 华为软件技术有限公司 A kind of Video segmentation point acquisition methods and device
US9734870B2 (en) 2015-01-05 2017-08-15 Gopro, Inc. Media identifier generation for camera-captured media
KR102306538B1 (en) * 2015-01-20 2021-09-29 삼성전자주식회사 Apparatus and method for editing content
US9679605B2 (en) 2015-01-29 2017-06-13 Gopro, Inc. Variable playback speed template for video editing application
KR101650153B1 (en) * 2015-03-19 2016-08-23 네이버 주식회사 Cartoon data modifying method and cartoon data modifying device
US10074015B1 (en) * 2015-04-13 2018-09-11 Google Llc Methods, systems, and media for generating a summarized video with video thumbnails
US10186012B2 (en) 2015-05-20 2019-01-22 Gopro, Inc. Virtual lens simulation for video and photo cropping
CN105007433B (en) * 2015-06-03 2020-05-15 南京邮电大学 Moving object arrangement method based on energy constraint minimization of object
JP2017045374A (en) * 2015-08-28 2017-03-02 富士ゼロックス株式会社 Information processing device and program
US9894393B2 (en) 2015-08-31 2018-02-13 Gopro, Inc. Video encoding for reduced streaming latency
US10204273B2 (en) 2015-10-20 2019-02-12 Gopro, Inc. System and method of providing recommendations of moments of interest within video clips post capture
US9721611B2 (en) 2015-10-20 2017-08-01 Gopro, Inc. System and method of generating video from video clips based on moments of interest within the video clips
US10229324B2 (en) 2015-12-24 2019-03-12 Intel Corporation Video summarization using semantic information
US10095696B1 (en) 2016-01-04 2018-10-09 Gopro, Inc. Systems and methods for generating recommendations of post-capture users to edit digital media content field
US10109319B2 (en) 2016-01-08 2018-10-23 Gopro, Inc. Digital media editing
US9812175B2 (en) 2016-02-04 2017-11-07 Gopro, Inc. Systems and methods for annotating a video
KR20170098079A (en) * 2016-02-19 2017-08-29 삼성전자주식회사 Electronic device method for video recording in electronic device
US9972066B1 (en) 2016-03-16 2018-05-15 Gopro, Inc. Systems and methods for providing variable image projection for spherical visual content
US10402938B1 (en) 2016-03-31 2019-09-03 Gopro, Inc. Systems and methods for modifying image distortion (curvature) for viewing distance in post capture
US9838730B1 (en) 2016-04-07 2017-12-05 Gopro, Inc. Systems and methods for audio track selection in video editing
US9838731B1 (en) 2016-04-07 2017-12-05 Gopro, Inc. Systems and methods for audio track selection in video editing with audio mixing option
US9794632B1 (en) 2016-04-07 2017-10-17 Gopro, Inc. Systems and methods for synchronization based on audio track changes in video editing
US9998769B1 (en) 2016-06-15 2018-06-12 Gopro, Inc. Systems and methods for transcoding media files
US10250894B1 (en) 2016-06-15 2019-04-02 Gopro, Inc. Systems and methods for providing transcoded portions of a video
US9922682B1 (en) 2016-06-15 2018-03-20 Gopro, Inc. Systems and methods for organizing video files
US10045120B2 (en) 2016-06-20 2018-08-07 Gopro, Inc. Associating audio with three-dimensional objects in videos
US10185891B1 (en) 2016-07-08 2019-01-22 Gopro, Inc. Systems and methods for compact convolutional neural networks
US10469909B1 (en) 2016-07-14 2019-11-05 Gopro, Inc. Systems and methods for providing access to still images derived from a video
US10395119B1 (en) 2016-08-10 2019-08-27 Gopro, Inc. Systems and methods for determining activities performed during video capture
US9836853B1 (en) 2016-09-06 2017-12-05 Gopro, Inc. Three-dimensional convolutional neural networks for video highlight detection
US10282632B1 (en) 2016-09-21 2019-05-07 Gopro, Inc. Systems and methods for determining a sample frame order for analyzing a video
US10268898B1 (en) 2016-09-21 2019-04-23 Gopro, Inc. Systems and methods for determining a sample frame order for analyzing a video via segments
US10002641B1 (en) 2016-10-17 2018-06-19 Gopro, Inc. Systems and methods for determining highlight segment sets
US10284809B1 (en) 2016-11-07 2019-05-07 Gopro, Inc. Systems and methods for intelligently synchronizing events in visual content with musical features in audio content
US10262639B1 (en) 2016-11-08 2019-04-16 Gopro, Inc. Systems and methods for detecting musical features in audio content
US10534966B1 (en) 2017-02-02 2020-01-14 Gopro, Inc. Systems and methods for identifying activities and/or events represented in a video
US10339443B1 (en) 2017-02-24 2019-07-02 Gopro, Inc. Systems and methods for processing convolutional neural network operations using textures
US10127943B1 (en) 2017-03-02 2018-11-13 Gopro, Inc. Systems and methods for modifying videos based on music
US10185895B1 (en) 2017-03-23 2019-01-22 Gopro, Inc. Systems and methods for classifying activities captured within images
US10083718B1 (en) 2017-03-24 2018-09-25 Gopro, Inc. Systems and methods for editing videos based on motion
US10187690B1 (en) 2017-04-24 2019-01-22 Gopro, Inc. Systems and methods to detect and correlate user responses to media content
US10395122B1 (en) 2017-05-12 2019-08-27 Gopro, Inc. Systems and methods for identifying moments in videos
US10402698B1 (en) 2017-07-10 2019-09-03 Gopro, Inc. Systems and methods for identifying interesting moments within videos
US10614114B1 (en) 2017-07-10 2020-04-07 Gopro, Inc. Systems and methods for creating compilations based on hierarchical clustering
US10402656B1 (en) 2017-07-13 2019-09-03 Gopro, Inc. Systems and methods for accelerating video analysis
US10929945B2 (en) 2017-07-28 2021-02-23 Google Llc Image capture devices featuring intelligent use of lightweight hardware-generated statistics
US10445586B2 (en) 2017-12-12 2019-10-15 Microsoft Technology Licensing, Llc Deep learning on image frames to generate a summary
CN110321799B (en) * 2019-06-04 2022-11-18 武汉大学 Scene number selection method based on SBR and average inter-class distance
CN113453040B (en) * 2020-03-26 2023-03-10 华为技术有限公司 Short video generation method and device, related equipment and medium
US20230205815A1 (en) 2020-05-26 2023-06-29 Nec Corporation Information processing device, control method and storage medium
CN117459665A (en) * 2023-10-25 2024-01-26 杭州友义文化传媒有限公司 Video editing method, system and storage medium

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2004040480A1 (en) * 2002-11-01 2004-05-13 Mitsubishi Denki Kabushiki Kaisha Method for summarizing unknown content of video

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6263507B1 (en) * 1996-12-05 2001-07-17 Interval Research Corporation Browser for use in navigating a body of information, with particular application to browsing information represented by audiovisual data
US6535639B1 (en) * 1999-03-12 2003-03-18 Fuji Xerox Co., Ltd. Automatic video summarization using a measure of shot importance and a frame-packing method
US7016540B1 (en) * 1999-11-24 2006-03-21 Nec Corporation Method and system for segmentation, classification, and summarization of video images
US7305389B2 (en) * 2004-04-15 2007-12-04 Microsoft Corporation Content propagation for enhanced document retrieval
US7809722B2 (en) * 2005-05-09 2010-10-05 Like.Com System and method for enabling search and retrieval from image files based on recognized information
US7551234B2 (en) * 2005-07-28 2009-06-23 Seiko Epson Corporation Method and apparatus for estimating shot boundaries in a digital video sequence

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2004040480A1 (en) * 2002-11-01 2004-05-13 Mitsubishi Denki Kabushiki Kaisha Method for summarizing unknown content of video

Non-Patent Citations (4)

* Cited by examiner, † Cited by third party
Title
ANER-WOLF A ET AL: "Video summaries and cross-referencing through mosaic-based representation", COMPUTER VISION AND IMAGE UNDERSTANDING, ACADEMIC PRESS, ELSEVIER INC., vol. 95, no. 2, August 2004 (2004-08-01), pages 201 - 237, XP004520274, ISSN: 1077-3142 *
LU S ET AL: "A Novel Video Summarization Framework for Document Preparation and Archival Applications", 2005 PROCEEDINGS IEEE AEROSPACE CONFERENCE BIG SKY, MT, 5 March 2005 (2005-03-05), pages 1 - 10, XP010864565, ISBN: 0-7803-8870-4 *
UCHIHASHI S ET AL: "VIDEO MANGA: GENERATING SEMANTICALLY MEANINGFUL VIDEO SUMMARIES", ACM MULTIMEDIA, PROCEEDINGS OF THE INTERNATIONAL CONFERENCE, NEW YORK, NY, US, 1999, pages 383 - 392, XP001148132 *
ZHU X ET AL: "HIERARCHICAL VIDEO CONTENT DESCRIPTION AND SUMMARIZATION USING UNIFIED SEMANTIC AND VISUAL SIMILARITY", MULTIMEDIA SYSTEMS, ACM, SPRINGER VERLAG, vol. 9, no. 1, July 2003 (2003-07-01), pages 31 - 53, XP001178581, ISSN: 0942-4962 *

Also Published As

Publication number Publication date
US8879862B2 (en) 2014-11-04
US20070245242A1 (en) 2007-10-18
WO2007120716A2 (en) 2007-10-25
US20140161351A1 (en) 2014-06-12
US8699806B2 (en) 2014-04-15

Similar Documents

Publication Publication Date Title
WO2007120716A3 (en) Method and apparatus for automatically summarizing video
WO2012131653A3 (en) Devices, systems, methods, and media for detecting, indexing, and comparing video signals from a video display in a background scene using a camera-enabled device
KR101464572B1 (en) A method of adapting video images to small screen sizes
US9392322B2 (en) Method of visually synchronizing differing camera feeds with common subject
WO2008113596A3 (en) Method for the temporal segmentation of a video into video image sequences and for the selection of key frames to locate image content, taking into consideration sub-shot detection
EP2763077A3 (en) Method and apparatus for sensor aided extraction of spatio-temporal features
WO2009016833A1 (en) Video analysis apparatus and method for calculating inter-person evaluation value using video analysis
WO2008109567A3 (en) System and method for tracking three dimensional objects
WO2007127590A3 (en) Method and system for fingerprinting digital video object based on multiresolution, multirate spatial and temporal signatures
WO2007080133A3 (en) Method for determining and fingerprinting a key frame of a video sequence
WO2007024351A3 (en) Region of interest tracking and integration into a video codec
WO2013015546A3 (en) Method and system for providing additional information on broadcasting content
WO2006105054A3 (en) Method and system for improving video metadata through the use of frame-to-frame correspondences
WO2009053901A8 (en) Method and system for selecting the viewing configuration of a rendered figure
WO2006022394A3 (en) Method for identifying highlight segments in a video including a sequence of frames
WO2013189465A3 (en) Method, device and system for obtaining the number of persons
JP2013504938A5 (en)
WO2007097853A3 (en) Arthroplasty jigs and related methods
WO2009037558A3 (en) Method and system for capturing an image from video
CN109565618B (en) Media environment driven content distribution platform
WO2007136691A3 (en) Determining a toll amount
WO2008097222A3 (en) System and method for video-processing algorithm improvement
WO2014152313A3 (en) Method and system for recording information about rendered assets
EP2083561A3 (en) Scene switching point detector, scene switching point detecting method, recording apparatus, event generator, event generating method, reproducing apparatus, and computer program
WO2005038717A3 (en) Method of counting objects in a monitored environment and apparatus for the same

Legal Events

Date Code Title Description
NENP Non-entry into the national phase

Ref country code: DE

121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 07755278

Country of ref document: EP

Kind code of ref document: A2

122 Ep: pct application non-entry in european phase

Ref document number: 07755278

Country of ref document: EP

Kind code of ref document: A2