US20050155055A1 - Summarization of sumo video content - Google Patents

Summarization of sumo video content Download PDF

Info

Publication number
US20050155055A1
US20050155055A1 US11/075,284 US7528405A US2005155055A1 US 20050155055 A1 US20050155055 A1 US 20050155055A1 US 7528405 A US7528405 A US 7528405A US 2005155055 A1 US2005155055 A1 US 2005155055A1
Authority
US
United States
Prior art keywords
video
segments
summarization
sumo
frames
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Abandoned
Application number
US11/075,284
Inventor
Baoxin Li
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Sharp Laboratories of America Inc
Original Assignee
Sharp Laboratories of America Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Sharp Laboratories of America Inc filed Critical Sharp Laboratories of America Inc
Priority to US11/075,284 priority Critical patent/US20050155055A1/en
Assigned to SHARP LABORATORIES OF AMERICA, INC. reassignment SHARP LABORATORIES OF AMERICA, INC. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: LI, BAOXIN
Publication of US20050155055A1 publication Critical patent/US20050155055A1/en
Abandoned legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/70Information retrieval; Database structures therefor; File system structures therefor of video data
    • G06F16/73Querying
    • G06F16/738Presentation of query results
    • G06F16/739Presentation of query results in form of a video summary, e.g. the video summary being a video sequence, a composite still image or having synthesized frames
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/70Information retrieval; Database structures therefor; File system structures therefor of video data
    • G06F16/78Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
    • G06F16/783Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
    • G06F16/7834Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content using audio features
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/70Information retrieval; Database structures therefor; File system structures therefor of video data
    • G06F16/78Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
    • G06F16/783Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
    • G06F16/7844Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content using original textual content or text extracted from visual content or transcript of audio data
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/70Information retrieval; Database structures therefor; File system structures therefor of video data
    • G06F16/78Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
    • G06F16/783Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
    • G06F16/7847Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content using low-level visual features of the video content
    • G06F16/785Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content using low-level visual features of the video content using colour or luminescence
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00Image analysis
    • G06T7/20Analysis of motion
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00Scenes; Scene-specific elements
    • G06V20/40Scenes; Scene-specific elements in video content
    • G06V20/46Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
    • G06V20/47Detecting features for summarising video content
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234Processing of video elementary streams, e.g. splicing of video streams, manipulating MPEG-4 scene graphs
    • H04N21/23418Processing of video elementary streams, e.g. splicing of video streams, manipulating MPEG-4 scene graphs involving operations for analysing video streams, e.g. detecting features or characteristics
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/80Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
    • H04N21/85Assembly of content; Generation of multimedia applications
    • H04N21/854Content authoring
    • H04N21/8549Creating video summaries, e.g. movie trailer

Definitions

  • the present invention relates to summarization of video content including sumo.
  • the amount of video content is expanding at an ever increasing rate, some of which includes sporting events. Simultaneously, the available time for viewers to consume or otherwise view all of the desirable video content is decreasing. With the increased amount of video content coupled with the decreasing time available to view the video content, it b1ecomes increasingly problematic for viewers to view all of the potentially desirable content in its entirety. Accordingly, viewers are increasingly selective regarding the video content that they select to view. To accommodate viewer demands, techniques have been developed to provide a summarization of the video representative in some manner of the entire video. Video summarization likewise facilitates additional features including browsing, filtering, indexing, retrieval, etc. The typical purpose for creating a video summarization is to obtain a compact representation of the original video for subsequent viewing.
  • the first approach for video summarization is key frame detection.
  • Key frame detection includes mechanisms that process low level characteristics of the video, such as its color distribution, to determine those particular isolated frames that are most representative of particular portions of the video. For example, a key frame summarization of a video may contain only a few isolated key frames which potentially highlight the most important events in the video. Thus some limited information about the video can be inferred from the selection of key frames.
  • Key frame techniques are especially suitable for indexing video content but are not especially suitable for summarizing sporting content.
  • the second approach for video summarization is directed at detecting events that are important for the particular video content.
  • Such techniques normally include a definition and model of anticipated events of particular importance for a particular type of content.
  • the video summarization may consist of many video segments, each of which is a continuous portion in the original video, allowing some detailed information from the video to be viewed by the user in a time effective manner.
  • Such techniques are especially suitable for the efficient consumption of the content of a video by browsing only its summary.
  • Such approaches facilitate what is sometimes referred to as “semantic summaries”.
  • FIG. 1 is an exemplary flowchart for play detection.
  • FIG. 2 is an exemplary illustration of a pre-bout scene in sumo.
  • FIG. 3 is a technique for detecting a start frame of a sumo “play.”.
  • FIG. 4 is a pre-bout scene in sumo.
  • FIG. 5 illustrates the skin color and ring color of FIG. 4 .
  • FIG. 6 illustrates binarized skin color of FIG. 5 .
  • FIG. 7 is a horizontal projection of FIG. 6 .
  • FIG. 8 is a vertical projection of FIG. 6 .
  • FIGS. 9A-9C is a series of sequential images in a video clip showing two sumo contestants colliding.
  • FIG. 10 is an illustration of temporal evidence accumulation.
  • FIG. 11 is an illustration of color histogram differences.
  • FIG. 12 is an illustration of absolute pixel-to-pixel differences in luminance domain.
  • FIG. 13 illustrates scene cut detection
  • FIG. 14 illustrates names in a sumo video.
  • FIGS. 15A-15C illustrate audio segments of different plays.
  • FIG. 16 illustrates forming a multi-layered summary of the original video sequence.
  • FIG. 17 illustrates the video summarization module as part of a media browser and/or a service application.
  • FIG. 18 illustrates a video processing system
  • FIG. 19 illustrates an exemplary overall structure of the sumo summarization system.
  • Sumo the national sport of Japan, is tremendously popular in eastern Asia and is growing in popularity elsewhere in the world.
  • Sumo is a sport comprising bouts in which two contestants meet in a circular ring 4.55 meters in diameter.
  • the rules of Sumo are uncomplicated. After the contestants and a referee have entered the circular ring, the bout begins with an initial charge—called a “tachiai”—where each contestant rushes towards, then collides with, the other.
  • the bout will end when one of the contestant loses by either stepping outside the circular ring or touching the ground with any part of the contestant's body other than the soles of the feet.
  • Sumo participants may compete against each another in one of a number of tournaments.
  • Japan sponsors six sanctioned Grand Sumo tournaments, held in odd-numbered months throughout the year, in which competitive sumo contestants face one another with the opportunity for advancement in rank.
  • Sumo contestants are ranked under a strict meritocracy; winning bouts in these sanctioned tournaments improves a competitor's rank while losing bouts diminishes that rank.
  • a sumo tournament will typically take place over several weeks with bouts scheduled throughout each day, most bouts of interest, i.e. those involving higher ranked contestants, are scheduled to begin late afternoon when live television broadcasts of the tournament occur. These portions of the sumo tournaments usually last 2-3 hours each day and are often video recorded for later distribution or for re-broadcast.
  • the remaining time during the sumo tournament is typically not exciting to watch on video. Such time would include for example inter-bout changes of players, pre-bout exercises and ceremonies, post-bout ceremonies and in the case of broadcast, nearly endless commercials. While it may indeed be entertaining to sit in an arena for several hours for a sumo tournament, many people who watch a video of a sumo tournament find it difficult to watch all of the tournament, even if they are rabid fans. Further, the tournaments are held during daytime hours, hence many fans are unable to attend a tournament or to watch a live broadcast due to work. Such fans may nonetheless be interested in watching specific bouts or some other condensed version of the tournament.
  • a video summarization of the sumo tournament that provides a summary of the tournament having a duration shorter than the original sumo video, may be appealing to many people.
  • the video summarization should provide nearly the same level of the excitement (e.g. interest) that the original game provided.
  • sumo would not be a suitable candidate to attempt automated video summarization.
  • each of these moves may involve significant player motion that is difficult to anticipate, difficult to track, and is not consistent between plays.
  • the players are flesh toned and the ring is likewise generally flesh toned making identification of the events difficult. Based upon such considerations it has been previously considered impractical, if not impossible, to attempt to summarize sumo.
  • the present inventors determined that this technique is ultimately flawed as the models will likely never be sufficiently robust to detect all the desirable content. Moreover, the number of different types of model sequences of potentially desirable content is difficult to quantify. In contrast to attempting to detect particular model sequences, the present inventors determined that the desirable segments of the sumo match are preferably selected based upon a “play”. A “play” may be defined as a sequence of events defined by the rules of sumo.
  • the sequence of events of a “play” may generally include the time between which the players line up to charge one another and one player loses the bout by either stepping outside the sumo ring or touching the clay surface with a part of his body other than the soles of the feet.
  • a play may also selectively include certain pre-bout ceremonies or events, such as the time during which the contestants throw salt in the ring or stare at one another prior to charging. Normally the “play” should include a related series of activities that could potentially result in a victory by one contestant and a loss by the other contestant.
  • temporal bounds of a particular type of “play” does not necessarily start or end at a particular instance, but rather at a time generally coincident with the start and end of the play or otherwise based upon, at least in part, a time (e.g., event) based upon a play.
  • a “play” starting with the contestants throwing salt into the ring may include the times during which the contestants charge each other.
  • a summarization of the video is created by including a plurality of video segments, where the summarization includes fewer frames than the original video from which the summarization was created.
  • a summarization that includes a plurality of the plays of the sumo match provides the viewer with a shorted video sequence while permitting the viewer to still enjoy the game because most of the exciting portions of the video are provided, preferably in the same temporally sequential manner as in the original sumo video.
  • a procedure for summarization of a sumo video includes receiving a video sequence 20 that includes at least a portion of a sumo match.
  • Block 22 detects the start of a play of a video segment of a plurality of frames of the video. After detecting the start of the play, block 24 detects the end of the play, thereby defining a segment of video between the start of the play and the end of the play, namely, a “play”.
  • Block 26 then checks to see if the end of the video (or the portion to be processed) has been reached. If the end of the video has not been reached block 26 branches to block 22 to detect the next play. Alternatively, if the end of the video has been reached then block 26 branches to the summary description 28 .
  • the summary description defines those portions of the video sequence 20 that contain the relevant segments for the video summarization.
  • the summary description may be compliant with the MPEG-7 Summary Description Scheme or TV-Anytime Segmentation Description Scheme.
  • a compliant media browser such as shown in FIG. 17 , may apply the summary description to the input video to provide summarized viewing of the input video without modifying it.
  • the summary description may be used to edit the input video and create a separate video sequence.
  • the summarized video sequence may comprise the selected segments which excludes at least a portion of the original video other than the plurality of segments.
  • the summarized video sequence excludes all portions of the original video other than the plurality of segments.
  • FIG. 1 is intended to show a basic procedure for obtaining such a summary, where the summary description contains only the start and end points of the detected plays.
  • the summarization shown in FIG. 1 is primarily a low-level one, though in more complex situations it may contain other information, i.e. names of contestants etc.
  • the benefit of a low-level summary is that it provides sufficient detail for people to appreciate a game from the summary.
  • the low-level summary may then form the basis for a higher level summarization, if desired.
  • a higher level summary can be obtained by keeping only those plays receiving loud audience acclaims, achieved by adding an audio analysis procedure.
  • a summary can be obtained of only those plays containing a specific contestant.
  • a yet higher summary level may contain only key frames from the plays for indexing purposes.
  • One component of the summarization procedure depicted in FIG. 1 is the detection of an event, or “play.” If the start and end points of all plays are detected, then the system may string all the plays together to obtain a summary from the original video and perform some post processing to smooth the transition boundaries, such as using dissolving techniques to reduce abrupt change between plays and smoothing the audio filed for better auditory effects. Further, the summary should ideally contain only those segments comprising a “play” as earlier defined, thus providing a compact representation of the original tournament; the user can spend only a few minutes to watch it, yet almost all of the excitement of the original game can be appreciated.
  • a summary is to be obtained by first detecting the boundaries of a “play.”
  • two contestants meet in a ring 4.55 meters across. Though they wear silk belts around their waists, the players are otherwise unclothed.
  • Cameras are situated at fixed locations around the ring capture the sumo bout. The cameras can typically pan, tilt, and zoom.
  • the primary camera typically is situated opposite to the side where the referee stands.
  • a bout usually starts with a scene as illustrated in FIG. 2 , and the bout will almost always be broadcast in its entirety by the primary camera from this vantage.
  • Video captured by any other camera is typically used exclusively for replays, player close-ups, or post-bout ceremonies, all of which take place after the bout has ended. This format is adhered to because the primary camera can best cover the action of the bout, which usually lasts for mere moments, making it impractical to switch camera angles during a bout.
  • a play starts with a scene as in FIG. 2 .
  • the time between the scene cut at the end of a current play and the start of the following play is not usually exciting and can thus be excluded from a compact summary.
  • a scene like that shown in FIG. 2 is typically merely a necessary condition, not a sufficient condition.
  • a sumo tournament there are many pre-game ceremonies that result in a scene like that shown in FIG. 2 , but the contestants, are not yet ready to initiate the bout.
  • One test would be to determine whether the contestants charge one another and collide, because that is how each bout begins.
  • the methodology of detecting whether the start of a “play” has occurred involves locating a frame similar to that shown in FIG. 2 then applying a test to determine whether the frame immediately precedes the start of a bout.
  • the location of frames similar to that shown in FIG. 2 may be based on the anticipated characteristics of the image, as opposed to an actual analysis of the events depicted in the video.
  • the lower part of such frame contains the stage in which the sumo ring is defined.
  • the stage in the lower part of the frame is usually of fixed color and lighter than the generally dark color of the upper part of the frame. This is usually true because a sumo stage is to be constructed according to the same specifications. Further, in a sumo tournament the lights are usually focused on the stage which give tends to shroud the background in darkness.
  • each bout is preceded with the two contestants facing one another in a symmetric position about the center of the ring with the referee to the side and between the contestants and the primary camera facing the referee.
  • the color of the stage can be estimated from sample data; given a set of sample frames containing the stage, a set of parameters can give an estimate for the stage color. Detecting the players is a more difficult task. Theoretically, one could use complex methods such as those explicitly modeling the shape of a human body. To achieve fast computation, the present inventors have identified a simpler method in describing a player: a player is represented by a color blob with skin tone. Thus assuming that an estimate for skin tone is obtained, two blobs corresponding to the two respective players could be segmented. As mentioned earlier, in a Sumo broadcast, there are pregame ceremonies that could result in frames like a start frame. To enable this type of false alarm to be eliminated, the players should be tracked after they are detected to see if they move towards each other and eventually collide with each other, as would occur at the beginning of a “play” as earlier defined.
  • a video frame image IM can be examined to determine whether the image represents the beginning of a “play.”
  • the color descriptions may be for example, a single color, a range of colors, a set of colors, in one or more color spaces.
  • the image is examined to determine if it has a dark upper portion and a lower portion dominated (25% or more, 50% or more, or 75% or more) by the color Cs+Ck. If not, then the image is determined as a non-start frame.
  • the image is examined to determine whether there are two dominant (25% or more, 50% or more, or 75% or more) color blobs of color Ck, nearly symmetric to each other with respect to a generally center column (+/ ⁇ 20% of the width of the frame off center) of the frame. If not, then the image is determined as a non-start frame. If yes, subsequent frames are examined to determine whether the two dominant color blobs move towards, and eventually collide with, one another. If so, the original frame image IM is determined a start frame, otherwise it is determined not to be a start frame.
  • the technique may be modified to include fewer tests or additional tests, in the same or a different sequence.
  • FIG. 4 shows a candidate image IM that is a representative start frame of a sumo “play” as earlier defined, and thus should be detected by the summarization procedure shown in FIG. 1 .
  • the candidate image shown in FIG. 4 may be reduced to the image shown in FIG. 5 where white pixels indicate a place where there is a pixel in the candidate image corresponding to either the stage color Ck or the skin color Cs.
  • the black pixels represent the dark background areas of the candidate image.
  • the image may be further decomposed using skin-tone based segmentation to isolate those portions of the image corresponding to the skin color Cs.
  • a binary image, shown in FIG. 6 may be used to represent the obtained body parts, in which numeral ones represent a pixel of that location representing skin in the original image.
  • This binary image may be projected along vertical and horizontal axes, shown in FIGS. 7 and 8 , respectively.
  • the analysis of the blob may be performed on those projections.
  • the proposed projection behaves effectively like an integration process, which makes the algorithm less sensitive to imperfection in the skin/stage segmentation. Note that in these projections, small and isolated peaks have been suppressed.
  • a real start frame will result in two peaks of similar size in the vertical projection, nearly symmetric about the center column of the image, as shown in FIGS. 7 and 8 , the horizontal projection of the binary image, may be used to check whether the two blobs are symmetric about a center column of the image. In subsequent frames, these two peaks should move closer and closer, eventually converging, as illustrated by FIGS. 10A, 10B , and 10 C.
  • the foregoing method relies mainly on color cues, and prior knowledge about the stage color Cs and the skin tone Ck are assumed. However, it is also possible to calibrate the colors for a specific bout or tournament. With other inputs such as a human operator's interactions, the calibration is of course easy to do. Without any human interaction, statistical models can still be used to calibrate the color. If a series of start scene candidates has been obtained, statistical outliers in this set can be detected with prior coarse knowledge about Cs and Ck. The remaining candidate frames can then be used to estimate the specifics of the colors. With the colors calibrated, the start-of-play detection can be performed more accurately.
  • the different camera or different camera angle may be modeled by determining the amount of change between the current frame (or set of frames) to the next frame (or set of frames).
  • FIG. 11 a model of the amount of change between frames using a color histogram difference technique for an exemplary 1,000 frame video sumo clip is shown.
  • the peaks typically correspond to scene cuts.
  • FIG. 11 demonstrates, some scene cuts, like the one depicted at around frame 325 , the camera break produces a relatively low peak in the color histogram difference curve, causing potential failure in scene cut detection.
  • FIG. 13 shows the sum of absolute pixel-to-pixel luminance differences for the same video clip as shown in FIG. 11 .
  • Sumo video may also include gradual transitions between plays and other activities, such as commentary. These gradual transitions tend to be computationally complex to detect in the general case. However, in the case of sumo it has been determined that detecting gradual transitions based upon the color histogram differences is especially suitable. Other techniques may likewise be used. Referring to FIG. 13 , the preferred technique may include starting from a start-of-play time (t o ) and looking forward until a sufficiently large scene change is detected or until time t o +t p is reached, whichever occurs first. T p relates to the maximum anticipated play duration and therefore automatically sets a maximum duration to the play. This time period for processing to locate gradual transitions is denoted as t clean — cut .
  • t clean — cut ⁇ t low the system will not look for a gradual scene cut and set the previously detected scene cut as the end of the play. This corresponds to an anticipated minimum time duration for a play and t low is used to denote the minimum time period. Otherwise, the system looks for the highest color histogram difference in the region t low , t clean — cut or other measure of a potential scene change. This region of the segment is from the minimum time duration to the next previously identified scene cut. This identifies the highest color histogram difference in the time duration which may be a potential scene change. The time of the highest color histogram difference is identified at t 1 .
  • a statistical computation is performed, such as computing the mean ml and the standard deviation F of the color histogram differences.
  • C 1 and c 2 are constants or statistically calculated temporal values for the region to examine around the highest color histogram difference.
  • a mean filtering emphasizes regions having a relatively large difference in a relatively short time interval. If the color histogram differences at t 1 exceeds m 1 +c 3 *F 1 , where c 3 is a constant (or otherwise) and some of its neighbors (or otherwise) are sufficiently large, then the system considers a gradual transition to have occurred at around time (frame) t 1 . The play is set to the shorter of the previously identified scene cut or the gradual transition, if any.
  • the summary obtained by the method described above contains only play segments from the original video. Even though a Sumo fan may be able to quickly recognize the players after they appear, it may help a viewer to follow the game better if we detect those pre-play frames that contains player's names. An example of such type of frames is given in FIG. 14 .
  • the problem is one of detecting Kanji (Chinese characters used in Japanese) in images.
  • Kanji Choinese characters used in Japanese
  • the system may train a convolution neural network to perform this task.
  • Sumo broadcasting there are a few special patterns that are typically adopted in presenting the graphical characters. For example, the names of the two players are the biggest characters. Also, it appears that the names normally appear in white (or substantial contrast to the background). This is probably due to the fact that the names are usually overlaid on a dark scene of the sumo stadium.
  • the graphical information is symmetric with respect to the center column, with one player's information on the left, and the other player's information on the right. The characters read vertically from top to bottom.
  • the system may include an algorithm to find frames with these patterns.
  • the present inventors have found that the following set of rules may successfully detect frames with the desired player names in a video: (1) the frame has white blocks that are nearly symmetrically distributed about the center column of the image; (2) except for these white blocks, there should be no other white areas of significant size in the frame; (3) these white blocks persist for at least a few seconds; and (4) the set of frames with persistent white blocks proceeds to the start of a play.
  • One or more of these rules may be included, as desired.
  • the system may add them to their respective plays and obtain a new summary. Unlike the baseline summary obtained before, in this new summary, there are a few seconds of video like that in FIG. 14 for introducing each play. Thus the new summary is easier to follow.
  • a slow motion replay detection module may be incorporated.
  • the system detects if a slow motion replay has occurred, which normally relates to important events.
  • the system will capture the replays of plays, the same as the typical non-slow motion replay (full speed), if the same type of camera angles are used.
  • the play segments detected may be identified with multiple characteristics, namely, slow motion replay-only segments, play only segments without slow motion replay segments, and slow motion replay that include associated full speed segments.
  • the resulting summary may include one or more of the different selections of the aforementioned options, as desired. For example, the resulting summary may have the slow-motion replays removed. These options may likewise be user selectable.
  • While an effective summarization of a sumo video may be based on the concept of the “play”, sometimes the viewer may prefer an even shorter summarization with the most exciting plays included.
  • One potential technique for the estimation of the excitement of a play is to perform statistical analysis on the segments to determine which durations are most likely to have the highest excitement. However, this technique will likely not provide sufficiently accurate results. Further, excitement tends to be a subjective measure that is hard to quantify.
  • the present inventors came to the realization that the audio provided together with the video provides a good indication of the excitement of the plays. For example, the volume of the response of the audience and/or the commentators provides a good indication of the excitement. The louder audience and/or commentator acclamations, the greater the degree of excitement.
  • FIGS. 15A-15C an exemplary illustration is shown of audio signals having a relatively quiet response ( FIG. 15A ), having a strong response ( FIG. 15B ), and having an extremely strong response ( FIG. 15C ).
  • the mean audio volume of the play is large.
  • the play contains more audio samples that have middle-ranged magnitudes.
  • the first layer of the summary is constructed using the play detection technique.
  • the second and third layers are extracted as being of increasingly greater excitement, based at least in part, on the audio levels of the respective audio of the video segments.
  • the preferred audio technique only uses the temporal domain, which results in a computationally efficient technique.
  • the level of the audio may be used as a basis for the modification of the duration of a particular play segment. For example, if a particular play segment has a high audio level then the boundaries of the play segment may be extended. This permits a greater emphasis to be placed on those segments more likely to be exciting. For example, if a particular play segment has a low audio level then the boundaries of the play segment may be contracted. This permits a reduced emphasis to be placed on those segments less likely to be exciting.
  • the layered summarization may be based upon other factors, as desired.
  • the video summarization may be included as part of an MPEG-7 based browser/filter, where summarization is included within the standard.
  • the media summarizer may be as shown in FIG. 1 .
  • the system can provide the user with varying levels of summaries according to their demands.
  • the summary information is described as an MPEG-7 compliant XML document, one can utilize all the offerings of MPEG-7, such as personalization, where different levels of summaries can be offered to the user on the basis of user's preferences described in an MPEG-7 compliant way.
  • Descriptions of user preferences in MPEG-7 include preference elements pertaining to different summary modes and detail levels.
  • the user downloads and receives the summary description encoded in MPEG-7 format.
  • the media and its summary description reside at the provider's VOD server and the user (e.g., remote) consumes the summary via a user-side browser interface.
  • the summary may be enriched further by additional information that may be added by the service provider.
  • summarization may also be performed by the client.
  • the output of the module that automatically detects important segments may be a set of indices of segments containing plays and important parts of the input video program.
  • a description document such as an MPEG-7 or TV-Anytime compliant description is generated in The Description Generation module.
  • Summary segments are made available to the Post-Processing module by The Extraction of Summary Segments module which processes the input video program according to the description.
  • a post-processing module processes the summary Segments and/or the description to generate the final summary video and final description.
  • the post-processing module puts the post-processed segments together to form the final summary video.
  • the post-processing module may transcode the resulting video to a format different that of the input video to meet the requirements of the storage/transmission channel.
  • Post-processing may include adding to the original audio track a commentary, insertion of advertisement segments, or metadata.
  • post-processing may be completely, or in part, manual processing. It may include, for example, automatic ranking and subset selection of events on the basis of automatic detection of features in the audio track associated with video segments. This processing may be performed at the server and then the resulting video transferred to the client, normally over a network. Alternatively, the resulting video is included in a VOD library and made available to users on a VOD server.
  • a system may be developed that incorporates start detection of a play, end detection of a play, and summarization.
  • the detection technique may be based upon processing a single frame, multiple frames, or a combination thereof.

Abstract

Summarization of video content including sumo.

Description

    BACKGROUND OF THE INVENTION
  • The present invention relates to summarization of video content including sumo.
  • The amount of video content is expanding at an ever increasing rate, some of which includes sporting events. Simultaneously, the available time for viewers to consume or otherwise view all of the desirable video content is decreasing. With the increased amount of video content coupled with the decreasing time available to view the video content, it b1ecomes increasingly problematic for viewers to view all of the potentially desirable content in its entirety. Accordingly, viewers are increasingly selective regarding the video content that they select to view. To accommodate viewer demands, techniques have been developed to provide a summarization of the video representative in some manner of the entire video. Video summarization likewise facilitates additional features including browsing, filtering, indexing, retrieval, etc. The typical purpose for creating a video summarization is to obtain a compact representation of the original video for subsequent viewing.
  • There are two major approaches to video summarization. The first approach for video summarization is key frame detection. Key frame detection includes mechanisms that process low level characteristics of the video, such as its color distribution, to determine those particular isolated frames that are most representative of particular portions of the video. For example, a key frame summarization of a video may contain only a few isolated key frames which potentially highlight the most important events in the video. Thus some limited information about the video can be inferred from the selection of key frames. Key frame techniques are especially suitable for indexing video content but are not especially suitable for summarizing sporting content.
  • The second approach for video summarization is directed at detecting events that are important for the particular video content. Such techniques normally include a definition and model of anticipated events of particular importance for a particular type of content. The video summarization may consist of many video segments, each of which is a continuous portion in the original video, allowing some detailed information from the video to be viewed by the user in a time effective manner. Such techniques are especially suitable for the efficient consumption of the content of a video by browsing only its summary. Such approaches facilitate what is sometimes referred to as “semantic summaries”.
  • What is desired, therefore, is a video summarization technique suitable for video content that includes sumo.
  • BRIEF DESCRIPTION OF THE DRAWINGS
  • FIG. 1 is an exemplary flowchart for play detection.
  • FIG. 2 is an exemplary illustration of a pre-bout scene in sumo.
  • FIG. 3 is a technique for detecting a start frame of a sumo “play.”.
  • FIG. 4 is a pre-bout scene in sumo.
  • FIG. 5 illustrates the skin color and ring color of FIG. 4.
  • FIG. 6 illustrates binarized skin color of FIG. 5.
  • FIG. 7 is a horizontal projection of FIG. 6.
  • FIG. 8 is a vertical projection of FIG. 6.
  • FIGS. 9A-9C is a series of sequential images in a video clip showing two sumo contestants colliding.
  • FIG. 10 is an illustration of temporal evidence accumulation.
  • FIG. 11 is an illustration of color histogram differences.
  • FIG. 12 is an illustration of absolute pixel-to-pixel differences in luminance domain.
  • FIG. 13 illustrates scene cut detection.
  • FIG. 14 illustrates names in a sumo video.
  • FIGS. 15A-15C illustrate audio segments of different plays.
  • FIG. 16 illustrates forming a multi-layered summary of the original video sequence.
  • FIG. 17 illustrates the video summarization module as part of a media browser and/or a service application.
  • FIG. 18 illustrates a video processing system.
  • FIG. 19 illustrates an exemplary overall structure of the sumo summarization system.
  • DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT
  • Sumo, the national sport of Japan, is tremendously popular in eastern Asia and is growing in popularity elsewhere in the world. Sumo is a sport comprising bouts in which two contestants meet in a circular ring 4.55 meters in diameter. The rules of Sumo are uncomplicated. After the contestants and a referee have entered the circular ring, the bout begins with an initial charge—called a “tachiai”—where each contestant rushes towards, then collides with, the other. The bout will end when one of the contestant loses by either stepping outside the circular ring or touching the ground with any part of the contestant's body other than the soles of the feet. Aside from a limited number of illegal moves, such as gouging the opponent's eyes, striking with a closed fist, or intentionally pulling at the opponent's hair, there are no rules that govern a sumo bout.
  • Sumo participants may compete against each another in one of a number of tournaments. Japan sponsors six sanctioned Grand Sumo tournaments, held in odd-numbered months throughout the year, in which competitive sumo contestants face one another with the opportunity for advancement in rank. Sumo contestants are ranked under a strict meritocracy; winning bouts in these sanctioned tournaments improves a competitor's rank while losing bouts diminishes that rank. Aside from the six sanctioned tournaments, a number of exhibition tournaments—called Jungyo—are scheduled throughout the year.
  • Though a sumo tournament will typically take place over several weeks with bouts scheduled throughout each day, most bouts of interest, i.e. those involving higher ranked contestants, are scheduled to begin late afternoon when live television broadcasts of the tournament occur. These portions of the sumo tournaments usually last 2-3 hours each day and are often video recorded for later distribution or for re-broadcast.
  • Though such a video of a sumo tournament might typically last about 2-3 hours, only about ten minutes turns out to include time during which two players are in a bout. An individual sumo bout is brief; the typical bout will end with the initial collision, though a rare bout might last two to three minutes. Interspersed between bouts are a large number of ceremonies that precede and follow each bout.
  • Though brief, the time intervals during which a bout is proceeding are intense and can captivate those in the viewing audience, many of whom are able to identify a myriad of named sumo techniques that may occur in rapid succession. Such techniques include a “kekaeshi” (a foot-sweep), a “kubinage” (a head-lock throw), and an “izori” (a technique where a contestant crouches below the opponent's rush, grabbing one of the opponent's legs, lifting the opponent upon the shoulders and falling backwards), as well as some sixty five to seventy more named sumo techniques or occurrences.
  • The remaining time during the sumo tournament is typically not exciting to watch on video. Such time would include for example inter-bout changes of players, pre-bout exercises and ceremonies, post-bout ceremonies and in the case of broadcast, nearly endless commercials. While it may indeed be entertaining to sit in an arena for several hours for a sumo tournament, many people who watch a video of a sumo tournament find it difficult to watch all of the tournament, even if they are rabid fans. Further, the tournaments are held during daytime hours, hence many fans are unable to attend a tournament or to watch a live broadcast due to work. Such fans may nonetheless be interested in watching specific bouts or some other condensed version of the tournament. Thus a video summarization of the sumo tournament that provides a summary of the tournament having a duration shorter than the original sumo video, may be appealing to many people. The video summarization should provide nearly the same level of the excitement (e.g. interest) that the original game provided.
  • Upon initial consideration, sumo would not be a suitable candidate to attempt automated video summarization. Initially, there are nearly an endless number of potential moves that may occur that would need to be accounted for in some manner. In addition, each of these moves may involve significant player motion that is difficult to anticipate, difficult to track, and is not consistent between plays. In addition, the players are flesh toned and the ring is likewise generally flesh toned making identification of the events difficult. Based upon such considerations it has been previously considered impractical, if not impossible, to attempt to summarize sumo.
  • It is conceivably possible to develop highly sophisticated models of a typical sumo video to identify potentially relevant portions of the video. However, such highly sophisticated models are difficult to create and are not normally robust. Further, the likelihood that a majority of the highly relevant portions of the sumo video will be included in such a video summarization is low because of the selectivity of the model. Thus the resulting video summarization of the sumo tournament may simply be unsatisfactory to the average viewer.
  • After consideration of the difficulty of developing highly sophisticated models of a sumo video to analyze the content of the sumo video, as the sole basis upon which to create a sumo summarization, the present inventors determined that this technique is ultimately flawed as the models will likely never be sufficiently robust to detect all the desirable content. Moreover, the number of different types of model sequences of potentially desirable content is difficult to quantify. In contrast to attempting to detect particular model sequences, the present inventors determined that the desirable segments of the sumo match are preferably selected based upon a “play”. A “play” may be defined as a sequence of events defined by the rules of sumo. In particular, and in one aspect, the sequence of events of a “play” may generally include the time between which the players line up to charge one another and one player loses the bout by either stepping outside the sumo ring or touching the clay surface with a part of his body other than the soles of the feet. A play may also selectively include certain pre-bout ceremonies or events, such as the time during which the contestants throw salt in the ring or stare at one another prior to charging. Normally the “play” should include a related series of activities that could potentially result in a victory by one contestant and a loss by the other contestant.
  • It is to be understood that the temporal bounds of a particular type of “play” does not necessarily start or end at a particular instance, but rather at a time generally coincident with the start and end of the play or otherwise based upon, at least in part, a time (e.g., event) based upon a play. For example, a “play” starting with the contestants throwing salt into the ring may include the times during which the contestants charge each other. A summarization of the video is created by including a plurality of video segments, where the summarization includes fewer frames than the original video from which the summarization was created. A summarization that includes a plurality of the plays of the sumo match provides the viewer with a shorted video sequence while permitting the viewer to still enjoy the game because most of the exciting portions of the video are provided, preferably in the same temporally sequential manner as in the original sumo video.
  • Referring to FIG. 1, a procedure for summarization of a sumo video includes receiving a video sequence 20 that includes at least a portion of a sumo match. Block 22 detects the start of a play of a video segment of a plurality of frames of the video. After detecting the start of the play, block 24 detects the end of the play, thereby defining a segment of video between the start of the play and the end of the play, namely, a “play”. Block 26 then checks to see if the end of the video (or the portion to be processed) has been reached. If the end of the video has not been reached block 26 branches to block 22 to detect the next play. Alternatively, if the end of the video has been reached then block 26 branches to the summary description 28. The summary description defines those portions of the video sequence 20 that contain the relevant segments for the video summarization. The summary description may be compliant with the MPEG-7 Summary Description Scheme or TV-Anytime Segmentation Description Scheme. A compliant media browser, such as shown in FIG. 17, may apply the summary description to the input video to provide summarized viewing of the input video without modifying it. Alternatively, the summary description may be used to edit the input video and create a separate video sequence. The summarized video sequence may comprise the selected segments which excludes at least a portion of the original video other than the plurality of segments. Preferably, the summarized video sequence excludes all portions of the original video other than the plurality of segments.
  • FIG. 1 is intended to show a basic procedure for obtaining such a summary, where the summary description contains only the start and end points of the detected plays. The summarization shown in FIG. 1 is primarily a low-level one, though in more complex situations it may contain other information, i.e. names of contestants etc. The benefit of a low-level summary is that it provides sufficient detail for people to appreciate a game from the summary. The low-level summary may then form the basis for a higher level summarization, if desired. As one example, a higher level summary can be obtained by keeping only those plays receiving loud audience acclaims, achieved by adding an audio analysis procedure. Alternatively, in combination with a caption detection/recognition module, a summary can be obtained of only those plays containing a specific contestant. A yet higher summary level may contain only key frames from the plays for indexing purposes.
  • One component of the summarization procedure depicted in FIG. 1 is the detection of an event, or “play.” If the start and end points of all plays are detected, then the system may string all the plays together to obtain a summary from the original video and perform some post processing to smooth the transition boundaries, such as using dissolving techniques to reduce abrupt change between plays and smoothing the audio filed for better auditory effects. Further, the summary should ideally contain only those segments comprising a “play” as earlier defined, thus providing a compact representation of the original tournament; the user can spend only a few minutes to watch it, yet almost all of the excitement of the original game can be appreciated.
  • One of the difficulties in the detection of a “play” in a sumo broadcast is that frames in one play may sweep a large range of color, yet all the frames belong to the same event, and form an uninterrupted video clip. Thus a generic summarization scheme that uses, for example, a color histogram as the cue for key frame detection or scene classification, may not be particularly effective. In light of such difficulties, the present inventors have developed an alternate method for detecting a “play” that is specifically tailored to sumo content.
  • Still referring to FIG. 1, a summary is to be obtained by first detecting the boundaries of a “play.” In a sumo bout, two contestants meet in a ring 4.55 meters across. Though they wear silk belts around their waists, the players are otherwise unclothed. There are strict rules as to where the players and the referee, called a “Gyoji,” are to stand in the moments immediately proceeding the initiation of the bout. Cameras are situated at fixed locations around the ring capture the sumo bout. The cameras can typically pan, tilt, and zoom. The primary camera typically is situated opposite to the side where the referee stands. Thus a bout usually starts with a scene as illustrated in FIG. 2, and the bout will almost always be broadcast in its entirety by the primary camera from this vantage. Video captured by any other camera is typically used exclusively for replays, player close-ups, or post-bout ceremonies, all of which take place after the bout has ended. This format is adhered to because the primary camera can best cover the action of the bout, which usually lasts for mere moments, making it impractical to switch camera angles during a bout.
  • Based on these observations, the inventors have developed a model for “play” detection. A play starts with a scene as in FIG. 2. The time between the scene cut at the end of a current play and the start of the following play is not usually exciting and can thus be excluded from a compact summary. Note that a scene like that shown in FIG. 2 is typically merely a necessary condition, not a sufficient condition. In a sumo tournament, there are many pre-game ceremonies that result in a scene like that shown in FIG. 2, but the contestants, are not yet ready to initiate the bout. Thus in order to detect the start of a play, in addition to finding a scene like that depicted in FIG. 2, it should determine whether the scene is an immediate precursor to the start of a bout. One test would be to determine whether the contestants charge one another and collide, because that is how each bout begins. In other words, the methodology of detecting whether the start of a “play” has occurred involves locating a frame similar to that shown in FIG. 2 then applying a test to determine whether the frame immediately precedes the start of a bout.
  • The location of frames similar to that shown in FIG. 2 may be based on the anticipated characteristics of the image, as opposed to an actual analysis of the events depicted in the video. Under the assumption that a camera gives a typical start frame like that shown in FIG. 2, one can observe that the lower part of such frame contains the stage in which the sumo ring is defined. The stage in the lower part of the frame is usually of fixed color and lighter than the generally dark color of the upper part of the frame. This is usually true because a sumo stage is to be constructed according to the same specifications. Further, in a sumo tournament the lights are usually focused on the stage which give tends to shroud the background in darkness. In addition, each bout is preceded with the two contestants facing one another in a symmetric position about the center of the ring with the referee to the side and between the contestants and the primary camera facing the referee.
  • The color of the stage can be estimated from sample data; given a set of sample frames containing the stage, a set of parameters can give an estimate for the stage color. Detecting the players is a more difficult task. Theoretically, one could use complex methods such as those explicitly modeling the shape of a human body. To achieve fast computation, the present inventors have identified a simpler method in describing a player: a player is represented by a color blob with skin tone. Thus assuming that an estimate for skin tone is obtained, two blobs corresponding to the two respective players could be segmented. As mentioned earlier, in a Sumo broadcast, there are pregame ceremonies that could result in frames like a start frame. To enable this type of false alarm to be eliminated, the players should be tracked after they are detected to see if they move towards each other and eventually collide with each other, as would occur at the beginning of a “play” as earlier defined.
  • One method for detecting the beginning of a play may proceed as shown in FIG. 3. Given a stage color description Cs and skin tone description Ck, a video frame image IM can be examined to determine whether the image represents the beginning of a “play.” The color descriptions, may be for example, a single color, a range of colors, a set of colors, in one or more color spaces. First, the image is examined to determine if it has a dark upper portion and a lower portion dominated (25% or more, 50% or more, or 75% or more) by the color Cs+Ck. If not, then the image is determined as a non-start frame. If yes, then the image is examined to determine whether there are two dominant (25% or more, 50% or more, or 75% or more) color blobs of color Ck, nearly symmetric to each other with respect to a generally center column (+/−20% of the width of the frame off center) of the frame. If not, then the image is determined as a non-start frame. If yes, subsequent frames are examined to determine whether the two dominant color blobs move towards, and eventually collide with, one another. If so, the original frame image IM is determined a start frame, otherwise it is determined not to be a start frame. The technique may be modified to include fewer tests or additional tests, in the same or a different sequence.
  • It turns out that a difficult part of this method is to segment the player blobs from the stage because the stage color Cs and the skin tone Ck are overlapping in typical color space. It is impossible to perfectly separate skin from the stage only using color information, which means that the player detection is always imperfect and the players are usually detected as fragmented pieces. In fact, this is inevitable, considering that the players often wear belts of various non-skin tone color. If a single blob is to be detected for each player, then an additional module must be used to group the fragmented pieces. This module may again introduce additional inaccuracies, aside from the demand for additional computation.
  • To avoid the computational burden and potential inaccuracies of such a grouping procedure, the present inventors discovered that the foregoing method for detecting the beginning of a play may be implemented by representing and tracking the blobs through their one-dimensional projections. FIG. 4 shows a candidate image IM that is a representative start frame of a sumo “play” as earlier defined, and thus should be detected by the summarization procedure shown in FIG. 1. Given the stage color description Cs and the skin tone description Ck, the candidate image shown in FIG. 4 may be reduced to the image shown in FIG. 5 where white pixels indicate a place where there is a pixel in the candidate image corresponding to either the stage color Ck or the skin color Cs. The black pixels represent the dark background areas of the candidate image. The image may be further decomposed using skin-tone based segmentation to isolate those portions of the image corresponding to the skin color Cs. A binary image, shown in FIG. 6 may be used to represent the obtained body parts, in which numeral ones represent a pixel of that location representing skin in the original image. This binary image may be projected along vertical and horizontal axes, shown in FIGS. 7 and 8, respectively. The analysis of the blob may be performed on those projections. The proposed projection behaves effectively like an integration process, which makes the algorithm less sensitive to imperfection in the skin/stage segmentation. Note that in these projections, small and isolated peaks have been suppressed.
  • Ideally, a real start frame will result in two peaks of similar size in the vertical projection, nearly symmetric about the center column of the image, as shown in FIGS. 7 and 8, the horizontal projection of the binary image, may be used to check whether the two blobs are symmetric about a center column of the image. In subsequent frames, these two peaks should move closer and closer, eventually converging, as illustrated by FIGS. 10A, 10B, and 10C.
  • The foregoing method relies mainly on color cues, and prior knowledge about the stage color Cs and the skin tone Ck are assumed. However, it is also possible to calibrate the colors for a specific bout or tournament. With other inputs such as a human operator's interactions, the calibration is of course easy to do. Without any human interaction, statistical models can still be used to calibrate the color. If a series of start scene candidates has been obtained, statistical outliers in this set can be detected with prior coarse knowledge about Cs and Ck. The remaining candidate frames can then be used to estimate the specifics of the colors. With the colors calibrated, the start-of-play detection can be performed more accurately.
  • The foregoing method is able to detect start frames successfully in most situations. However, if the detection of a start frame is declared after finding only one candidate frame, then the method may be susceptible to false-positives. By examining a set of consecutive frames (or other temporally related frames) and accumulating evidence, the system can reduce the false-positive rate. Referring to FIG. 10, the following approach may be used to achieve temporal evidence of accumulation: when detecting the start of a “play”, a sliding window of width w is used (e.g., w frames are considered at the same time). A start is declared only if more than p out of the w frames in the current window are determined to be start scene candidates, as previously described. A suitable value of p is such that p/w=70%. Other statistical measures may be used of a fixed number of frames or dynamic number of frames to more accurately determine start scenes.
  • While the start of a “play” may be found according to the aforementioned method, the end of a “play” can occur in a variety of different ways due to the numerous techniques used to either force the opposing contestant to the ground or out of the ring. Image analysis techniques may be used to analyze the image content of the frames after the beginning of a bout to attempt to determine what occurred, but with the nearly endless possibilities and the difficulty of interpreting the content of the frames, this technique is at least, extremely difficult and computationally intensive. In contrast to attempting to analyze the content of the subsequent frames of a potential play, the present inventors determined that a more efficient manner for the determination of the extent of a play in sumo is to base the end of the play on camera activities. After analysis of a sumo video the present inventors were surprised to determine that the approximate end of a play may be modeled by scene changes, normally as a result of switching to a different camera or a different camera angle. The different camera or different camera angle may be modeled by determining the amount of change between the current frame (or set of frames) to the next frame (or set of frames).
  • Referring to FIG. 11, a model of the amount of change between frames using a color histogram difference technique for an exemplary 1,000 frame video sumo clip is shown. The peaks typically correspond to scene cuts. Unfortunately, FIG. 11 demonstrates, some scene cuts, like the one depicted at around frame 325, the camera break produces a relatively low peak in the color histogram difference curve, causing potential failure in scene cut detection.
  • To solve this problem, the inventors have discovered that the use of color histogram differences in conjunction with the sum of absolute pixel-to-pixel differences in the luminance domain is more effective when detecting a scene cut. To gain robustness in using the sum of absolute pixel-to-pixel differences, the luminance images are first down-sampled, or smoothed. FIG. 13 shows the sum of absolute pixel-to-pixel luminance differences for the same video clip as shown in FIG. 11.
  • Even with the aforementioned technique there may be some false detections which do not correspond to a real play. Also, there are situations in which a play is broken into two segments due to for example, dramatic lighting fluctuations (mistaken by the system as a scene cut). Some of these problems can be remedied by post-processing. One example of a suitable post processing technique is if two plays are only separated by a sufficiently short time duration, such as less than a predetermined time period, then they should be connected as a single play. The time period between the two detected plays may be included within the total play, if desired. Even if the two detected plays are separated by a short time period and the system puts the two plays together, and they are in fact two separate plays, this results in an acceptable segment (or two plays) because it avoids frequent audio and visual disruptions in the summary, which may be objectionable to some viewers. Another example of a suitable post processing technique is that if a play has a sufficiently short duration, such as less than 2 seconds, then the system should remove it from being a play because it is likely a false positive. Also, post-processing may be applied to smoothen the connection between adjacent plays, for both video and audio.
  • Sumo video may also include gradual transitions between plays and other activities, such as commentary. These gradual transitions tend to be computationally complex to detect in the general case. However, in the case of sumo it has been determined that detecting gradual transitions based upon the color histogram differences is especially suitable. Other techniques may likewise be used. Referring to FIG. 13, the preferred technique may include starting from a start-of-play time (to) and looking forward until a sufficiently large scene change is detected or until time to+tp is reached, whichever occurs first. Tp relates to the maximum anticipated play duration and therefore automatically sets a maximum duration to the play. This time period for processing to locate gradual transitions is denoted as tclean cut. If tclean cut<tlow then the system will not look for a gradual scene cut and set the previously detected scene cut as the end of the play. This corresponds to an anticipated minimum time duration for a play and tlow is used to denote the minimum time period. Otherwise, the system looks for the highest color histogram difference in the region tlow, tclean cut or other measure of a potential scene change. This region of the segment is from the minimum time duration to the next previously identified scene cut. This identifies the highest color histogram difference in the time duration which may be a potential scene change. The time of the highest color histogram difference is identified at t1. In a neighborhood of t1, [t1−c1, t2+c2], a statistical computation is performed, such as computing the mean ml and the standard deviation F of the color histogram differences. C1 and c2 are constants or statistically calculated temporal values for the region to examine around the highest color histogram difference. A mean filtering emphasizes regions having a relatively large difference in a relatively short time interval. If the color histogram differences at t1 exceeds m1+c3*F1, where c3 is a constant (or otherwise) and some of its neighbors (or otherwise) are sufficiently large, then the system considers a gradual transition to have occurred at around time (frame) t1. The play is set to the shorter of the previously identified scene cut or the gradual transition, if any.
  • The summary obtained by the method described above contains only play segments from the original video. Even though a Sumo fan may be able to quickly recognize the players after they appear, it may help a viewer to follow the game better if we detect those pre-play frames that contains player's names. An example of such type of frames is given in FIG. 14.
  • There are various ways of detecting overlaid graphical text content from an original image or video. In this application, the problem is one of detecting Kanji (Chinese characters used in Japanese) in images. With sufficient sample data, the system may train a convolution neural network to perform this task. In Sumo broadcasting there are a few special patterns that are typically adopted in presenting the graphical characters. For example, the names of the two players are the biggest characters. Also, it appears that the names normally appear in white (or substantial contrast to the background). This is probably due to the fact that the names are usually overlaid on a dark scene of the sumo stadium. In addition the graphical information is symmetric with respect to the center column, with one player's information on the left, and the other player's information on the right. The characters read vertically from top to bottom.
  • These special patterns can be utilized to facilitate a neural network based character detection module. The system may include an algorithm to find frames with these patterns. The present inventors have found that the following set of rules may successfully detect frames with the desired player names in a video: (1) the frame has white blocks that are nearly symmetrically distributed about the center column of the image; (2) except for these white blocks, there should be no other white areas of significant size in the frame; (3) these white blocks persist for at least a few seconds; and (4) the set of frames with persistent white blocks proceeds to the start of a play. One or more of these rules may be included, as desired.
  • After the frames with the player names are detected, the system may add them to their respective plays and obtain a new summary. Unlike the baseline summary obtained before, in this new summary, there are a few seconds of video like that in FIG. 14 for introducing each play. Thus the new summary is easier to follow.
  • If desired, a slow motion replay detection module may be incorporated. The system detects if a slow motion replay has occurred, which normally relates to important events. The system will capture the replays of plays, the same as the typical non-slow motion replay (full speed), if the same type of camera angles are used. The play segments detected may be identified with multiple characteristics, namely, slow motion replay-only segments, play only segments without slow motion replay segments, and slow motion replay that include associated full speed segments. The resulting summary may include one or more of the different selections of the aforementioned options, as desired. For example, the resulting summary may have the slow-motion replays removed. These options may likewise be user selectable.
  • While an effective summarization of a sumo video may be based on the concept of the “play”, sometimes the viewer may prefer an even shorter summarization with the most exciting plays included. One potential technique for the estimation of the excitement of a play is to perform statistical analysis on the segments to determine which durations are most likely to have the highest excitement. However, this technique will likely not provide sufficiently accurate results. Further, excitement tends to be a subjective measure that is hard to quantify. After further consideration the present inventors came to the realization that the audio provided together with the video provides a good indication of the excitement of the plays. For example, the volume of the response of the audience and/or the commentators provides a good indication of the excitement. The louder audience and/or commentator acclamations, the greater the degree of excitement.
  • Referring to FIGS. 15A-15C, an exemplary illustration is shown of audio signals having a relatively quiet response (FIG. 15A), having a strong response (FIG. 15B), and having an extremely strong response (FIG. 15C). In general, it has been determined that more exciting plays have the following audio features. First, the mean audio volume of the play is large. The mean audio volume may be computed by defining the mean volume of a play as v = 1 N i = 0 N - 1 S 2 ( i )
    where S(i) is the i-th sample, and the N is the total number of samples in the play. Second, the play contains more audio samples that have middle-ranged magnitudes. The second feature may be reflected by the percentage of the middle-range-magnituded samples in the play, which may be computed as P = 1 N i = 0 N - 1 I ( _s ( i ) _ > t1 and_s ( i ) _ < t2 )
    with I( ) being the indicator function (I(true)=1, and I(false)=0), t1 and t2 are two thresholds defining the middle range.
  • Referring to FIG. 16, the first layer of the summary is constructed using the play detection technique. The second and third layers (and other) are extracted as being of increasingly greater excitement, based at least in part, on the audio levels of the respective audio of the video segments. Also, it would be noted that the preferred audio technique only uses the temporal domain, which results in a computationally efficient technique. In addition, the level of the audio may be used as a basis for the modification of the duration of a particular play segment. For example, if a particular play segment has a high audio level then the boundaries of the play segment may be extended. This permits a greater emphasis to be placed on those segments more likely to be exciting. For example, if a particular play segment has a low audio level then the boundaries of the play segment may be contracted. This permits a reduced emphasis to be placed on those segments less likely to be exciting. It is to be understood that the layered summarization may be based upon other factors, as desired.
  • Referring to FIG. 17, the video summarization may be included as part of an MPEG-7 based browser/filter, where summarization is included within the standard. The media summarizer may be as shown in FIG. 1. With different levels of summarization built on top of the aforementioned video summarization technique, the system can provide the user with varying levels of summaries according to their demands. Once the summary information is described as an MPEG-7 compliant XML document, one can utilize all the offerings of MPEG-7, such as personalization, where different levels of summaries can be offered to the user on the basis of user's preferences described in an MPEG-7 compliant way. Descriptions of user preferences in MPEG-7 include preference elements pertaining to different summary modes and detail levels.
  • In the case that the summarization is performed at a server or service provider, the user downloads and receives the summary description encoded in MPEG-7 format. Alternatively, in an interactive video on demand (VOD) application, the media and its summary description reside at the provider's VOD server and the user (e.g., remote) consumes the summary via a user-side browser interface. In this case, the summary may be enriched further by additional information that may be added by the service provider. Further, summarization may also be performed by the client.
  • Referring to FIG. 18, the output of the module that automatically detects important segments may be a set of indices of segments containing plays and important parts of the input video program. A description document, such as an MPEG-7 or TV-Anytime compliant description is generated in The Description Generation module. Summary segments are made available to the Post-Processing module by The Extraction of Summary Segments module which processes the input video program according to the description. A post-processing module processes the summary Segments and/or the description to generate the final summary video and final description. The post-processing module puts the post-processed segments together to form the final summary video. The post-processing module may transcode the resulting video to a format different that of the input video to meet the requirements of the storage/transmission channel. The final description may also be encoded, e.g., binarized if it is generated originally in textual format such as XML. Post-processing may include adding to the original audio track a commentary, insertion of advertisement segments, or metadata. In contrast to play detection, post-processing may be completely, or in part, manual processing. It may include, for example, automatic ranking and subset selection of events on the basis of automatic detection of features in the audio track associated with video segments. This processing may be performed at the server and then the resulting video transferred to the client, normally over a network. Alternatively, the resulting video is included in a VOD library and made available to users on a VOD server.
  • Referring to FIG. 19, a system may be developed that incorporates start detection of a play, end detection of a play, and summarization. The detection technique may be based upon processing a single frame, multiple frames, or a combination thereof.
  • The terms and expressions which have been employed in the foregoing specification are used therein as terms of description and not of limitation, and there is no intention, in the use of such terms and expressions, of excluding equivalents of the features shown and described or portions thereof, it being recognized that the scope of the invention is defined and limited only by the claims which follow.

Claims (36)

1-17. (canceled)
18. A method of processing a video including sumo comprising:
(a) identifying a plurality of segments of said video, wherein the start of said plurality of segments is identified based upon a frame of said video having an upper spatial region being substantially darker than a lower spatial region of said frame, where each of said segments includes a plurality of frames of said video; and
(b) creating a summarization of said video by including said plurality of segments, where said summarization includes fewer frames than said video.
19. The method of claim 18 wherein said lower spatial region comprises, at least in part, a pair of regions having a dominant color description representative of skin tone.
20. The method of claim 19 further comprising said lower spatial region comprises, at least in part, a pair of regions having a dominant color description representative of stage color.
21. The method of claim 19 further comprising said lower spatial region comprises, at least in part, a pair of regions having a dominant color description representative of stage color.
22. A method of processing a video including sumo comprising:
(a) identifying a plurality of segments of said video, wherein the start of said plurality of segments is identified based upon a pair of regions having a dominant color description representative of skin tone, where each of said segments includes a plurality of frames of said video; and
(b) creating a summarization of said video by including said plurality of segments, where said summarization includes fewer frames than said video.
23. The method of claim 22 wherein said dominant color description includes 25 percent of said pair of regions.
24. The method of claim 22 wherein said dominant color description includes 50 percent of said pair of regions.
25. The method of claim 22 wherein said dominant color description includes 75 percent of said pair of regions.
26. The method of claim 22 wherein said pair of regions is in the lower portion of said video.
27. A method of processing a video including sumo comprising:
(a) identifying a plurality of segments of said video, wherein the start of said plurality of segments is identified based upon a pair of regions generally symmetric to each other with respect to a generally center column of a frame of said video, where each of said segments includes a plurality of frames of said video; and
(b) creating a summarization of said video by including said plurality of segments, where said summarization includes fewer frames than said video.
28. The method of claim 27 wherein said pair of spatial regions have a dominant color description representative of skin tone.
29. The method of claim 27 wherein said center column is within 20 percent of the center of said frame.
30. The method of claim 29 wherein said center column is the center of said frame.
31. A method of processing a video including sumo comprising:
(a) identifying a plurality of segments of said video, wherein the start of said plurality of segments is identified based upon a pair of spatial regions that move toward one another, where each of said segments includes a plurality of frames of said video; and
(b) creating a summarization of said video by including said plurality of segments, where said summarization includes fewer frames than said video.
32. The method of claim 31 wherein said pair of spatial regions have a dominant color description representative of skin tone.
33. The method of claim 31 wherein said pair of spatial regions collide with one another.
34. A method of processing a video including sumo comprising:
(a) identifying a plurality of segments of said video,
(i) wherein the start of said plurality of segments is identified based upon a frame of said video having an upper spatial region being substantially darker than a lower spatial region of said frame,
(ii) wherein said lower spatial region comprises, at least in part, a pair of regions having a dominant color description representative of skin tone,
(iii) wherein said lower spatial region comprises, at least in part, said pair of regions having a dominant color description representative of stage color;
(iv) wherein said pair of regions are generally symmetric to each other with respect to a generally center column of a frame of said video;
(v) wherein said pair of regions move toward one another;
(vi) where each of said segments includes a plurality of frames of said video; and
(b) creating a summarization of said video by including said plurality of segments, where said summarization includes fewer frames than said video.
35. A method of processing a video including sumo comprising:
(a) identifying a plurality of segments of said video, wherein said identifying for at least one of said segments includes detecting the start of said segment based upon processing of a first single frame of said video, where each of said segments includes a plurality of frames of said video;
(b) verifying that said first single frame is an appropriate start of said segment based upon processing of another single frame temporally relevant to said first single frame; and
(c) creating a summarization of said video by including said plurality of segments, where said summarization includes fewer frames than said video.
36. A method of processing a video including sumo comprising:
(a) identifying a plurality of segments of said sumo video, wherein said identifying for the end of at least one of said segments is based upon detecting a scene change, where each of said segments includes a plurality of frames of said sumo video; and
(b) creating a summarization of said sumo video by including said plurality of segments, where said summarization includes fewer frames than said sumo video.
37. The method of claim 36 wherein said scene change is based upon a threshold between at least two frames.
38. The method of claim 36 wherein said scene change is based upon a gradual transition below a threshold level.
39. A method of processing a video including sumo comprising:
(a) identifying a plurality of segments of said video, where each of said segments includes a plurality of frames of said video;
(b) identifying a plurality of segments that are temporally separated by a sufficiently short duration;
(c) based upon said identifying as a result of (b) connecting said identified plurality of segments; and
(d) creating a summarization of said video by including said plurality of segments, where said summarization includes fewer frames than said video.
40. The method of claim 39 wherein said connecting includes discarding the frames of said video between said identified plurality of segments.
41. The method of claim 39 wherein said connecting results in a single segment that includes said identified plurality of segments together with the frames of said video between said identified plurality of segments.
42. A method of processing a video including sumo comprising:
(a) identifying a plurality of segments of said video, where each of said segments includes a plurality of frames of said video;
(b) identifying at least one of said segments that has a temporally sufficiently short duration;
(c) based upon said identifying as a result of (b) removing said identified segment from said summarization; and
(d) creating a summarization of said video by including said plurality of segments, where said summarization includes fewer frames than said video.
43. The method of claim 42 wherein said connecting includes discarding the frames of said video between said identified plurality of segments.
44. The method of claim 42 wherein said connecting results in a single segment that includes said identified plurality of segments together with the frames of said video between said identified plurality of segments.
45. A method of processing a video including sumo comprising:
(a) identifying a plurality of segments of said video wherein each of said segments includes a play of sumo, wherein said segments include full-speed plays and slow motion plays of said full-speed plays; and
(b) creating a summarization of said video by including said plurality of segments, where said summarization includes fewer frames than said video, where a user may select from:
(i) said summarization including only full-speed plays;
(ii) said summarization including only slow motion plays;
(iii) said summarization including both full-speed plays and slow motion plays.
46. A method of processing a video including sumo comprising:
(a) identifying a plurality of segments of said video wherein each of said segments includes a play of sumo;
(b) creating a summarization of said video by including said plurality of segments, where said summarization includes fewer frames than said video; and
(c) removing at least one of said segments from said summary based, at least in part, upon audio information related to said at least one of said segments.
47. The method of claim 46 wherein said audio information is obtained exclusively from a temporal analysis.
48. A method of processing a video including sumo comprising:
(a) identifying a plurality of segments of said video wherein each of said segments includes a play of sumo;
(b) creating a summarization of said video by including said plurality of segments, where said summarization includes fewer frames than said video; and
(c) modifying the duration of at least one of said segments from said summary based, at least in part, upon audio information related to said at least one of said segments.
49. The method of claim 48 wherein said audio information is obtained exclusively from a temporal analysis.
50-60. (canceled)
61. A method of processing a video including sumo comprising:
(a) identifying a plurality of segments of said video, wherein the detection of graphical text segments is identified based upon:
(i) a pair of substantially white regions generally symmetric with respect to the center of the image,
(ii) said image free from other significant substantially white areas;
(iii) said white regions persist for a plurality of seconds;
(iv) said white regions preceding the start of a play;
(v) where each of said segments includes a plurality of frames of said video; and
(b) creating a summarization of said video by including said plurality of segments, where said summarization includes fewer frames than said video.
62-64. (canceled)
US11/075,284 2002-01-28 2005-03-08 Summarization of sumo video content Abandoned US20050155055A1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US11/075,284 US20050155055A1 (en) 2002-01-28 2005-03-08 Summarization of sumo video content

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US10/058,684 US7120873B2 (en) 2002-01-28 2002-01-28 Summarization of sumo video content
US11/075,284 US20050155055A1 (en) 2002-01-28 2005-03-08 Summarization of sumo video content

Related Parent Applications (1)

Application Number Title Priority Date Filing Date
US10/058,684 Division US7120873B2 (en) 2002-01-28 2002-01-28 Summarization of sumo video content

Publications (1)

Publication Number Publication Date
US20050155055A1 true US20050155055A1 (en) 2005-07-14

Family

ID=27609651

Family Applications (4)

Application Number Title Priority Date Filing Date
US10/058,684 Expired - Fee Related US7120873B2 (en) 2002-01-28 2002-01-28 Summarization of sumo video content
US11/075,284 Abandoned US20050155055A1 (en) 2002-01-28 2005-03-08 Summarization of sumo video content
US11/074,860 Expired - Fee Related US8028234B2 (en) 2002-01-28 2005-03-08 Summarization of sumo video content
US11/075,283 Abandoned US20050155054A1 (en) 2002-01-28 2005-03-08 Summarization of sumo video content

Family Applications Before (1)

Application Number Title Priority Date Filing Date
US10/058,684 Expired - Fee Related US7120873B2 (en) 2002-01-28 2002-01-28 Summarization of sumo video content

Family Applications After (2)

Application Number Title Priority Date Filing Date
US11/074,860 Expired - Fee Related US8028234B2 (en) 2002-01-28 2005-03-08 Summarization of sumo video content
US11/075,283 Abandoned US20050155054A1 (en) 2002-01-28 2005-03-08 Summarization of sumo video content

Country Status (1)

Country Link
US (4) US7120873B2 (en)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8392183B2 (en) 2006-04-25 2013-03-05 Frank Elmo Weber Character-based automated media summarization
US20130121575A1 (en) * 2011-11-11 2013-05-16 Seoul National University R&B Foundation Image analysis apparatus using main color and method of controlling the same
WO2016160304A1 (en) * 2015-03-31 2016-10-06 Zepp Labs, Inc. Detect sports video highlights for mobile computing devices

Families Citing this family (35)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11109114B2 (en) * 2001-04-18 2021-08-31 Grass Valley Canada Advertisement management method, system, and computer program product
US9123380B2 (en) 1998-12-18 2015-09-01 Gvbb Holdings S.A.R.L. Systems, methods, and computer program products for automated real-time execution of live inserts of repurposed stored content distribution, and multiple aspect ratio automated simulcast production
US6993182B2 (en) * 2002-03-29 2006-01-31 Koninklijke Philips Electronics N.V. Method and apparatus for detecting scene changes in video using a histogram of frame differences
US20040073437A1 (en) * 2002-10-15 2004-04-15 Halgas Joseph F. Methods and systems for providing enhanced access to televised sporting events
US7664292B2 (en) * 2003-12-03 2010-02-16 Safehouse International, Inc. Monitoring an output from a camera
US7697026B2 (en) * 2004-03-16 2010-04-13 3Vr Security, Inc. Pipeline architecture for analyzing multiple video streams
US7676590B2 (en) * 2004-05-03 2010-03-09 Microsoft Corporation Background transcoding
GB2421135A (en) 2004-12-09 2006-06-14 Sony Uk Ltd User resizable video display of catalogue entries
US7669121B2 (en) * 2005-01-19 2010-02-23 Microsoft Corporation Transcode matrix
US7738766B2 (en) * 2005-04-18 2010-06-15 Microsoft Corporation Sanctioned transcoding of digital-media content
US7558463B2 (en) * 2005-04-18 2009-07-07 Microsoft Corporation Retention of information about digital-media rights in transformed digital media content
US9363481B2 (en) 2005-04-22 2016-06-07 Microsoft Technology Licensing, Llc Protected media pipeline
US9436804B2 (en) 2005-04-22 2016-09-06 Microsoft Technology Licensing, Llc Establishing a unique session key using a hardware functionality scan
US8553151B2 (en) * 2005-09-02 2013-10-08 Gvbb Holdings S.A.R.L. Video effect recall technique
US7924913B2 (en) * 2005-09-15 2011-04-12 Microsoft Corporation Non-realtime data transcoding of multimedia content
KR100803747B1 (en) * 2006-08-23 2008-02-15 삼성전자주식회사 System for creating summery clip and method of creating summary clip using the same
JP4297141B2 (en) * 2006-09-11 2009-07-15 ソニー株式会社 Information processing apparatus and method, and program
US20090133060A1 (en) * 2007-11-21 2009-05-21 Microsoft Corporation Still-Frame Content Navigation
US9407942B2 (en) * 2008-10-03 2016-08-02 Finitiv Corporation System and method for indexing and annotation of video content
CN102073635B (en) * 2009-10-30 2015-08-26 索尼株式会社 Program endpoint time detection apparatus and method and programme information searching system
JP5536491B2 (en) * 2010-03-01 2014-07-02 ダンロップスポーツ株式会社 Golf swing diagnosis method
US20120027371A1 (en) * 2010-07-28 2012-02-02 Harris Corporation Video summarization using video frames from different perspectives
US9171578B2 (en) * 2010-08-06 2015-10-27 Futurewei Technologies, Inc. Video skimming methods and systems
CN106127796B (en) 2012-03-07 2019-03-26 奥林巴斯株式会社 Image processing apparatus and image processing method
EP2823749A4 (en) * 2012-03-08 2016-07-20 Olympus Corp Image processing device, program, and image processing method
WO2013157354A1 (en) 2012-04-18 2013-10-24 オリンパス株式会社 Image processing device, program, and image processing method
US10091552B2 (en) * 2012-09-19 2018-10-02 Rovi Guides, Inc. Methods and systems for selecting optimized viewing portions
US9351588B2 (en) 2012-11-29 2016-05-31 Kids Ii, Inc. Child support unit for a play yard
KR102217186B1 (en) * 2014-04-11 2021-02-19 삼성전자주식회사 Broadcasting receiving apparatus and method for providing summary contents service
US9386358B2 (en) * 2014-05-29 2016-07-05 Echostar Technologies L.L.C. Automatic identification of relevant video content through replays
US20150370907A1 (en) * 2014-06-19 2015-12-24 BrightSky Labs, Inc. Systems and methods for intelligent filter application
CN106130839B (en) * 2016-07-12 2019-03-01 电子科技大学 A kind of business recognition method applied to broadband access network
US10679669B2 (en) * 2017-01-18 2020-06-09 Microsoft Technology Licensing, Llc Automatic narration of signal segment
US10445586B2 (en) 2017-12-12 2019-10-15 Microsoft Technology Licensing, Llc Deep learning on image frames to generate a summary
US10403326B1 (en) 2018-01-18 2019-09-03 Gopro, Inc. Systems and methods for detecting moments within videos

Citations (83)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4183056A (en) * 1977-05-23 1980-01-08 Kewp Electronic Systems, Inc. Apparatus and method for monitoring sports contests
US4253108A (en) * 1979-06-04 1981-02-24 Zenith Radio Corporation Control for color killer and automatic color limiter
US4321635A (en) * 1979-04-20 1982-03-23 Teac Corporation Apparatus for selective retrieval of information streams or items
US4520404A (en) * 1982-08-23 1985-05-28 Kohorn H Von System, apparatus and method for recording and editing broadcast transmissions
US4729044A (en) * 1985-02-05 1988-03-01 Lex Computing & Management Corporation Method and apparatus for playing serially stored segments in an arbitrary sequence
US4937685A (en) * 1983-12-02 1990-06-26 Lex Computer And Management Corporation Method of display presentation for video editing
US5027400A (en) * 1988-08-19 1991-06-25 Hitachi Ltd. Multimedia bidirectional broadcast system
US5109482A (en) * 1989-01-11 1992-04-28 David Bohrman Interactive video control system for displaying user-selectable clips
US5200825A (en) * 1992-07-01 1993-04-06 Beam Laser Systems, Inc. Commercial insertion system remotely controlling multiple video switches
USD348251S (en) * 1992-12-09 1994-06-28 Discovery Communications, Inc. Menu control panel for a universal remote control unit
US5333393A (en) * 1993-06-11 1994-08-02 Tyrone Hill Shoe retaining apparatus and method
USD354059S (en) * 1992-12-03 1995-01-03 Discovery Communications, Inc. Remote control unit
US5424770A (en) * 1993-04-16 1995-06-13 Cable Service Technologies, Inc. Method and apparatus for automatic insertion of a television signal from a remote source
US5434678A (en) * 1993-01-11 1995-07-18 Abecassis; Max Seamless transmission of non-sequential video segments
USD368263S (en) * 1994-07-12 1996-03-26 Discovery Communications, Inc. Remote control unit
US5521841A (en) * 1994-03-31 1996-05-28 Siemens Corporate Research, Inc. Browsing contents of a given video sequence
US5600573A (en) * 1992-12-09 1997-02-04 Discovery Communications, Inc. Operations center with video storage for a television program packaging and delivery system
US5600364A (en) * 1992-12-09 1997-02-04 Discovery Communications, Inc. Network controller for cable television delivery systems
US5610653A (en) * 1992-02-07 1997-03-11 Abecassis; Max Method and system for automatically tracking a zoomed video image
US5635982A (en) * 1994-06-27 1997-06-03 Zhang; Hong J. System for automatic video segmentation and key frame extraction for video sequences having both sharp and gradual transitions
US5654769A (en) * 1992-03-11 1997-08-05 Texas Instruments Incorporated Digital color control and chroma killer device
USD381991S (en) * 1994-07-12 1997-08-05 Discovery Communications, Inc. Remote control unit
US5659350A (en) * 1992-12-09 1997-08-19 Discovery Communications, Inc. Operations center for a television program packaging and delivery system
US5710884A (en) * 1995-03-29 1998-01-20 Intel Corporation System for automatically updating personal profile server with updates to additional user information gathered from monitoring user's electronic consuming habits generated on computer during use
US5717814A (en) * 1992-02-07 1998-02-10 Max Abecassis Variable-content video retriever
US5734853A (en) * 1992-12-09 1998-03-31 Discovery Communications, Inc. Set top terminal for cable television delivery systems
US5761881A (en) * 1995-05-10 1998-06-09 Wall; Benjamin Process and apparatus for wrapping paper rolls
US5774357A (en) * 1991-12-23 1998-06-30 Hoffberg; Steven M. Human factored interface incorporating adaptive pattern recognition based controller apparatus
US5778108A (en) * 1996-06-07 1998-07-07 Electronic Data Systems Corporation Method and system for detecting transitional markers such as uniform fields in a video signal
US5861881A (en) * 1991-11-25 1999-01-19 Actv, Inc. Interactive computer system for providing an interactive presentation with personalized video, audio and graphics responses for multiple viewers
US5875108A (en) * 1991-12-23 1999-02-23 Hoffberg; Steven M. Ergonomic man-machine interface incorporating adaptive pattern recognition based control system
US5875107A (en) * 1996-12-05 1999-02-23 Mitsubishi Denki Kabushiki Kaisha Inverter apparatus
US5892536A (en) * 1996-10-03 1999-04-06 Personal Audio Systems and methods for computer enhanced broadcast monitoring
US5901246A (en) * 1995-06-06 1999-05-04 Hoffberg; Steven M. Ergonomic man-machine interface incorporating adaptive pattern recognition based control system
US5900867A (en) * 1995-07-17 1999-05-04 Gateway 2000, Inc. Self identifying remote control device having a television receiver for use in a computer
US5913013A (en) * 1993-01-11 1999-06-15 Abecassis; Max Seamless transmission of non-sequential video segments
US5920360A (en) * 1996-06-07 1999-07-06 Electronic Data Systems Corporation Method and system for detecting fade transitions in a video signal
US5923365A (en) * 1993-10-12 1999-07-13 Orad Hi-Tech Systems, Ltd Sports event video manipulating system for highlighting movement
US5926624A (en) * 1996-09-12 1999-07-20 Audible, Inc. Digital information library and delivery system with logic for generating files targeted to the playback device
US6014183A (en) * 1997-08-06 2000-01-11 Imagine Products, Inc. Method and apparatus for detecting scene changes in a digital video stream
US6038367A (en) * 1992-02-07 2000-03-14 Abecassis; Max Playing a Video Responsive to a comparison of two sets of Content Preferences
US6055018A (en) * 1997-11-04 2000-04-25 Ati Technologies, Inc. System and method for reconstructing noninterlaced captured content for display on a progressive screen
US6081750A (en) * 1991-12-23 2000-06-27 Hoffberg; Steven Mark Ergonomic man-machine interface incorporating adaptive pattern recognition based control system
US6088455A (en) * 1997-01-07 2000-07-11 Logan; James D. Methods and apparatus for selectively reproducing segments of broadcast programming
US6169542B1 (en) * 1998-12-14 2001-01-02 Gte Main Street Incorporated Method of delivering advertising through an interactive video distribution system
US6181335B1 (en) * 1992-12-09 2001-01-30 Discovery Communications, Inc. Card for a set top terminal
US6195497B1 (en) * 1993-10-25 2001-02-27 Hitachi, Ltd. Associated image retrieving apparatus and method
US6201536B1 (en) * 1992-12-09 2001-03-13 Discovery Communications, Inc. Network manager for cable television system headends
US6215526B1 (en) * 1998-11-06 2001-04-10 Tivo, Inc. Analog video tagging and encoding system
US6216129B1 (en) * 1998-12-03 2001-04-10 Expanse Networks, Inc. Advertisement selection system supporting discretionary target market characteristics
US6219837B1 (en) * 1997-10-23 2001-04-17 International Business Machines Corporation Summary frames in video
US6233389B1 (en) * 1998-07-30 2001-05-15 Tivo, Inc. Multimedia time warping system
US6230501B1 (en) * 1994-04-14 2001-05-15 Promxd Technology, Inc. Ergonomic systems and methods providing intelligent adaptive surfaces and temperature control
US6236395B1 (en) * 1999-02-01 2001-05-22 Sharp Laboratories Of America, Inc. Audiovisual information management system
US6252544B1 (en) * 1998-01-27 2001-06-26 Steven M. Hoffberg Mobile communication device
US6342904B1 (en) * 1998-12-17 2002-01-29 Newstakes, Inc. Creating a slide presentation from full motion video
US20020013943A1 (en) * 2000-04-07 2002-01-31 Seth Haberman System and method for simultaneous broadcast for personalized messages
US20020018594A1 (en) * 2000-07-06 2002-02-14 Mitsubishi Electric Research Laboratories, Inc. Method and system for high-level structure analysis and event detection in domain specific videos
US6363160B1 (en) * 1999-01-22 2002-03-26 Intel Corporation Interface using pattern recognition and tracking
US20020083473A1 (en) * 2000-12-21 2002-06-27 Philips Electronics North America Corporation System and method for accessing a multimedia summary of a video program
US20020080162A1 (en) * 2000-11-02 2002-06-27 Hao Pan Method for automatic extraction of semantically significant events from video
US6418168B1 (en) * 1999-03-24 2002-07-09 Sony Corporation Motion vector detection apparatus, method of the same, and image processing apparatus
US20030001880A1 (en) * 2001-04-18 2003-01-02 Parkervision, Inc. Method, system, and computer program product for producing and distributing enhanced media
US20030026592A1 (en) * 2000-12-28 2003-02-06 Minoru Kawahara Content creating device and method
US6549643B1 (en) * 1999-11-30 2003-04-15 Siemens Corporate Research, Inc. System and method for selecting key-frames of video data
US6556767B2 (en) * 1995-12-15 2003-04-29 Hitachi Ltd Video capture device
US20030081937A1 (en) * 2001-07-03 2003-05-01 Baoxin Li Summarization of video content
US6570608B1 (en) * 1998-09-30 2003-05-27 Texas Instruments Incorporated System and method for detecting interactions of people and vehicles
US6585521B1 (en) * 2001-12-21 2003-07-01 Hewlett-Packard Development Company, L.P. Video indexing based on viewers' behavior and emotion feedback
US20030133511A1 (en) * 2002-01-15 2003-07-17 Romain Cabasson Summarizing videos using motion activity descriptors correlated with audio features
US6597859B1 (en) * 1999-12-16 2003-07-22 Intel Corporation Method and apparatus for abstracting video data
US6678635B2 (en) * 2001-01-23 2004-01-13 Intel Corporation Method and system for detecting semantic events
US20040017389A1 (en) * 2002-07-25 2004-01-29 Hao Pan Summarization of soccer video content
US6724933B1 (en) * 2000-07-28 2004-04-20 Microsoft Corporation Media segmentation system and related methods
US20040088289A1 (en) * 2001-03-29 2004-05-06 Li-Qun Xu Image processing
US20040125124A1 (en) * 2000-07-24 2004-07-01 Hyeokman Kim Techniques for constructing and browsing a hierarchical video structure
US20040125877A1 (en) * 2000-07-17 2004-07-01 Shin-Fu Chang Method and system for indexing and content-based adaptive streaming of digital video content
US6870956B2 (en) * 2001-06-14 2005-03-22 Microsoft Corporation Method and apparatus for shot detection
US6993245B1 (en) * 1999-11-18 2006-01-31 Vulcan Patents Llc Iterative, maximally probable, batch-mode commercial detection for audiovisual content
US7003154B1 (en) * 2000-11-17 2006-02-21 Mitsubishi Electric Research Laboratories, Inc. Adaptively processing a video based on content characteristics of frames in a video
US20060098889A1 (en) * 2000-08-18 2006-05-11 Jiebo Luo Digital image processing system and method for emphasizing a main subject of an image
US7181757B1 (en) * 1999-10-11 2007-02-20 Electronics And Telecommunications Research Institute Video summary description scheme and method and system of video summary description data generation for efficient overview and browsing
US7221780B1 (en) * 2000-06-02 2007-05-22 Sony Corporation System and method for human face detection in color graphics images

Family Cites Families (62)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US625254A (en) * 1899-05-16 Joseph g
US4298884A (en) 1980-03-31 1981-11-03 Zenith Radio Corporation Chroma amplifier and color killer
US5101364A (en) * 1990-02-09 1992-03-31 Massachusetts Institute Of Technology Method and facility for dynamic video composition and viewing
US5148154A (en) 1990-12-04 1992-09-15 Sony Corporation Of America Multi-dimensional user interface
US5684918A (en) 1992-02-07 1997-11-04 Abecassis; Max System for integrating video and communications
US5371551A (en) 1992-10-29 1994-12-06 Logan; James Time delayed digital video system using concurrent recording and playback
US5986690A (en) 1992-12-09 1999-11-16 Discovery Communications, Inc. Electronic book selection and delivery system
US5798785A (en) * 1992-12-09 1998-08-25 Discovery Communications, Inc. Terminal for suggesting programs offered on a television program delivery system
US5333091B2 (en) * 1993-01-08 1996-12-17 Arthur D Little Enterprises Method and apparatus for controlling a videotape player to automatically scan past recorded commercial messages
US5339393A (en) * 1993-04-15 1994-08-16 Sony Electronics, Inc. Graphical user interface for displaying available source material for editing
US6275268B1 (en) 1993-09-09 2001-08-14 United Video Properties, Inc. Electronic television program guide with remote product ordering
US5675752A (en) 1994-09-15 1997-10-07 Sony Corporation Interactive applications generator for an interactive presentation environment
US5664227A (en) 1994-10-14 1997-09-02 Carnegie Mellon University System and method for skimming digital audio/video data
USD402310S (en) 1994-11-07 1998-12-08 Discovery Communications, Inc. Electronic book
US5617565A (en) * 1994-11-29 1997-04-01 Hitachi America, Ltd. Broadcast interactive multimedia system
US5805733A (en) 1994-12-12 1998-09-08 Apple Computer, Inc. Method and system for detecting scenes and summarizing video sequences
US5821945A (en) 1995-02-03 1998-10-13 The Trustees Of Princeton University Method and apparatus for video browsing based on content and structure
US5958006A (en) 1995-11-13 1999-09-28 Motorola, Inc. Method and apparatus for communicating summarized data
KR0183149B1 (en) 1995-12-30 1999-05-01 김광호 Motion image detection method
US5969755A (en) 1996-02-05 1999-10-19 Texas Instruments Incorporated Motion based event detection system and method
US5959697A (en) 1996-06-07 1999-09-28 Electronic Data Systems Corporation Method and system for detecting dissolve transitions in a video signal
US5933811A (en) 1996-08-20 1999-08-03 Paul D. Angles System and method for delivering customized advertisements within interactive communication systems
US5986692A (en) 1996-10-03 1999-11-16 Logan; James D. Systems and methods for computer enhanced broadcast monitoring
US5917553A (en) 1996-10-22 1999-06-29 Fox Sports Productions Inc. Method and apparatus for enhancing the broadcast of a live event
US5931901A (en) 1996-12-09 1999-08-03 Robert L. Wolfe Programmed music on demand from the internet
US6496228B1 (en) * 1997-06-02 2002-12-17 Koninklijke Philips Electronics N.V. Significant scene detection and frame filtering for a visual indexing system using dynamic thresholds
US5881881A (en) * 1997-06-16 1999-03-16 Carrington; Thomas Evacuateable bag
US5973683A (en) 1997-11-24 1999-10-26 International Business Machines Corporation Dynamic regulation of television viewing content based on viewer profile and viewing history
US5956026A (en) 1997-12-19 1999-09-21 Sharp Laboratories Of America, Inc. Method for hierarchical summarization and browsing of digital video
US5990980A (en) 1997-12-23 1999-11-23 Sarnoff Corporation Detection of transitions in video sequences
US6304665B1 (en) 1998-04-03 2001-10-16 Sportvision, Inc. System for determining the end of a path for a moving object
JP3907839B2 (en) 1998-06-17 2007-04-18 松下電器産業株式会社 Broadcast system
US6141041A (en) 1998-06-22 2000-10-31 Lucent Technologies Inc. Method and apparatus for determination and visualization of player field coverage in a sporting event
JP4198786B2 (en) * 1998-06-30 2008-12-17 株式会社東芝 Information filtering system, information filtering apparatus, video equipment, and information filtering method
US6100941A (en) 1998-07-28 2000-08-08 U.S. Philips Corporation Apparatus and method for locating a commercial disposed within a video data stream
US6144375A (en) 1998-08-14 2000-11-07 Praja Inc. Multi-perspective viewer for content-based interactivity
US6408128B1 (en) 1998-11-12 2002-06-18 Max Abecassis Replaying with supplementary information a segment of a video
US6492998B1 (en) 1998-12-05 2002-12-10 Lg Electronics Inc. Contents-based video story browsing system
US6774917B1 (en) 1999-03-11 2004-08-10 Fuji Xerox Co., Ltd. Methods and apparatuses for interactive similarity searching, retrieval, and browsing of video
US7072398B2 (en) 2000-12-06 2006-07-04 Kai-Kuang Ma System and method for motion vector generation and analysis of digital video clips
US6665423B1 (en) 2000-01-27 2003-12-16 Eastman Kodak Company Method and system for object-oriented motion-based video description
WO2001078396A1 (en) * 2000-04-07 2001-10-18 Stockhouse Media Corporation Customized multimedia content method, apparatus, media and signals
US6970510B1 (en) 2000-04-25 2005-11-29 Wee Susie J Method for downstream editing of compressed video
US6829781B1 (en) 2000-05-24 2004-12-07 At&T Corp. Network-based service to provide on-demand video summaries of television programs
US6691126B1 (en) 2000-06-14 2004-02-10 International Business Machines Corporation Method and apparatus for locating multi-region objects in an image or video database
KR20020030809A (en) * 2000-07-10 2002-04-25 요트.게.아. 롤페즈 Device for and method of presenting program information
US20020051077A1 (en) * 2000-07-19 2002-05-02 Shih-Ping Liou Videoabstracts: a system for generating video summaries
US6685423B1 (en) * 2000-09-25 2004-02-03 Starcon International, Inc. Method and apparatus for extracting and installing heat exchanger bundles
US6774908B2 (en) 2000-10-03 2004-08-10 Creative Frontier Inc. System and method for tracking an object in a video and linking information thereto
US6981129B1 (en) 2000-11-02 2005-12-27 Intel Corporation Breaking replay dependency loops in a processor using a rescheduled replay queue
US20020083471A1 (en) * 2000-12-21 2002-06-27 Philips Electronics North America Corporation System and method for providing a multimedia summary of a video program
US20020081062A1 (en) * 2000-12-22 2002-06-27 Jian-Jun He Optical grating based multi-input demultiplexer for multiple sets of interleaved wavelength channels
US6978420B2 (en) * 2001-02-12 2005-12-20 Aplix Research, Inc. Hierarchical document cross-reference system and method
US20020120929A1 (en) 2001-02-28 2002-08-29 Schwalb Eddie M. Method and system for mass customization of digital television broadcasts
US20020144293A1 (en) * 2001-03-27 2002-10-03 Koninklijke Philips Electronics N.V. Automatic video retriever genie
US6813372B2 (en) * 2001-03-30 2004-11-02 Logitech, Inc. Motion and audio detection based webcamming and bandwidth control
JP3663362B2 (en) 2001-03-30 2005-06-22 インターナショナル・ビジネス・マシーンズ・コーポレーション Index generator
US7904814B2 (en) 2001-04-19 2011-03-08 Sharp Laboratories Of America, Inc. System for presenting audio-video content
US7054335B2 (en) * 2001-05-04 2006-05-30 Hewlett-Packard Development Company, L.P. Method and system for midstream transcoding of secure scalable packets in response to downstream requirements
US20020194589A1 (en) 2001-05-08 2002-12-19 Cristofalo Michael Technique for optimizing the delivery of advertisements and other programming segments by making bandwidth tradeoffs
US6904263B2 (en) * 2001-08-01 2005-06-07 Paul Grudnitski Method and system for interactive case and video-based teacher training
WO2003051031A2 (en) * 2001-12-06 2003-06-19 The Trustees Of Columbia University In The City Of New York Method and apparatus for planarization of a material by growing and removing a sacrificial film

Patent Citations (98)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4183056A (en) * 1977-05-23 1980-01-08 Kewp Electronic Systems, Inc. Apparatus and method for monitoring sports contests
US4321635A (en) * 1979-04-20 1982-03-23 Teac Corporation Apparatus for selective retrieval of information streams or items
US4253108A (en) * 1979-06-04 1981-02-24 Zenith Radio Corporation Control for color killer and automatic color limiter
US4520404A (en) * 1982-08-23 1985-05-28 Kohorn H Von System, apparatus and method for recording and editing broadcast transmissions
US4937685A (en) * 1983-12-02 1990-06-26 Lex Computer And Management Corporation Method of display presentation for video editing
US4729044A (en) * 1985-02-05 1988-03-01 Lex Computing & Management Corporation Method and apparatus for playing serially stored segments in an arbitrary sequence
US5027400A (en) * 1988-08-19 1991-06-25 Hitachi Ltd. Multimedia bidirectional broadcast system
US5109482A (en) * 1989-01-11 1992-04-28 David Bohrman Interactive video control system for displaying user-selectable clips
US5861881A (en) * 1991-11-25 1999-01-19 Actv, Inc. Interactive computer system for providing an interactive presentation with personalized video, audio and graphics responses for multiple viewers
US6081750A (en) * 1991-12-23 2000-06-27 Hoffberg; Steven Mark Ergonomic man-machine interface incorporating adaptive pattern recognition based control system
US5920477A (en) * 1991-12-23 1999-07-06 Hoffberg; Steven M. Human factored interface incorporating adaptive pattern recognition based controller apparatus
US5867386A (en) * 1991-12-23 1999-02-02 Hoffberg; Steven M. Morphological pattern recognition based controller system
US5903454A (en) * 1991-12-23 1999-05-11 Hoffberg; Linda Irene Human-factored interface corporating adaptive pattern recognition based controller apparatus
US5774357A (en) * 1991-12-23 1998-06-30 Hoffberg; Steven M. Human factored interface incorporating adaptive pattern recognition based controller apparatus
US5875108A (en) * 1991-12-23 1999-02-23 Hoffberg; Steven M. Ergonomic man-machine interface incorporating adaptive pattern recognition based control system
US6011895A (en) * 1992-02-07 2000-01-04 Abecassis; Max Keyword responsive variable content video program
US6208805B1 (en) * 1992-02-07 2001-03-27 Max Abecassis Inhibiting a control function from interfering with a playing of a video
US5724472A (en) * 1992-02-07 1998-03-03 Abecassis; Max Content map for seamlessly skipping a retrieval of a segment of a video
US5610653A (en) * 1992-02-07 1997-03-11 Abecassis; Max Method and system for automatically tracking a zoomed video image
US6091886A (en) * 1992-02-07 2000-07-18 Abecassis; Max Video viewing responsive to content and time restrictions
US5717814A (en) * 1992-02-07 1998-02-10 Max Abecassis Variable-content video retriever
US6038367A (en) * 1992-02-07 2000-03-14 Abecassis; Max Playing a Video Responsive to a comparison of two sets of Content Preferences
US5654769A (en) * 1992-03-11 1997-08-05 Texas Instruments Incorporated Digital color control and chroma killer device
US5200825A (en) * 1992-07-01 1993-04-06 Beam Laser Systems, Inc. Commercial insertion system remotely controlling multiple video switches
USD354059S (en) * 1992-12-03 1995-01-03 Discovery Communications, Inc. Remote control unit
US5659350A (en) * 1992-12-09 1997-08-19 Discovery Communications, Inc. Operations center for a television program packaging and delivery system
US6052554A (en) * 1992-12-09 2000-04-18 Discovery Communications, Inc. Television program delivery system
US5600364A (en) * 1992-12-09 1997-02-04 Discovery Communications, Inc. Network controller for cable television delivery systems
US5734853A (en) * 1992-12-09 1998-03-31 Discovery Communications, Inc. Set top terminal for cable television delivery systems
US5600573A (en) * 1992-12-09 1997-02-04 Discovery Communications, Inc. Operations center with video storage for a television program packaging and delivery system
US6181335B1 (en) * 1992-12-09 2001-01-30 Discovery Communications, Inc. Card for a set top terminal
US6201536B1 (en) * 1992-12-09 2001-03-13 Discovery Communications, Inc. Network manager for cable television system headends
USD348251S (en) * 1992-12-09 1994-06-28 Discovery Communications, Inc. Menu control panel for a universal remote control unit
US6067401A (en) * 1993-01-11 2000-05-23 Abecassis; Max Playing a version of and from within a video by means of downloaded segment information
US6269216B1 (en) * 1993-01-11 2001-07-31 Max Abecassis Intermittently arranged frames for instantaneously shifting among video segments
US6072934A (en) * 1993-01-11 2000-06-06 Abecassis; Max Video previewing method and apparatus
US5634849A (en) * 1993-01-11 1997-06-03 Abecassis; Max Content-on-demand interactive video method and apparatus
US5913013A (en) * 1993-01-11 1999-06-15 Abecassis; Max Seamless transmission of non-sequential video segments
US5434678A (en) * 1993-01-11 1995-07-18 Abecassis; Max Seamless transmission of non-sequential video segments
US5424770A (en) * 1993-04-16 1995-06-13 Cable Service Technologies, Inc. Method and apparatus for automatic insertion of a television signal from a remote source
US5333393A (en) * 1993-06-11 1994-08-02 Tyrone Hill Shoe retaining apparatus and method
US5923365A (en) * 1993-10-12 1999-07-13 Orad Hi-Tech Systems, Ltd Sports event video manipulating system for highlighting movement
US6195497B1 (en) * 1993-10-25 2001-02-27 Hitachi, Ltd. Associated image retrieving apparatus and method
US5521841A (en) * 1994-03-31 1996-05-28 Siemens Corporate Research, Inc. Browsing contents of a given video sequence
US6230501B1 (en) * 1994-04-14 2001-05-15 Promxd Technology, Inc. Ergonomic systems and methods providing intelligent adaptive surfaces and temperature control
US5635982A (en) * 1994-06-27 1997-06-03 Zhang; Hong J. System for automatic video segmentation and key frame extraction for video sequences having both sharp and gradual transitions
USD368263S (en) * 1994-07-12 1996-03-26 Discovery Communications, Inc. Remote control unit
USD381991S (en) * 1994-07-12 1997-08-05 Discovery Communications, Inc. Remote control unit
US5710884A (en) * 1995-03-29 1998-01-20 Intel Corporation System for automatically updating personal profile server with updates to additional user information gathered from monitoring user's electronic consuming habits generated on computer during use
US5761881A (en) * 1995-05-10 1998-06-09 Wall; Benjamin Process and apparatus for wrapping paper rolls
US5901246A (en) * 1995-06-06 1999-05-04 Hoffberg; Steven M. Ergonomic man-machine interface incorporating adaptive pattern recognition based control system
US5900867A (en) * 1995-07-17 1999-05-04 Gateway 2000, Inc. Self identifying remote control device having a television receiver for use in a computer
US6556767B2 (en) * 1995-12-15 2003-04-29 Hitachi Ltd Video capture device
US5920360A (en) * 1996-06-07 1999-07-06 Electronic Data Systems Corporation Method and system for detecting fade transitions in a video signal
US5778108A (en) * 1996-06-07 1998-07-07 Electronic Data Systems Corporation Method and system for detecting transitional markers such as uniform fields in a video signal
US5926624A (en) * 1996-09-12 1999-07-20 Audible, Inc. Digital information library and delivery system with logic for generating files targeted to the playback device
US5892536A (en) * 1996-10-03 1999-04-06 Personal Audio Systems and methods for computer enhanced broadcast monitoring
US5875107A (en) * 1996-12-05 1999-02-23 Mitsubishi Denki Kabushiki Kaisha Inverter apparatus
US6088455A (en) * 1997-01-07 2000-07-11 Logan; James D. Methods and apparatus for selectively reproducing segments of broadcast programming
US6014183A (en) * 1997-08-06 2000-01-11 Imagine Products, Inc. Method and apparatus for detecting scene changes in a digital video stream
US6219837B1 (en) * 1997-10-23 2001-04-17 International Business Machines Corporation Summary frames in video
US6055018A (en) * 1997-11-04 2000-04-25 Ati Technologies, Inc. System and method for reconstructing noninterlaced captured content for display on a progressive screen
US6252544B1 (en) * 1998-01-27 2001-06-26 Steven M. Hoffberg Mobile communication device
US6233389B1 (en) * 1998-07-30 2001-05-15 Tivo, Inc. Multimedia time warping system
US6570608B1 (en) * 1998-09-30 2003-05-27 Texas Instruments Incorporated System and method for detecting interactions of people and vehicles
US6215526B1 (en) * 1998-11-06 2001-04-10 Tivo, Inc. Analog video tagging and encoding system
US6216129B1 (en) * 1998-12-03 2001-04-10 Expanse Networks, Inc. Advertisement selection system supporting discretionary target market characteristics
US6169542B1 (en) * 1998-12-14 2001-01-02 Gte Main Street Incorporated Method of delivering advertising through an interactive video distribution system
US6342904B1 (en) * 1998-12-17 2002-01-29 Newstakes, Inc. Creating a slide presentation from full motion video
US6363160B1 (en) * 1999-01-22 2002-03-26 Intel Corporation Interface using pattern recognition and tracking
US6236395B1 (en) * 1999-02-01 2001-05-22 Sharp Laboratories Of America, Inc. Audiovisual information management system
US6418168B1 (en) * 1999-03-24 2002-07-09 Sony Corporation Motion vector detection apparatus, method of the same, and image processing apparatus
US7181757B1 (en) * 1999-10-11 2007-02-20 Electronics And Telecommunications Research Institute Video summary description scheme and method and system of video summary description data generation for efficient overview and browsing
US20070101266A1 (en) * 1999-10-11 2007-05-03 Electronics And Telecommunications Research Institute Video summary description scheme and method and system of video summary description data generation for efficient overview and browsing
US6993245B1 (en) * 1999-11-18 2006-01-31 Vulcan Patents Llc Iterative, maximally probable, batch-mode commercial detection for audiovisual content
US6549643B1 (en) * 1999-11-30 2003-04-15 Siemens Corporate Research, Inc. System and method for selecting key-frames of video data
US6597859B1 (en) * 1999-12-16 2003-07-22 Intel Corporation Method and apparatus for abstracting video data
US20020013943A1 (en) * 2000-04-07 2002-01-31 Seth Haberman System and method for simultaneous broadcast for personalized messages
US7221780B1 (en) * 2000-06-02 2007-05-22 Sony Corporation System and method for human face detection in color graphics images
US20020018594A1 (en) * 2000-07-06 2002-02-14 Mitsubishi Electric Research Laboratories, Inc. Method and system for high-level structure analysis and event detection in domain specific videos
US20040125877A1 (en) * 2000-07-17 2004-07-01 Shin-Fu Chang Method and system for indexing and content-based adaptive streaming of digital video content
US20040125124A1 (en) * 2000-07-24 2004-07-01 Hyeokman Kim Techniques for constructing and browsing a hierarchical video structure
US6724933B1 (en) * 2000-07-28 2004-04-20 Microsoft Corporation Media segmentation system and related methods
US20060098889A1 (en) * 2000-08-18 2006-05-11 Jiebo Luo Digital image processing system and method for emphasizing a main subject of an image
US20020080162A1 (en) * 2000-11-02 2002-06-27 Hao Pan Method for automatic extraction of semantically significant events from video
US7003154B1 (en) * 2000-11-17 2006-02-21 Mitsubishi Electric Research Laboratories, Inc. Adaptively processing a video based on content characteristics of frames in a video
US20020083473A1 (en) * 2000-12-21 2002-06-27 Philips Electronics North America Corporation System and method for accessing a multimedia summary of a video program
US20030026592A1 (en) * 2000-12-28 2003-02-06 Minoru Kawahara Content creating device and method
US6678635B2 (en) * 2001-01-23 2004-01-13 Intel Corporation Method and system for detecting semantic events
US20040088289A1 (en) * 2001-03-29 2004-05-06 Li-Qun Xu Image processing
US20030001880A1 (en) * 2001-04-18 2003-01-02 Parkervision, Inc. Method, system, and computer program product for producing and distributing enhanced media
US20050100213A1 (en) * 2001-06-14 2005-05-12 Microsoft Corporation Method and apparatus for shot detection
US6870956B2 (en) * 2001-06-14 2005-03-22 Microsoft Corporation Method and apparatus for shot detection
US20030081937A1 (en) * 2001-07-03 2003-05-01 Baoxin Li Summarization of video content
US7203620B2 (en) * 2001-07-03 2007-04-10 Sharp Laboratories Of America, Inc. Summarization of video content
US6585521B1 (en) * 2001-12-21 2003-07-01 Hewlett-Packard Development Company, L.P. Video indexing based on viewers' behavior and emotion feedback
US20030133511A1 (en) * 2002-01-15 2003-07-17 Romain Cabasson Summarizing videos using motion activity descriptors correlated with audio features
US20040017389A1 (en) * 2002-07-25 2004-01-29 Hao Pan Summarization of soccer video content

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8392183B2 (en) 2006-04-25 2013-03-05 Frank Elmo Weber Character-based automated media summarization
US20130121575A1 (en) * 2011-11-11 2013-05-16 Seoul National University R&B Foundation Image analysis apparatus using main color and method of controlling the same
US8913828B2 (en) * 2011-11-11 2014-12-16 Samsung Electronics Co., Ltd. Image analysis apparatus using main color and method of controlling the same
WO2016160304A1 (en) * 2015-03-31 2016-10-06 Zepp Labs, Inc. Detect sports video highlights for mobile computing devices
US10572735B2 (en) 2015-03-31 2020-02-25 Beijing Shunyuan Kaihua Technology Limited Detect sports video highlights for mobile computing devices

Also Published As

Publication number Publication date
US20050155054A1 (en) 2005-07-14
US20050155053A1 (en) 2005-07-14
US7120873B2 (en) 2006-10-10
US8028234B2 (en) 2011-09-27
US20030141665A1 (en) 2003-07-31

Similar Documents

Publication Publication Date Title
US8028234B2 (en) Summarization of sumo video content
US7143354B2 (en) Summarization of baseball video content
US7657836B2 (en) Summarization of soccer video content
US7499077B2 (en) Summarization of football video content
US7203620B2 (en) Summarization of video content
US7653131B2 (en) Identification of replay segments
Hanjalic Adaptive extraction of highlights from a sport video based on excitement modeling
US7006945B2 (en) Processing of video content
US6931595B2 (en) Method for automatic extraction of semantically significant events from video
Truong et al. Scene extraction in motion pictures
KR20040077708A (en) A method and apparatus for multimodal story segmentation for linking multimedia content
EP1265154A2 (en) Summarization of football video content
Zhang Intelligent keyframe extraction for video printing
Tjondronegoro et al. A statistical-driven approach for automatic classification of events in AFL video highlights
KR20030087357A (en) Method and Apparatus for Automatic Detection of Golf Video Event
Kim et al. Semantic event detection using MPEG-7

Legal Events

Date Code Title Description
AS Assignment

Owner name: SHARP LABORATORIES OF AMERICA, INC., WASHINGTON

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:LI, BAOXIN;REEL/FRAME:016367/0171

Effective date: 20020112

STCB Information on status: application discontinuation

Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION