US20070011206A1 - Interactive playlist generation using annotations - Google Patents
Interactive playlist generation using annotations Download PDFInfo
- Publication number
- US20070011206A1 US20070011206A1 US11/522,160 US52216006A US2007011206A1 US 20070011206 A1 US20070011206 A1 US 20070011206A1 US 52216006 A US52216006 A US 52216006A US 2007011206 A1 US2007011206 A1 US 2007011206A1
- Authority
- US
- United States
- Prior art keywords
- annotations
- annotation
- user
- media
- presentation
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/10—Text processing
- G06F40/166—Editing, e.g. inserting or deleting
- G06F40/169—Annotation, e.g. comment data or footnotes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/40—Information retrieval; Database structures therefor; File system structures therefor of multimedia data, e.g. slideshows comprising image and additional audio data
- G06F16/43—Querying
- G06F16/438—Presentation of query results
- G06F16/4387—Presentation of query results by the use of playlists
-
- G—PHYSICS
- G11—INFORMATION STORAGE
- G11B—INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER
- G11B27/00—Editing; Indexing; Addressing; Timing or synchronising; Monitoring; Measuring tape travel
- G11B27/02—Editing, e.g. varying the order of information signals recorded on, or reproduced from, record carriers
- G11B27/031—Electronic editing of digitised analogue information signals, e.g. audio or video signals
- G11B27/034—Electronic editing of digitised analogue information signals, e.g. audio or video signals on discs
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y10—TECHNICAL SUBJECTS COVERED BY FORMER USPC
- Y10S—TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y10S707/00—Data processing: database and file management or data structures
- Y10S707/912—Applications of a database
- Y10S707/913—Multimedia
- Y10S707/914—Video
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y10—TECHNICAL SUBJECTS COVERED BY FORMER USPC
- Y10S—TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y10S707/00—Data processing: database and file management or data structures
- Y10S707/912—Applications of a database
- Y10S707/913—Multimedia
- Y10S707/916—Audio
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y10—TECHNICAL SUBJECTS COVERED BY FORMER USPC
- Y10S—TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y10S707/00—Data processing: database and file management or data structures
- Y10S707/912—Applications of a database
- Y10S707/917—Text
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y10—TECHNICAL SUBJECTS COVERED BY FORMER USPC
- Y10S—TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y10S707/00—Data processing: database and file management or data structures
- Y10S707/99941—Database schema or data structure
- Y10S707/99943—Generating database or data structure, e.g. via user interface
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y10—TECHNICAL SUBJECTS COVERED BY FORMER USPC
- Y10S—TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y10S707/00—Data processing: database and file management or data structures
- Y10S707/99941—Database schema or data structure
- Y10S707/99944—Object-oriented database structure
- Y10S707/99945—Object-oriented database structure processing
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y10—TECHNICAL SUBJECTS COVERED BY FORMER USPC
- Y10S—TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y10S707/00—Data processing: database and file management or data structures
- Y10S707/99941—Database schema or data structure
- Y10S707/99948—Application of database or data structure, e.g. distributed, multimedia, or image
Definitions
- This invention relates to networked client/server systems and to methods of delivering and rendering multimedia content in such systems. More particularly, the invention relates to systems and methods of selecting and providing such content.
- Synchronized media means multiple media objects that share a common timeline.
- Video and audio are examples of synchronized media-each is a separate data stream with its own data structure, but the two data streams are played back in synchronization with each other.
- Virtually any media type can have a timeline.
- an image object can change like an animated .gif file, text can change and move, and animation and digital effects can happen over time. This concept of synchronizing multiple media types is gaining greater meaning and currency with the emergence of more sophisticated media composition frameworks implied by MPEG-4, Dynamic HTML, and other media playback environments.
- streaming is used to indicate that the data representing the various media types is provided over a network to a client computer on a real-time, as-needed basis, rather than being pre-delivered in its entirety before playback.
- client computer renders streaming data as it is received from a network server, rather than waiting for an entire “file” to be delivered.
- Multimedia presentations may also include “annotations” relating to the multimedia presentation.
- An annotation is data (e.g., audio, text, video, etc.) that corresponds to a multimedia presentation.
- Annotations can be added by anyone with appropriate access rights to the annotation system (e.g., the lecturer/trainer or any of the students/trainees).
- These annotations typically correspond to a particular temporal location in the multimedia presentation and can provide a replacement for much of the “in-person” interaction and “classroom discussion” that is lost when the presentation is not made “in-person” or “live”.
- a student can comment on a particular point, to which another student (or lecturer) can respond in a subsequent annotation. This process can continue, allowing a “classroom discussion” to occur via these annotations.
- some systems allow a user to select a particular one of these annotations and begin playback of the presentation starting at approximately the point in the presentation to which the annotation corresponds.
- the invention described below addresses this and other disadvantages of annotations, providing a way to improve multimedia presentation using annotations.
- Annotations correspond to media segments of one or more multimedia streams.
- a playlist generation interface is presented to the user in the form of annotation titles or summaries for a group of annotations.
- This group of annotations corresponds to the media segments that are part of a playlist.
- the playlist can then be altered by the user to suit his or her desires or needs by interacting with the annotation title/summary interface.
- the media segments of the playlist can then be presented to the user in a seamless, contiguous manner.
- the ordering of the annotation titles/summaries can be altered by the user, resulting in a corresponding change in order of presentation of the media segments.
- the ordering of the annotation titles/summaries can be changed by moving the titles or summaries in a drag and drop manner.
- the media segments of the playlist can themselves be stored as an additional multimedia stream.
- This additional multimedia stream can then be annotated in the same manner as other multimedia streams.
- FIG. 1 shows a client/server network system and environment in accordance with one embodiment of the invention.
- FIG. 2 shows a general example of a computer that can be used as a client or server in accordance with the invention.
- FIG. 3 is a block diagram illustrating an annotation server and a client computer in more detail in accordance with one embodiment of the invention.
- FIG. 4 is a block diagram illustrating the structure for an annotation according to one embodiment of the invention.
- FIG. 5 is a block diagram illustrating exemplary annotation collections.
- FIG. 6 illustrates an annotation toolbar in accordance with one embodiment of the invention.
- FIG. 7 illustrates an “add new annotation” dialog box in accordance with one embodiment of the invention.
- FIG. 8 illustrates a “query annotations” dialog box in accordance with one embodiment of the invention.
- FIG. 9 illustrates a “view annotations” dialog box in accordance with one embodiment of the invention.
- FIG. 10 is a diagrammatic illustration of a graphical user interface window displaying annotations and corresponding media segments concurrently in accordance with one embodiment of the invention.
- FIG. 11 illustrates methodological aspects of one embodiment of the invention in retrieving and presenting annotations and media segments to a user.
- FIG. 1 shows a client/server network system and environment in accordance with one embodiment of the invention.
- the system includes multiple network server computers 10 , 11 , 12 , and 13 , and multiple (n) network client computers 15 .
- the computers communicate with each other over a data communications network.
- the communications network in FIG. 1 comprises a public network 16 such as the Internet.
- the data communications network might also include, either in addition to or in place of the Internet, local-area networks and/or private wide-area networks.
- Streaming media server computer 11 has access to streaming media content in the form of different media streams.
- These media streams can be individual media streams (e.g., audio, video, graphical, etc.), or alternatively can be composite media streams including two or more of such individual streams.
- Some media streams might be stored as files in a database or other file storage system, while other media streams might be supplied to the server on a “live” basis from other data source components through dedicated communications channels or through the Internet itself.
- ASF Advanced Streaming Format
- Annotation server 10 controls the storage of annotations and their provision to client computers 15 .
- the annotation server 10 manages the annotation meta data store 18 and the annotation content store 17 .
- the annotation server 10 communicates with the client computers 15 via any of a wide variety of known protocols, such as the Hypertext Transfer Protocol (HTTP).
- HTTP Hypertext Transfer Protocol
- the annotation server 10 can receive and provide annotations via direct contact with a client computer 15 , or alternatively via electronic mail (email) via email server 13 .
- the annotation server 10 similarly communicates with the email server 13 via any of a wide variety of known protocols, such as the Simple Mail Transfer Protocol (SMTP).
- SMTP Simple Mail Transfer Protocol
- annotation server 10 corresponds to the streaming media available from media server computer 11 .
- annotations are discussed as corresponding to streaming media.
- annotations can similarly correspond to “pre-delivered” rather than streaming media, such as media previously stored at the client computers 15 via the network 16 , via removable magnetic or optical disks, etc.
- a conventional web browser of the client computer 15 contacts the web server 12 to get the Hypertext Markup Language (HTML) page, the media server 11 to get the streaming data, and the annotation server 10 to get any annotations associated with that media.
- the client computer 15 contacts the annotation server 10 to perform the desired addition/retrieval.
- program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types.
- program modules may be located in both local and remote memory storage devices.
- FIG. 2 shows a general example of a computer 20 that can be used as a client or server in accordance with the invention.
- Computer 20 is shown as an example of a computer that can perform the functions of any of server computers 10 - 13 or a client computer 15 of FIG. 1 .
- Computer 20 includes one or more processors or processing units 21 , a system memory 22 , and a bus 23 that couples various system components including the system memory 22 to processors 21 .
- the bus 23 represents one or more of any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures.
- the system memory includes read only memory (ROM) 24 and random access memory (RAM) 25 .
- ROM read only memory
- RAM random access memory
- a basic input/output system (BIOS) 26 containing the basic routines that help to transfer information between elements within computer 20 , such as during start-up, is stored in ROM 24 .
- Computer 20 further includes a hard disk drive 27 for reading from and writing to a hard disk, not shown, a magnetic disk drive 28 for reading from and writing to a removable magnetic disk 29 , and an optical disk drive 30 for reading from or writing to a removable optical disk 31 such as a CD ROM or other optical media.
- the hard disk drive 27 , magnetic disk drive 28 , and optical disk drive 30 are connected to the system bus 23 by an SCSI interface 32 or some other appropriate interface.
- the drives and their associated computer-readable media provide nonvolatile storage of computer readable instructions, data structures, program modules and other data for computer 20 .
- a number of program modules may be stored on the hard disk, magnetic disk 29 , optical disk 31 , ROM 24 , or RAM 25 , including an operating system 35 , one or more application programs 36 , other program modules 37 , and program data 38 .
- a user may enter commands and information into computer 20 through input devices such as keyboard 40 and pointing device 42 .
- Other input devices may include a microphone, joystick, game pad, satellite dish, scanner, or the like.
- These and other input devices are connected to the processing unit 21 through an interface 46 that is coupled to the system bus.
- a monitor 47 or other type of display device is also connected to the system bus 23 via an interface, such as a video adapter 48 .
- personal computers typically include other peripheral output devices (not shown) such as speakers and printers.
- Computer 20 operates in a networked environment using logical connections to one or more remote computers, such as a remote computer 49 .
- the remote computer 49 may be another personal computer, a server, a router, a network PC, a peer device or other common network node, and typically includes many or all of the elements described above relative to computer 20 , although only a memory storage device 50 has been illustrated in FIG. 2 .
- the logical connections depicted in FIG. 2 include a local area network (LAN) 51 and a wide area network (WAN) 52 .
- LAN local area network
- WAN wide area network
- Such networking environments are commonplace in offices, enterprise-wide computer networks, intranets, and the Internet.
- remote computer 49 executes an Internet Web browser program such as the “Internet Explorer” Web browser manufactured and distributed by Microsoft Corporation of Redmond, Washington.
- computer 20 When used in a LAN networking environment, computer 20 is connected to the local network 51 through a network interface or adapter 53 .
- computer 20 When used in a WAN networking environment, computer 20 typically includes a modem 54 or other means for establishing communications over the wide area network 52 , such as the Internet.
- the modem 54 which may be internal or external, is connected to the system bus 23 via a serial port interface 33 .
- program modules depicted relative to the personal computer 20 may be stored in the remote memory storage device. It will be appreciated that the network connections shown are exemplary and other means of establishing a communications link between the computers may be used.
- the data processors of computer 20 are programmed by means of instructions stored at different times in the various computer-readable storage media of the computer.
- Programs and operating systems are typically distributed, for example, on floppy disks or CD-ROMs. From there, they are installed or loaded into the secondary memory of a computer. At execution, they are loaded at least partially into the computer's primary electronic memory.
- the invention described herein includes these and other various types of computer-readable storage media when such media contain instructions or programs for implementing the steps described below in conjunction with a microprocessor or other data processor.
- the invention also includes the computer itself when programmed according to the methods and techniques described below.
- certain sub-components of the computer may be programmed to perform the functions and steps described below. The invention includes such sub-components when they are programmed as described.
- the invention described herein includes data structures, described below, as embodied on various types of memory media.
- programs and other executable program components such as the operating system are illustrated herein as discrete blocks, although it is recognized that such programs and components reside at various times in different storage components of the computer, and are executed by the data processor(s) of the computer.
- FIG. 3 illustrates an annotation server and a client computer in more detail.
- commands are formulated at client computer 15 and forwarded to annotation server 10 via HTTP requests.
- communication between client 15 and server 10 is performed via HTTP, using commands encoded as Uniform Resource Locators (URLs) and data formatted as object linking and embedding (OLE) structured storage documents, or alternatively using Extensible Markup Language (XML).
- URLs Uniform Resource Locators
- OLE object linking and embedding
- XML Extensible Markup Language
- Client 15 includes an HTTP services (HttpSvcs) module 152 , which manages communication with server 10 , and an annotation back end (ABE) module 151 , which translates user actions into commands destined for server 10 .
- a user interface (MMA) module 150 provides the user interface (UI) for a user to add and select different annotations, and be presented with the annotations.
- the user interface module 150 supports ActiveX controls that display an annotation interface for streaming video on the Web.
- Client 15 also includes a web browser module 153 , which provides a conventional web browsing interface and capabilities for the user to access various servers via network 16 of FIG. 1 .
- Web browser 153 also provides the interface for a user to be presented with media streams.
- the user can select which one of different versions of multimedia content he or she wishes to receive from media server 11 of FIG. 1 . This selection can be direct (e.g., entry of a particular URL or selection of a “low resolution” option), or indirect (e.g., entry of a particular desired playback duration or an indication of system capabilities, such as “slow system” or “fast system”). Alternatively, other media presentation interfaces could be used.
- Annotation server 10 includes the Multimedia Annotation Web Server (MAWS) module 130 , which is an Internet Services Application Programming Interface (ISAPI) plug-in for Internet Information Server (IIS) module 135 . Together, these two modules provide the web server functionality of annotation server 10 .
- Annotation server 10 also includes an HTTP Services module 131 which manages communication with client 15 .
- annotation server 10 utilizes The Windows Messaging Subsystem 134 to facilitate communication with email server 13 of FIG. 1 , and an email reply server 133 for processing incoming email received from email server 13 .
- Annotation server 10 further includes an annotation back end (ABE) module 132 , which contains functionality for accessing annotation stores 17 and 18 , for composing outgoing email based on annotation data, and for processing incoming email.
- Incoming email is received and passed to the ABE module 132 by the Email Reply Server 133 .
- Annotation content authored at client 15 is received by ABE 132 and maintained in annotation content store 17 .
- Received meta data (control information) corresponding to the annotation content is maintained in annotation meta data store 18 .
- the annotation content and meta data can be stored in any of a variety of conventional manners, such as in SQL relational databases (e.g., using Microsoft “SQL Server” version 7.0, available from Microsoft Corporation).
- Annotation server 10 is illustrated in FIG. 3 as maintaining the annotation content and associated control information (meta data) separately in two different stores.
- annotation data content and meta information
- all of the annotation data can be stored together in a single store, or content may be stored by another distinct storage system on the network 16 of FIG. 1 , such as a file system, media server, email server, or other data store.
- ABE 132 of annotation server 10 also manages the interactive generation and presentation of streaming media data from server computer 11 of FIG. 1 using “playlists”.
- a “playlist” is a listing of one or more multimedia segments to be retrieved and presented in a given order.
- Each of the multimedia segments in the playlist is defined by a source identifier, a start time, and an end time.
- the source identifier identifies which media stream the segment is part of, the start time identifies the temporal location within the media stream where the segment begins, and the end time identifies the temporal location within the media stream where the segment ends.
- ABE 132 allows playlists to be generated interactively based on annotations maintained in annotation stores 17 and 18 .
- ABE 132 provides a user at client 15 with multiple possible annotation identifiers (e.g., titles or summaries) from which the user can select those of interest to him or her. Based on the selected annotations, ABE 132 coordinates provision of the associated media segments to the user.
- ABE 132 can directly communicate with video server computer 11 to identify which segments are to be provided, or alternatively can provide the appropriate information to the browser of client computer 15 , which in turn can request the media segments from server computer 11 .
- FIG. 4 shows an exemplary structure for an annotation entry 180 that is maintained by annotation server 10 in annotation meta data store 18 of FIG. 3 .
- an annotation entry 180 includes an author field 182 , a time range field 184 , a time units field 186 , a creation time field 188 , a title field 190 , a content field 192 , an identifier field 194 , a related annotation identifier field 196 , a set identifier(s) field 198 , a media content identifier field 200 , an arbitrary number of user-defined property fields 202 , and a sequence number 204 .
- Each of fields 182 - 204 is a collection of data which define a particular characteristic of annotation entry 180 .
- Annotation entry 180 is maintained by annotation server 10 of FIG. 3 in annotation meta data store 18 .
- Content field 192 includes a pointer to (or other identifier of) the annotation content, which in turn is stored in annotation content store 17 .
- Author field 182 contains data identifying the user who created annotation entry 180 and who is therefore the author of the annotation. The author is identified by ABE 151 of FIG. 3 based on the user logged into client 15 at the time the annotation is created.
- Time range field 184 contains data representing “begin” and “end” times defining a segment of media timeline to which annotation entry 180 is associated.
- Time units field 186 contains data representing the units of time represented in time range field 184 .
- time range field 184 and time units field 186 identify the relative time range of the annotation represented by annotation entry 180 . This relative time range corresponds to a particular segment of the media stream to which annotation entry 180 is associated.
- the begin and end times for the annotation are provided by the user via interface 150 of FIG. 3 , or alternatively can be automatically or implicitly derived using a variety of audio and video signal processing techniques, such as sentence detection in audio streams or video object tracking.
- a first annotation may correspond to a segment ranging between the first and fourth minutes of media content
- a second annotation may correspond to a segment ranging between the second and seventh minutes of the media content
- a third annotation may correspond to a segment ranging between the second and third minutes of the media content.
- annotations could be associated with (or “anchored” on) specific objects in the video content, or specific events in the audio content.
- Creation time field 188 contains data specifying the date and time at which annotation entry 180 is created. It should be noted that the time of creation of annotation entry 180 is absolute and is not relative to the video or audio content of the media stream to which annotation entry 180 is associated. Accordingly, a user can specify that annotations which are particularly old, e.g., created more than two weeks earlier, are not to be displayed. ABE 132 of FIG. 3 stores the creation time and date when the annotation is created.
- Title field 190 contains data representing a title by which the annotation represented by annotation entry 180 is identified.
- the title is generally determined by the user and the user enters the data representing the title using conventional and well known user interface techniques.
- the data can be as simple as ASCII text or as complex as HTML code which can include text having different fonts and type styles, graphics including wallpaper, motion video images, audio, and links to other multimedia documents.
- Content field 192 contains data representing the substantive content of the annotation as authored by the user.
- the actual data can be stored in content field 192 , or alternatively content field 192 may store a pointer to (or other indicator of) the content that is stored separately from the entry 180 itself.
- content field 192 includes a pointer to (or other identifier of) the annotation content, which in turn is stored in annotation content store 17 .
- the user enters the data representing the content using conventional and well known user interface techniques.
- the content added by the user in creating annotation entry 180 can include any one or more of text, graphics, video, audio, etc. or links thereto.
- content field 192 contains data representing the substantive content the user wishes to include with the presentation of the corresponding media stream at the relative time range represented by time range field 184 and time units field 186 .
- Annotation identifier field 194 stores data that uniquely identifies annotation entry 180
- related annotation identifier field 196 stores data that uniquely identifies a related annotation.
- Annotation identifier field 194 can be used by other annotation entries to associate such other annotation entries with annotation entry 180 .
- threads of discussion can develop in which a second annotation responds to a first annotation, a third annotation responds to the second annotation and so on.
- an identifier of the first annotation would be stored in related annotation identifier field 196 of the second annotation
- an identifier of the second annotation would be stored in related annotation identifier field 196 of the third annotation, and so on.
- Set identifier(s) field 198 stores data that identifies a particular one or more sets to which annotation entry 180 belongs.
- a media stream can have multiple sets of annotations, sets can span multiple media content, and a particular annotation can correspond to one or more of these sets. Which set(s) an annotation belongs to is identified by the author of the annotation.
- a media stream corresponding to a lecture may include the following sets: “instructor's comments”, “assistant's comments”, “audio comments”, “text comments”, “student questions”, and each student's personal comments.
- Media content identifier field 200 contains data that uniquely identifies particular multimedia content as the content to which annotation entry 180 corresponds.
- Media content identifier 200 can identify a single media stream (either an individual stream or a composite stream), or alternatively identify multiple different streams that are different versions of the same media content.
- Media content identifier 200 can identify media versions in a variety of different manners.
- the data represents a real-time transport protocol (RTP) address of the different media streams.
- RTP address is a type of uniform resource locator (URL) by which multimedia documents can be identified in a network.
- URL uniform resource locator
- a unique identifier is assigned to the content rather than to the individual media streams.
- a different unique identifier of the media streams could be created by annotation server 10 of FIG. 3 and assigned to the media streams. Such a unique identifier would also be used by streaming media server 11 of FIG. 1 to identify the media streams.
- a uniform resource name such as those described by K. Sollins and L. Mosinter in “Functional Requirements for Uniform Resource Names,” IETF RFC 1733 (December 1994) could be used to identify the media stream.
- User-defined property fields 202 are one or more user-definable fields that allow users (or user interface designers) to customize the annotation system.
- additional property fields include a “reference URL” property which contains the URL of a web page used as reference material for the content of the annotation; a “help URL” property containing the URL of a help page which can be accessed concerning the content of the annotation; a “view script” property containing JavaScript which is to be executed whenever the annotation is viewed; a “display type” property, which gives the client user interface information about how the annotation is to be displayed; etc.
- Sequence number 204 allows a user to define (via user interface 150 of FIG. 3 ) a custom ordering for the display of annotation identifiers, as discussed in more detail below. Sequence number 204 stores the relative position of the annotations with respect to one another in the custom ordering, allowing the custom ordering to be saved for future used.
- annotation entry 180 stores a single sequence number. Alternatively, multiple sequence numbers 204 may be included in annotation entry 180 each corresponding to a different custom ordering, or a different annotation set, or a different user, etc.
- FIG. 5 illustrates exemplary implicit annotation collections for annotations maintained by annotation server 10 of FIG. 3 .
- a collection of annotations refers to annotation entries 180 of FIG. 4 that correspond to the same media stream(s), based on the media content identifier 200 .
- Annotation entries 180 can be viewed conceptually as part of the same annotation collection if they have the same media content identifiers 200 , even though the annotation entries may not be physically stored together by annotation server 10 .
- Annotation database 206 includes two annotation collections 208 and 210 .
- Annotation server 10 dynamically adds, deletes, and modifies annotation entries in annotation database 206 based on commands from client 15 .
- Annotation entries can be created and added to annotation database 206 at any time a user cares to comment upon the content of the stream (or another annotation) in the form of an annotation.
- Annotation server 10 forms an annotation entry from identification data, content data, title data, and author data of an “add annotation” request received from the client's ABE 151 ( FIG. 3 ), and adds the annotation entry to annotation database 206 .
- Annotation database 206 includes a fields 212 , 214 , and 216 that specify common characteristics of all annotation entries of database 206 or an annotation collection 208 or 210 .
- fields 212 - 216 can be included redundantly in each annotation entry 180 .
- Creator field 212 contains data identifying the user who was responsible for creating annotation database 206 .
- RTP address fields 214 and 216 contain data representing an RTP address of the media stream (e.g., the RTP address of the stream identified in media content identifier 200 of FIG. 5 ) for the annotation collection.
- An RTP address provides an alternative mechanism, in addition to the data in identifier field 200 , for associating the media stream with annotation entries 180 .
- RTP address fields 214 and 216 need not be included, particularly embodiments in which media content identifier 200 contains the RTP address of the media stream.
- An annotation can be created by a user of any of the client computers 15 of FIG. 1 .
- client 15 includes an interface module 150 that presents an interface to a user (e.g., a graphical user interface), allowing a user to make requests of annotation server 10 .
- a user can access annotation server 10 via an annotation toolbar provided by interface 150 .
- FIG. 6 illustrates an annotation toolbar in accordance with one embodiment of the invention.
- Annotation toolbar 240 includes various identifying information and user-selectable options 242 - 254 .
- a server identifier 244 identifies the annotation server with which client 15 is currently configured to communicate (annotation server 10 of FIG. 1 . in the illustrated embodiment).
- Selection of a connection button 246 causes ABE 151 of FIG. 3 to establish a connection with the annotation server identified by identifier 244 .
- Selection of a query button 248 causes interface module 150 to open a “query” dialog box, from which a user can search for particular annotations.
- Selection of an add button 250 causes interface module 150 to open an “add new annotation” dialog box, from which a user can create a new annotation.
- Selection of a show annotations button 252 causes interface module 150 to open a “view annotations” dialog box, from which a user can select particular annotations for presentation.
- Selection of a preferences button 254 causes interface 150 of FIG. 3 to open a “preferences” dialog box, from which a user can specify various UI preferences, such as an automatic server query refresh interval, or default query criteria values to be persisted between sessions.
- FIG. 7 shows an “add new annotation” dialog box 260 that results from user selection of add button 250 of FIG. 6 to create a new annotation.
- Interface 150 assumes that the current media stream being presented to the user is the media stream to which this new annotation will be associated.
- the media stream to which an annotation is associated is referred to as the “target” of the annotation.
- An identifier of this stream is displayed in a target specification area 262 of dialog box 260 .
- a user could change the target of the annotation, such as by typing in a new identifier in target area 262 , or by selection of a “browse” button (not shown) that allows the user to browse through different directories of media streams.
- a time strip 264 is also provided as part of dialog box 260 .
- Time strip 264 represents the entire presentation time of the corresponding media stream.
- a thumb 265 that moves within time strip 264 indicates a particular temporal position within the media stream.
- the annotation being created via dialog box 260 has a begin time and an end time, which together define a particular segment of the media stream.
- thumb 265 represents the begin time for the segment relative to the media stream.
- thumb 265 represents the end time for the segment relative to the media stream.
- two different thumbs could be displayed, one for the begin time and one for the end time.
- the begin and end times are also displayed in an hours/minutes/seconds format in boxes 266 and 270 , respectively.
- Thumb 265 can be moved along time strip 264 in any of a variety of conventional manners. For example, a user can depress a button of a mouse (or other cursor control device) while a pointer is “on top” of thumb 265 and move the pointer along time strip 264 , causing thumb 265 to move along with the pointer; The appropriate begin or end time is then set when the mouse button is released. Alternatively, the begin and end times can be set by entering (e.g., via an alphanumeric keyboard) particular times in boxes 266 and 270 .
- Dialog box 260 also includes a “play” button 274 .
- Selection of play button 274 causes interface module 150 of FIG. 3 to forward a segment specification to web browser 153 of client 15 .
- the segment specification includes the target identifier from target display 262 and the begin and end times from boxes 266 and 270 , respectively.
- the browser Upon receipt of the segment specification from interface module 150 , the browser communicates with media server 11 and requests the identified media segment using conventional HTTP requests. In response, media server 11 streams the media segment to client 15 for presentation to the user. This presentation allows, for example, the user to verify the portion of the media stream to which his or her annotation will correspond.
- Dialog box 260 also includes an annotation set identifier 272 , an email field 275 , and a summary 276 .
- Annotation set identifier 272 allows the user to identify a named set to which the new annotation will belong. This set can be a previously defined set, or a new set being created by the user. Selection of the particular set can be made from a drop-down menu activated by selection of a button 273 , or alternatively can be directly input by the user (e.g., typed in using an alphanumeric keyboard).
- annotation server 10 of FIG. 3 supports read and write access controls, allowing the creator of the set to identify which users are able to read and/or write to the annotation set. In this embodiment, only those sets for which the user has write access can be entered as set identifier 272 .
- Email identifier 275 allows the user to input the email address of a recipient of the annotation.
- the newly created annotation is electronically mailed to the recipient indicated in identifier 275 in addition to being added to the annotation database.
- the recipient of the electronic mail message can reply to the message to create an additional annotation.
- the original email message is configured with annotation server 10 as the sender. Because of this, a “reply to sender” request from the recipient will cause an email reply to be sent to annotation server 10 .
- annotation server 10 creates a new annotation and uses the reply message content as the content of the new annotation.
- This new annotation identifies, as a related annotation, the original annotation that was created when the original mail message was sent by annotation server 10 .
- this related annotation identifier is stored in field 196 of FIG. 4 .
- Summary 276 allows the user to provide a short summary or title of the annotation content. Although the summary is illustrated as being text, it could include any of a wide variety of characters, alphanumerics, graphics, etc. In the illustrated embodiment, summary 276 is stored in the title field 190 of the annotation entry of FIG. 4 .
- Dialog box 260 further includes radio buttons 280 and 282 , which allow an annotation to be created as text and/or audio. Although not shown, other types of annotations could also be accommodated, such as graphics, HTML documents, etc.
- Input controls 278 are also provided as part of dialog box. The illustrated controls are enabled when the annotation includes audio data. Input controls 278 include conventional audio control buttons such as fast forward, rewind, play, pause, stop and record. Additionally, an audio display bar 279 can be included to provide visual progress feedback when the audio is playing or recording.
- input controls 278 may simply include a box into which text can be input by the user via an alphanumeric keyboard. Additionally, a keyboard layout may also be provided to the user, allowing him or her to “point and click” using a mouse and pointer to select particular characters for entry.
- FIG. 8 shows a “query annotations” dialog box 330 that results from a user selecting query button 248 of FIG. 6 .
- Many of the options presented to the user in dialog box 330 are similar to those presented in the “add new annotation” dialog box 260 of FIG. 7 , however, those in dialog box 330 are used as search criteria rather than data for a new annotation.
- Dialog box 330 includes a target display 332 that contains an identifier of the target stream.
- This identifier can be input in any of a variety of manners, such as by typing in a new identifier in target display 332 , or by selection of a “browse” button (not shown) that allows the user to browse through different directories of media streams.
- the identifier is an URL.
- alternate embodiments can use different identifier formats.
- Dialog box 330 also includes target information 334 , which includes a time strip, thumb, “from” button, “to” button, “play” button, and begin and end times, which are analogous to the time strip, thumb, “from” button, “to” button, “play” button, begin and end times of dialog box 260 of FIG. 7 .
- the begin and end times in target information 334 limit the query for annotations to only those annotations having a time range that corresponds to at least part of the media segment between the begin and end times of target information 334 .
- Dialog box 330 also includes an annotation set list 336 .
- Annotation set list 336 includes a listing of the various sets that correspond to the target media stream. According to one implementation, only those sets for which an annotation has been created are displayed in set list 336 .
- annotation server 10 of FIG. 3 supports read and write security, allowing the creator of the set to identify which users are able to read and/or write to the annotation set. In this embodiment, only those sets for which the user has read access are displayed in set list 336 .
- a user can select sets from annotation set list 336 in a variety of manners. For example, using a mouse and pointer to “click” on a set in list 336 , which highlights the set to provide feedback to the user that the set has been selected. Clicking on the selected set again de-selects the set (leaving it no longer highlighted). Additionally, a “select all” button 338 allows a user to select all sets in set list 336 , while a “deselect all” button 340 allows a user to de-select all sets in set list 336 .
- the sets displayed as part of annotation set list 336 contain annotations which correspond to the target identifier in target display 332 .
- the sets in selection list 338 need not necessarily contain annotations which correspond to the target identifier in target display 332 .
- Interface module 150 allows a user to select different target streams during the querying process. Thus, a user may identify a first target stream and select one or more sets to query annotations from for the first target stream, and then identify a second target stream and select one or more sets to query annotations from for the second target stream.
- Additional search criteria can also be input by the user.
- a particular creation date and time identifier 342 can be input, along with a relation 344 (e.g., “after” or “before”).
- a relation 344 e.g., “after” or “before”.
- particular words, phrases, characters, graphics, etc. that must appear in the summary can be input in a summary keyword search identifier 346 .
- a maximum number of annotations to retrieve in response to the query can also be included as a max identifier 348 .
- the query can be limited to only annotations that correspond to the target identifier in target display 332 by selecting check box 360 .
- a level of detail 350 to retrieve can also be selected by the user. Examples of different levels that could be retrieved include the “full level” (that is, all content of the annotation), or a “deferred download” where only an identifier of the annotations (e.g., a summary or title) is downloaded. In the illustrated example, selection of checkbox 354 selects the deferred download level, whereas if checkbox 354 is not selected then the full level of detail is implicitly selected.
- a server identifier 356 identifies the annotation server with which client 15 is currently configured to communicate. Different annotation servers can be selected by the user by inputting the appropriate identifier as server identifier 356 . This input can be provided in any of a variety of manners, such as by typing in a new identifier in server identifier 356 or by selection of a “browse” button (not shown) that allows the user to browse through different directories of annotation servers.
- a user can request automatic display of the retrieved annotations by selecting a “display retrieved annotations” checkbox 358 .
- Selection of “advanced” button 362 reduces the number of options available to the user, simplifying dialog box 330 .
- the simplified dialog box may not include fields 342 , 344 , 348 , 346 , 350 , 332 , 334 , or 336 .
- interface 150 closes the query dialog box 330 and forwards the search criteria to annotation server 10 . Additionally, if checkbox 358 is selected then interface 150 displays a “view annotations” dialog box 400 of FIG. 9 . Alternatively, a user can provide a view request, causing interface 150 to display dialog box 400 , by selecting show annotations button 252 in annotation toolbar 240 of FIG. 6 .
- FIG. 9 shows a dialog box 400 that identifies annotations corresponding to a playlist of media segments.
- the playlist is a result of the query input by the user as discussed above with reference to FIG. 8 .
- annotation identifiers in the form of user identifiers 406 and summaries 408 are displayed within an annotation listing box 402 .
- the user can scroll through annotation identifiers in a conventional manner via scroll bars 404 and 405 .
- the annotation identifiers are presented in annotation listing box 402 according to a default criteria, such as chronological by creation time/date, by user, alphabetical by summaries, etc.
- annotation listing 402 in a hierarchical, horizontally offset manner.
- the identifier of an annotation that is related to a previous annotation is “indented” from that previous annotation's identifier and a connecting line between the two identifiers is shown.
- Dialog box 400 can be displayed concurrently with a multimedia player that is presenting multimedia content that corresponds to the annotations in annotation listing 402 (e.g., as illustrated in FIG. 10 below).
- Interface module 150 can have the annotations “track” the corresponding multimedia content being played back, so that the user is presented with an indication (e.g., an arrow) as to which annotation(s) correspond to the current temporal position of the multimedia content.
- Such tracking can be enabled by selecting checkbox 422 , or disabled by de-selecting checkbox 422 .
- Dialog box 400 also includes a merge annotation sets checkbox 424 .
- Selection of merge annotation sets checkbox 424 causes interface module 150 to present annotation identifiers in listing box 402 in a chronological order regardless of what set(s) the annotations in annotation listing 402 belong to. If checkbox 424 is not selected, then annotations from different sets are grouped and displayed together in annotation listing 402 (e.g., under the same tree item). Thus, when checkbox 424 is not selected, interface 150 displays one playlist for each annotation set that has been retrieved from annotation server 10 .
- Dialog box 400 also includes a refresh button 428 , a close button 430 , and an advanced button 432 .
- Selection of refresh button 428 causes interface module 150 to communicate with annotation back end 151 to access annotation server 10 and obtain any additional annotations that correspond to the query that resulted in listing box 402 .
- Selection of close button 430 causes interface 150 to terminate the display of dialog box 400 .
- Selection of advanced button 432 causes interface 150 to display a different view annotations box having additional details, such as annotation target information (analogous to target display 332 discussed below with reference to FIG. 8 ), user-selectable preferences for information displayed as annotation identifiers in listing box 402 , etc.
- preview information is presented in a preview section 416 , and a selection box or menu 410 is provided.
- the exact nature of the preview information is dependent on the data type and amount of information that was requested (e.g., as identified in level of detail 350 of FIG. 8 ).
- Menu 410 includes the following options: play, export ASX playlist, export annotations, time order, custom order, save, and reset.
- Selection of the “play” option causes playback of the multimedia content to begin starting with the selected annotation in annotation list 402 .
- Selection of the “export ASX playlist” option causes annotation backend 151 to output a record (e.g., create a file) that identifies the temporal segments of multimedia content that the annotations identified in list 402 correspond to, as determined by the begin and end times of the annotations.
- Selection of the “export annotations” option causes annotation backend 151 to output a record (e.g., create a file) that includes the annotation content of each annotation identified in list 402 .
- Selection of the “time order” option causes interface module 150 to display the identifiers in list 402 in chronological order based on the begin time for each annotation.
- Selection of the “custom order” option allows the user to identify some other criteria to be used in determining the order of the identifiers in list 402 (e.g., identifiers can be re-ordered in a conventional drag and drop manner).
- Re-ordering annotation identifiers causes the sequence numbers 204 (of FIG. 4 ) of the annotations to be re-ordered accordingly.
- Selection of the “save” option causes interface module 150 to save the current custom ordering to annotation server 10 of FIG. 3 by saving the current sequence numbers of the annotations.
- Selection of the “reset” option causes interface module 150 to ignore any changes that have been made since the last saved custom ordering and revert to the last saved custom ordering.
- Transfer of the corresponding media segments (and/or the annotations) to client 15 is initiated when the “play” option of menu 410 is selected.
- interface 150 of FIG. 3 provides the list of annotation identifiers being displayed to web browser 153 (or other multimedia presentation application) in the order of their display, including the target identifier and temporal range information.
- web browser 153 receives a list of multimedia segments that it is to present to the user in a particular order.
- Web browser 153 accesses media server 11 to stream the multimedia segments to client 15 for presentation in that order.
- a user is able to review the information regarding the annotations that satisfy his or her search criteria and then modify the annotation playlist (e.g., by deleting or reordering annotation identifiers) before the corresponding media segments (and/or the annotations) are presented to him or her.
- modify the annotation playlist e.g., by deleting or reordering annotation identifiers
- transfer of the media segments may be initiated in other manners rather than by selection of the play option in menu 410 .
- a “start” button may be included as part of dialog box 400 , selection of which initiates transfer of the media segments to client 15 .
- annotations and/or corresponding media segments are presented to the user “back to back” with very little or no noticeable gap between different annotations and between different segments. Thus, the presentation of the annotations and/or media segments is “seamless”.
- a user is able to reorder the media segments of the playlist and thereby alter their order of presentation.
- media segments are reordered by changing the ordering of the annotation identifiers in annotation listing 402 in a drag and drop manner. For example, using a mouse and pointer a user can select a particular annotation identifier (e.g., identifier 420 ) and drag it to a different location within the dialog box (e.g., between identifiers 419 and 421 ), thereby changing when the media segment corresponding to the annotation identified by identifier 420 is presented relative to the other annotations.
- a particular annotation identifier e.g., identifier 420
- web browser 153 sends a message to the appropriate media server 11 of FIG. 1 to begin streaming the appropriate segment to client computer 15 .
- Web browser 153 knowing the duration of each of the segments being provided to client computer 15 , forwards additional messages to media server 11 to continue with the provision of the next segment, according to the playlist, when appropriate.
- web browser 153 can keep the media segments being provided to the user in a seamless manner.
- the media segments could be streamed to annotation server 10 for temporary buffering and subsequent streaming to client computer 15 .
- identifying information e.g., source, start time, and end time
- media server 11 e.g., source, start time, and end time
- the collection of media segments identified by the playlist can be stored as an additional media stream by selecting “export ASF playlist” option in menu 410 of FIG. 9 .
- the collection can be retrieved by the user (or other users) at a later time without having to go through another querying process.
- the collection of segments, stored as a media stream can itself be annotated.
- the collection of segments can be stored as a media stream in any of a variety of different locations and formats.
- the media stream can be stored in an additional data store (not shown) managed by annotation server 10 of FIG. 3 , or alternatively stored at media server 11 of FIG. I or another media server (not shown) of FIG. 1 .
- the media stream includes the source information, start time, and end time for each of the segments in the playlist. Thus, little storage space is required and the identifying information for each of the segments is independent of the annotations.
- the media stream includes pointers to each of the annotations. For subsequent retrieval of the media segments, the stored pointers can be used to retrieve each of the appropriate annotations, from which the corresponding media segments can be retrieved.
- the media segments themselves could be copied from media server 11 of FIG. I and those segments stored as the media stream.
- FIG. 10 shows one implementation of a graphical user interface window 450 that concurrently displays annotations and corresponding media segments.
- This UI window 450 has an annotation screen 454 , a media screen 456 , and a toolbar 240 .
- Media screen 456 is the region of the UI within which the multimedia content is rendered.
- the video is displayed on screen 456 .
- screen 456 displays static or dynamic images representing the content.
- audio content for example, a dynamically changing frequency wave that represents an audio signal is displayed in media screen 456 .
- Annotation screen 454 is the region of the UI within which the annotation identifiers and/or annotation content are rendered.
- dialog box 400 of FIG. 9 can be annotation screen 454 .
- FIG. 11 illustrates methodological aspects of one embodiment of the invention in retrieving and presenting annotations and media segments to a user.
- a step 500 comprises displaying a query dialog box 330 of FIG. 8 .
- Interface 150 of FIG. 3 provides dialog box 330 in response to a query request from a user, allowing the user to search for annotations that satisfy various user-definable criteria.
- a step 502 comprises receiving query input from the user.
- Interface 150 of FIG. 3 receives the user's input(s) to the query dialog box and provides the inputs to annotation server 10 of FIG. 3 .
- a step 504 comprises generating an annotation list.
- ABE 132 of FIG. 3 uses the user inputs to the query dialog box to select annotations from stores 17 and 18 .
- ABE 132 searches through annotation meta data store 18 for the annotations that satisfy the criteria provided by the user.
- the annotations that satisfy that criteria then become part of the annotation list and identifying information, such as the annotation titles or summaries, are provided to client 15 by annotation server 10 .
- a step 506 comprises displaying a view annotations dialog box 400 of FIG. 9 that contains the annotation identifying information from the annotation list generated in step 504 .
- Steps 508 and 510 comprise receiving user input selecting various annotations from the identifying information displayed in step 506 . Steps 508 and 510 repeat until the user has finished his or her selecting.
- a step 512 comprises retrieving the selected annotations and corresponding media segments.
- ABE 132 in annotation server 10 of FIG. 3 is responsible for retrieving the selected annotations from stores 17 and 18 .
- a step 514 comprises presenting the selected annotations and corresponding media segments to the user in a seamless manner.
- both the selected annotations as well as the corresponding media segments are provided to the user.
- only the media segments corresponding to the annotations (and not the annotations themselves) are provided to the user.
- only the annotations (and not the corresponding segments of the media stream) are provided to the user.
- the annotations are downloaded to the client computer first, and the media segments are downloaded to the client computer later in an on-demand manner.
- annotation data is buffered in annotation server 10 of FIG. 1 for provision to client 15 and media stream data is buffered in media server 11 for provision to client 15 .
- Sufficient buffering is provided to allow the annotation and media stream data to be provided to the client seamlessly. For example, when streaming two media segments to client 15 , as the end of the first media segment draws near media server 11 is working on obtaining and streaming the beginning of the second media segment to client 15 . By doing so, there is little or no noticeable gap between the first and second media segments as presented to the user.
- additional buffering can be provided by client 15 to allow the seamless presentation of the data.
- Annotations correspond to media segments of one or more multimedia streams. Identifiers of the annotations can be presented to a user and their order advantageously changed by the user to suit his or her desires. These annotation identifiers correspond to a playlist of media segments. The media segments (and/or the annotations themselves) of the playlist are then advantageously presented to the user in a seamless, contiguous manner.
Abstract
Description
- This application claims priority to U.S. Provisional Application No. 60/100,452, filed Sep. 15, 1998, entitled “Annotations for Streaming Video on the Web: System Design and Usage”, to Anoop Gupta and David M. Bargeron.
- This invention relates to networked client/server systems and to methods of delivering and rendering multimedia content in such systems. More particularly, the invention relates to systems and methods of selecting and providing such content.
- The advent of computers and their continued technological advancement has revolutionized the manner in which people work and live. An example of such is in the education field, wherein educational presentations (such as college lectures, workplace training sessions, etc.) can be provided to a computer user as multimedia data (e.g., video, audio, text, and/or animation data). Today, such presentations are primarily video and audio, but a richer, broader digital media era is emerging. Educational multimedia presentations provide many benefits, such as allowing the presentation data to be created at a single time yet be presented to different users at different times and in different locations throughout the world.
- These multimedia presentations are provided to a user as synchronized media. Synchronized media means multiple media objects that share a common timeline. Video and audio are examples of synchronized media-each is a separate data stream with its own data structure, but the two data streams are played back in synchronization with each other. Virtually any media type can have a timeline. For example, an image object can change like an animated .gif file, text can change and move, and animation and digital effects can happen over time. This concept of synchronizing multiple media types is gaining greater meaning and currency with the emergence of more sophisticated media composition frameworks implied by MPEG-4, Dynamic HTML, and other media playback environments.
- The term “streaming” is used to indicate that the data representing the various media types is provided over a network to a client computer on a real-time, as-needed basis, rather than being pre-delivered in its entirety before playback. Thus, the client computer renders streaming data as it is received from a network server, rather than waiting for an entire “file” to be delivered.
- Multimedia presentations may also include “annotations” relating to the multimedia presentation. An annotation is data (e.g., audio, text, video, etc.) that corresponds to a multimedia presentation. Annotations can be added by anyone with appropriate access rights to the annotation system (e.g., the lecturer/trainer or any of the students/trainees). These annotations typically correspond to a particular temporal location in the multimedia presentation and can provide a replacement for much of the “in-person” interaction and “classroom discussion” that is lost when the presentation is not made “in-person” or “live”. As part of an annotation, a student can comment on a particular point, to which another student (or lecturer) can respond in a subsequent annotation. This process can continue, allowing a “classroom discussion” to occur via these annotations. Additionally, some systems allow a user to select a particular one of these annotations and begin playback of the presentation starting at approximately the point in the presentation to which the annotation corresponds.
- However, current systems typically allow a user to select multimedia playback based only on individual annotations. This limitation provides a cumbersome process for the user, as he or she may wish to view several different portions of the presentation corresponding to several different annotations. Using current systems, the user would be required to undergo the painstaking process of selecting a first annotation, viewing/listening to the multimedia presentation corresponding to the first annotation, selecting a second annotation, viewing/listening to the multimedia presentation corresponding to the second annotation, selecting a third annotation, viewing/listening to the multimedia presentation corresponding to the third annotation, and so on through several annotations.
- The invention described below addresses this and other disadvantages of annotations, providing a way to improve multimedia presentation using annotations.
- Annotations correspond to media segments of one or more multimedia streams. A playlist generation interface is presented to the user in the form of annotation titles or summaries for a group of annotations. This group of annotations corresponds to the media segments that are part of a playlist. The playlist can then be altered by the user to suit his or her desires or needs by interacting with the annotation title/summary interface. The media segments of the playlist can then be presented to the user in a seamless, contiguous manner.
- According to one aspect of the invention, the ordering of the annotation titles/summaries can be altered by the user, resulting in a corresponding change in order of presentation of the media segments. The ordering of the annotation titles/summaries can be changed by moving the titles or summaries in a drag and drop manner.
- According to another aspect of the invention, the media segments of the playlist can themselves be stored as an additional multimedia stream. This additional multimedia stream can then be annotated in the same manner as other multimedia streams.
-
FIG. 1 shows a client/server network system and environment in accordance with one embodiment of the invention. -
FIG. 2 shows a general example of a computer that can be used as a client or server in accordance with the invention. -
FIG. 3 is a block diagram illustrating an annotation server and a client computer in more detail in accordance with one embodiment of the invention. -
FIG. 4 is a block diagram illustrating the structure for an annotation according to one embodiment of the invention. -
FIG. 5 is a block diagram illustrating exemplary annotation collections. -
FIG. 6 illustrates an annotation toolbar in accordance with one embodiment of the invention. -
FIG. 7 illustrates an “add new annotation” dialog box in accordance with one embodiment of the invention. -
FIG. 8 illustrates a “query annotations” dialog box in accordance with one embodiment of the invention. -
FIG. 9 illustrates a “view annotations” dialog box in accordance with one embodiment of the invention. -
FIG. 10 is a diagrammatic illustration of a graphical user interface window displaying annotations and corresponding media segments concurrently in accordance with one embodiment of the invention. -
FIG. 11 illustrates methodological aspects of one embodiment of the invention in retrieving and presenting annotations and media segments to a user. - General Network Structure
-
FIG. 1 shows a client/server network system and environment in accordance with one embodiment of the invention. Generally, the system includes multiplenetwork server computers network client computers 15. The computers communicate with each other over a data communications network. The communications network inFIG. 1 comprises apublic network 16 such as the Internet. The data communications network might also include, either in addition to or in place of the Internet, local-area networks and/or private wide-area networks. - Streaming
media server computer 11 has access to streaming media content in the form of different media streams. These media streams can be individual media streams (e.g., audio, video, graphical, etc.), or alternatively can be composite media streams including two or more of such individual streams. Some media streams might be stored as files in a database or other file storage system, while other media streams might be supplied to the server on a “live” basis from other data source components through dedicated communications channels or through the Internet itself. - There are various standards for streaming media content and composite media streams. The “Advanced Streaming Format” (ASF) is an example of such a standard, including both accepted versions of the standard and proposed standards for future adoption. ASF specifies the way in which multimedia content is stored, streamed, and presented by the tools, servers, and clients of various multimedia vendors. Further details about ASF are available from Microsoft Corporation of Redmond, Wash.
-
Annotation server 10 controls the storage of annotations and their provision toclient computers 15. Theannotation server 10 manages the annotationmeta data store 18 and theannotation content store 17. Theannotation server 10 communicates with theclient computers 15 via any of a wide variety of known protocols, such as the Hypertext Transfer Protocol (HTTP). Theannotation server 10 can receive and provide annotations via direct contact with aclient computer 15, or alternatively via electronic mail (email) viaemail server 13. Theannotation server 10 similarly communicates with theemail server 13 via any of a wide variety of known protocols, such as the Simple Mail Transfer Protocol (SMTP). - The annotations managed by
annotation server 10 correspond to the streaming media available frommedia server computer 11. In the discussions to follow, the annotations are discussed as corresponding to streaming media. However, it should be noted that the annotations can similarly correspond to “pre-delivered” rather than streaming media, such as media previously stored at theclient computers 15 via thenetwork 16, via removable magnetic or optical disks, etc. - When a user of a
client computer 15 accesses a web page containing streaming media, a conventional web browser of theclient computer 15 contacts theweb server 12 to get the Hypertext Markup Language (HTML) page, themedia server 11 to get the streaming data, and theannotation server 10 to get any annotations associated with that media. When a user of aclient computer 15 desires to add or retrieve annotations, theclient computer 15 contacts theannotation server 10 to perform the desired addition/retrieval. - Exemplary Computer Environment
- In the discussion below, the invention will be described in the general context of computer-executable instructions, such as program modules, being executed by one or more conventional personal computers. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. Moreover, those skilled in the art will appreciate that the invention may be practiced with other computer system configurations, including hand-held devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, and the like. In a distributed computer environment, program modules may be located in both local and remote memory storage devices.
-
FIG. 2 shows a general example of acomputer 20 that can be used as a client or server in accordance with the invention.Computer 20 is shown as an example of a computer that can perform the functions of any of server computers 10-13 or aclient computer 15 ofFIG. 1 . -
Computer 20 includes one or more processors orprocessing units 21, asystem memory 22, and abus 23 that couples various system components including thesystem memory 22 toprocessors 21. - The
bus 23 represents one or more of any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. The system memory includes read only memory (ROM) 24 and random access memory (RAM) 25. A basic input/output system (BIOS) 26, containing the basic routines that help to transfer information between elements withincomputer 20, such as during start-up, is stored inROM 24.Computer 20 further includes ahard disk drive 27 for reading from and writing to a hard disk, not shown, amagnetic disk drive 28 for reading from and writing to a removablemagnetic disk 29, and anoptical disk drive 30 for reading from or writing to a removableoptical disk 31 such as a CD ROM or other optical media. Thehard disk drive 27,magnetic disk drive 28, andoptical disk drive 30 are connected to thesystem bus 23 by anSCSI interface 32 or some other appropriate interface. The drives and their associated computer-readable media provide nonvolatile storage of computer readable instructions, data structures, program modules and other data forcomputer 20. Although the exemplary environment described herein employs a hard disk, a removablemagnetic disk 29 and a removableoptical disk 31, it should be appreciated by those skilled in the art that other types of computer readable media which can store data that is accessible by a computer, such as magnetic cassettes, flash memory cards, digital video disks, random access memories (RAMs) read only memories (ROM), and the like, may also be used in the exemplary operating environment. - A number of program modules may be stored on the hard disk,
magnetic disk 29,optical disk 31,ROM 24, orRAM 25, including anoperating system 35, one ormore application programs 36,other program modules 37, andprogram data 38. A user may enter commands and information intocomputer 20 through input devices such askeyboard 40 andpointing device 42. Other input devices (not shown) may include a microphone, joystick, game pad, satellite dish, scanner, or the like. These and other input devices are connected to theprocessing unit 21 through aninterface 46 that is coupled to the system bus. Amonitor 47 or other type of display device is also connected to thesystem bus 23 via an interface, such as avideo adapter 48. In addition to the monitor, personal computers typically include other peripheral output devices (not shown) such as speakers and printers. -
Computer 20 operates in a networked environment using logical connections to one or more remote computers, such as aremote computer 49. Theremote computer 49 may be another personal computer, a server, a router, a network PC, a peer device or other common network node, and typically includes many or all of the elements described above relative tocomputer 20, although only amemory storage device 50 has been illustrated inFIG. 2 . The logical connections depicted inFIG. 2 include a local area network (LAN) 51 and a wide area network (WAN) 52. Such networking environments are commonplace in offices, enterprise-wide computer networks, intranets, and the Internet. In the described embodiment of the invention,remote computer 49 executes an Internet Web browser program such as the “Internet Explorer” Web browser manufactured and distributed by Microsoft Corporation of Redmond, Washington. - When used in a LAN networking environment,
computer 20 is connected to thelocal network 51 through a network interface oradapter 53. When used in a WAN networking environment,computer 20 typically includes amodem 54 or other means for establishing communications over thewide area network 52, such as the Internet. Themodem 54, which may be internal or external, is connected to thesystem bus 23 via aserial port interface 33. In a networked environment, program modules depicted relative to thepersonal computer 20, or portions thereof, may be stored in the remote memory storage device. It will be appreciated that the network connections shown are exemplary and other means of establishing a communications link between the computers may be used. - Generally, the data processors of
computer 20 are programmed by means of instructions stored at different times in the various computer-readable storage media of the computer. Programs and operating systems are typically distributed, for example, on floppy disks or CD-ROMs. From there, they are installed or loaded into the secondary memory of a computer. At execution, they are loaded at least partially into the computer's primary electronic memory. The invention described herein includes these and other various types of computer-readable storage media when such media contain instructions or programs for implementing the steps described below in conjunction with a microprocessor or other data processor. The invention also includes the computer itself when programmed according to the methods and techniques described below. Furthermore, certain sub-components of the computer may be programmed to perform the functions and steps described below. The invention includes such sub-components when they are programmed as described. In addition, the invention described herein includes data structures, described below, as embodied on various types of memory media. - For purposes of illustration, programs and other executable program components such as the operating system are illustrated herein as discrete blocks, although it is recognized that such programs and components reside at various times in different storage components of the computer, and are executed by the data processor(s) of the computer.
- Client/Server Relationship
-
FIG. 3 illustrates an annotation server and a client computer in more detail. As noted above, generally, commands are formulated atclient computer 15 and forwarded toannotation server 10 via HTTP requests. In the illustrated embodiment ofFIG. 3 , communication betweenclient 15 andserver 10 is performed via HTTP, using commands encoded as Uniform Resource Locators (URLs) and data formatted as object linking and embedding (OLE) structured storage documents, or alternatively using Extensible Markup Language (XML). -
Client 15 includes an HTTP services (HttpSvcs)module 152, which manages communication withserver 10, and an annotation back end (ABE)module 151, which translates user actions into commands destined forserver 10. A user interface (MMA)module 150 provides the user interface (UI) for a user to add and select different annotations, and be presented with the annotations. According to one implementation, theuser interface module 150 supports ActiveX controls that display an annotation interface for streaming video on the Web. -
Client 15 also includes aweb browser module 153, which provides a conventional web browsing interface and capabilities for the user to access various servers vianetwork 16 ofFIG. 1 .Web browser 153 also provides the interface for a user to be presented with media streams. In addition to the use of playlists discussed below, the user can select which one of different versions of multimedia content he or she wishes to receive frommedia server 11 ofFIG. 1 . This selection can be direct (e.g., entry of a particular URL or selection of a “low resolution” option), or indirect (e.g., entry of a particular desired playback duration or an indication of system capabilities, such as “slow system” or “fast system”). Alternatively, other media presentation interfaces could be used. -
Annotation server 10 includes the Multimedia Annotation Web Server (MAWS)module 130, which is an Internet Services Application Programming Interface (ISAPI) plug-in for Internet Information Server (IIS)module 135. Together, these two modules provide the web server functionality ofannotation server 10.Annotation server 10 also includes anHTTP Services module 131 which manages communication withclient 15. In addition,annotation server 10 utilizes TheWindows Messaging Subsystem 134 to facilitate communication withemail server 13 ofFIG. 1 , and anemail reply server 133 for processing incoming email received fromemail server 13. -
Annotation server 10 further includes an annotation back end (ABE)module 132, which contains functionality for accessingannotation stores ABE module 132 by theEmail Reply Server 133. Annotation content authored atclient 15, usinguser interface 150, is received byABE 132 and maintained inannotation content store 17. Received meta data (control information) corresponding to the annotation content is maintained in annotationmeta data store 18. The annotation content and meta data can be stored in any of a variety of conventional manners, such as in SQL relational databases (e.g., using Microsoft “SQL Server” version 7.0, available from Microsoft Corporation).Annotation server 10 is illustrated inFIG. 3 as maintaining the annotation content and associated control information (meta data) separately in two different stores. Alternatively, all of the annotation data (content and meta information) can be stored together in a single store, or content may be stored by another distinct storage system on thenetwork 16 ofFIG. 1 , such as a file system, media server, email server, or other data store. -
ABE 132 ofannotation server 10 also manages the interactive generation and presentation of streaming media data fromserver computer 11 ofFIG. 1 using “playlists”. A “playlist” is a listing of one or more multimedia segments to be retrieved and presented in a given order. Each of the multimedia segments in the playlist is defined by a source identifier, a start time, and an end time. The source identifier identifies which media stream the segment is part of, the start time identifies the temporal location within the media stream where the segment begins, and the end time identifies the temporal location within the media stream where the segment ends. -
ABE 132 allows playlists to be generated interactively based on annotations maintained inannotation stores ABE 132 provides a user atclient 15 with multiple possible annotation identifiers (e.g., titles or summaries) from which the user can select those of interest to him or her. Based on the selected annotations,ABE 132 coordinates provision of the associated media segments to the user.ABE 132 can directly communicate withvideo server computer 11 to identify which segments are to be provided, or alternatively can provide the appropriate information to the browser ofclient computer 15, which in turn can request the media segments fromserver computer 11. -
FIG. 4 shows an exemplary structure for anannotation entry 180 that is maintained byannotation server 10 in annotationmeta data store 18 ofFIG. 3 . In the illustrated embodiment, anannotation entry 180 includes anauthor field 182, atime range field 184, atime units field 186, acreation time field 188, atitle field 190, acontent field 192, anidentifier field 194, a relatedannotation identifier field 196, a set identifier(s)field 198, a mediacontent identifier field 200, an arbitrary number of user-definedproperty fields 202, and asequence number 204. Each of fields 182-204 is a collection of data which define a particular characteristic ofannotation entry 180.Annotation entry 180 is maintained byannotation server 10 ofFIG. 3 in annotationmeta data store 18.Content field 192, as discussed in more detail below, includes a pointer to (or other identifier of) the annotation content, which in turn is stored inannotation content store 17. -
Author field 182 contains data identifying the user who createdannotation entry 180 and who is therefore the author of the annotation. The author is identified byABE 151 ofFIG. 3 based on the user logged intoclient 15 at the time the annotation is created. -
Time range field 184 contains data representing “begin” and “end” times defining a segment of media timeline to whichannotation entry 180 is associated. Time units field 186 contains data representing the units of time represented intime range field 184. Together,time range field 184 and time units field 186 identify the relative time range of the annotation represented byannotation entry 180. This relative time range corresponds to a particular segment of the media stream to whichannotation entry 180 is associated. The begin and end times for the annotation are provided by the user viainterface 150 ofFIG. 3 , or alternatively can be automatically or implicitly derived using a variety of audio and video signal processing techniques, such as sentence detection in audio streams or video object tracking. - It should be noted that the time ranges for different annotations can overlap. Thus, for example, a first annotation may correspond to a segment ranging between the first and fourth minutes of media content, a second annotation may correspond to a segment ranging between the second and seventh minutes of the media content, and a third annotation may correspond to a segment ranging between the second and third minutes of the media content.
- Alternatively, rather than using the presentation timeline of the media content, different media characteristics can be used to associate the annotation with a particular segment(s) of the media content. For example, annotations could be associated with (or “anchored” on) specific objects in the video content, or specific events in the audio content.
-
Creation time field 188 contains data specifying the date and time at whichannotation entry 180 is created. It should be noted that the time of creation ofannotation entry 180 is absolute and is not relative to the video or audio content of the media stream to whichannotation entry 180 is associated. Accordingly, a user can specify that annotations which are particularly old, e.g., created more than two weeks earlier, are not to be displayed.ABE 132 ofFIG. 3 stores the creation time and date when the annotation is created. -
Title field 190 contains data representing a title by which the annotation represented byannotation entry 180 is identified. The title is generally determined by the user and the user enters the data representing the title using conventional and well known user interface techniques. The data can be as simple as ASCII text or as complex as HTML code which can include text having different fonts and type styles, graphics including wallpaper, motion video images, audio, and links to other multimedia documents. -
Content field 192 contains data representing the substantive content of the annotation as authored by the user. The actual data can be stored incontent field 192, or alternativelycontent field 192 may store a pointer to (or other indicator of) the content that is stored separately from theentry 180 itself. In the illustrated example,content field 192 includes a pointer to (or other identifier of) the annotation content, which in turn is stored inannotation content store 17. The user enters the data representing the content using conventional and well known user interface techniques. The content added by the user in creatingannotation entry 180 can include any one or more of text, graphics, video, audio, etc. or links thereto. In essence,content field 192 contains data representing the substantive content the user wishes to include with the presentation of the corresponding media stream at the relative time range represented bytime range field 184 andtime units field 186. -
Annotation identifier field 194 stores data that uniquely identifiesannotation entry 180, while relatedannotation identifier field 196 stores data that uniquely identifies a related annotation.Annotation identifier field 194 can be used by other annotation entries to associate such other annotation entries withannotation entry 180. In this way, threads of discussion can develop in which a second annotation responds to a first annotation, a third annotation responds to the second annotation and so on. By way of example, an identifier of the first annotation would be stored in relatedannotation identifier field 196 of the second annotation, an identifier of the second annotation would be stored in relatedannotation identifier field 196 of the third annotation, and so on. - Set identifier(s)
field 198 stores data that identifies a particular one or more sets to whichannotation entry 180 belongs. A media stream can have multiple sets of annotations, sets can span multiple media content, and a particular annotation can correspond to one or more of these sets. Which set(s) an annotation belongs to is identified by the author of the annotation. By way of example, a media stream corresponding to a lecture may include the following sets: “instructor's comments”, “assistant's comments”, “audio comments”, “text comments”, “student questions”, and each student's personal comments. - Media
content identifier field 200 contains data that uniquely identifies particular multimedia content as the content to whichannotation entry 180 corresponds.Media content identifier 200 can identify a single media stream (either an individual stream or a composite stream), or alternatively identify multiple different streams that are different versions of the same media content.Media content identifier 200 can identify media versions in a variety of different manners. According to one embodiment, the data represents a real-time transport protocol (RTP) address of the different media streams. An RTP address is a type of uniform resource locator (URL) by which multimedia documents can be identified in a network. According to an alternate embodiment, a unique identifier is assigned to the content rather than to the individual media streams. According to another alternate embodiment, a different unique identifier of the media streams could be created byannotation server 10 ofFIG. 3 and assigned to the media streams. Such a unique identifier would also be used by streamingmedia server 11 ofFIG. 1 to identify the media streams. According to another alternate embodiment, a uniform resource name (URN) such as those described by K. Sollins and L. Mosinter in “Functional Requirements for Uniform Resource Names,” IETF RFC 1733 (December 1994) could be used to identify the media stream. - User-defined
property fields 202 are one or more user-definable fields that allow users (or user interface designers) to customize the annotation system. Examples of such additional property fields include a “reference URL” property which contains the URL of a web page used as reference material for the content of the annotation; a “help URL” property containing the URL of a help page which can be accessed concerning the content of the annotation; a “view script” property containing JavaScript which is to be executed whenever the annotation is viewed; a “display type” property, which gives the client user interface information about how the annotation is to be displayed; etc. -
Sequence number 204 allows a user to define (viauser interface 150 ofFIG. 3 ) a custom ordering for the display of annotation identifiers, as discussed in more detail below.Sequence number 204 stores the relative position of the annotations with respect to one another in the custom ordering, allowing the custom ordering to be saved for future used. In the illustrated example,annotation entry 180 stores a single sequence number. Alternatively,multiple sequence numbers 204 may be included inannotation entry 180 each corresponding to a different custom ordering, or a different annotation set, or a different user, etc. -
FIG. 5 illustrates exemplary implicit annotation collections for annotations maintained byannotation server 10 ofFIG. 3 . A collection of annotations refers toannotation entries 180 ofFIG. 4 that correspond to the same media stream(s), based on themedia content identifier 200.Annotation entries 180 can be viewed conceptually as part of the same annotation collection if they have the samemedia content identifiers 200, even though the annotation entries may not be physically stored together byannotation server 10. -
Annotation database 206 includes twoannotation collections Annotation server 10 dynamically adds, deletes, and modifies annotation entries inannotation database 206 based on commands fromclient 15. Annotation entries can be created and added toannotation database 206 at any time a user cares to comment upon the content of the stream (or another annotation) in the form of an annotation.Annotation server 10 forms an annotation entry from identification data, content data, title data, and author data of an “add annotation” request received from the client's ABE 151 (FIG. 3 ), and adds the annotation entry toannotation database 206. -
Annotation database 206 includes afields database 206 or anannotation collection annotation entry 180. -
Creator field 212 contains data identifying the user who was responsible for creatingannotation database 206. - RTP address fields 214 and 216 contain data representing an RTP address of the media stream (e.g., the RTP address of the stream identified in
media content identifier 200 ofFIG. 5 ) for the annotation collection. An RTP address provides an alternative mechanism, in addition to the data inidentifier field 200, for associating the media stream withannotation entries 180. In alternative embodiments, RTP address fields 214 and 216 need not be included, particularly embodiments in whichmedia content identifier 200 contains the RTP address of the media stream. - User Interface
- An annotation can be created by a user of any of the
client computers 15 ofFIG. 1 . As discussed above with reference toFIG. 3 ,client 15 includes aninterface module 150 that presents an interface to a user (e.g., a graphical user interface), allowing a user to make requests ofannotation server 10. In the illustrated embodiment, a user can accessannotation server 10 via an annotation toolbar provided byinterface 150. -
FIG. 6 illustrates an annotation toolbar in accordance with one embodiment of the invention.Annotation toolbar 240 includes various identifying information and user-selectable options 242-254. - Selection of an exit or “X”
button 242 causes interface 150 to terminate display of thetoolbar 240. Aserver identifier 244 identifies the annotation server with whichclient 15 is currently configured to communicate (annotation server 10 ofFIG. 1 . in the illustrated embodiment). - Selection of a
connection button 246 causesABE 151 ofFIG. 3 to establish a connection with the annotation server identified byidentifier 244. Selection of aquery button 248 causesinterface module 150 to open a “query” dialog box, from which a user can search for particular annotations. Selection of anadd button 250 causesinterface module 150 to open an “add new annotation” dialog box, from which a user can create a new annotation. - Selection of a
show annotations button 252 causesinterface module 150 to open a “view annotations” dialog box, from which a user can select particular annotations for presentation. - Selection of a
preferences button 254 causes interface 150 ofFIG. 3 to open a “preferences” dialog box, from which a user can specify various UI preferences, such as an automatic server query refresh interval, or default query criteria values to be persisted between sessions. - Annotation Creation
-
FIG. 7 shows an “add new annotation”dialog box 260 that results from user selection ofadd button 250 ofFIG. 6 to create a new annotation.Interface 150 assumes that the current media stream being presented to the user is the media stream to which this new annotation will be associated. The media stream to which an annotation is associated is referred to as the “target” of the annotation. An identifier of this stream is displayed in atarget specification area 262 ofdialog box 260. Alternatively, a user could change the target of the annotation, such as by typing in a new identifier intarget area 262, or by selection of a “browse” button (not shown) that allows the user to browse through different directories of media streams. - A
time strip 264 is also provided as part ofdialog box 260.Time strip 264 represents the entire presentation time of the corresponding media stream. Athumb 265 that moves withintime strip 264 indicates a particular temporal position within the media stream. The annotation being created viadialog box 260 has a begin time and an end time, which together define a particular segment of the media stream. When “from”button 268 is selected,thumb 265 represents the begin time for the segment relative to the media stream. When “to”button 271 is selected,thumb 265 represents the end time for the segment relative to the media stream. Alternatively, two different thumbs could be displayed, one for the begin time and one for the end time. The begin and end times are also displayed in an hours/minutes/seconds format inboxes -
Thumb 265 can be moved alongtime strip 264 in any of a variety of conventional manners. For example, a user can depress a button of a mouse (or other cursor control device) while a pointer is “on top” ofthumb 265 and move the pointer alongtime strip 264, causingthumb 265 to move along with the pointer; The appropriate begin or end time is then set when the mouse button is released. Alternatively, the begin and end times can be set by entering (e.g., via an alphanumeric keyboard) particular times inboxes -
Dialog box 260 also includes a “play”button 274. Selection ofplay button 274 causesinterface module 150 ofFIG. 3 to forward a segment specification toweb browser 153 ofclient 15. The segment specification includes the target identifier fromtarget display 262 and the begin and end times fromboxes interface module 150, the browser communicates withmedia server 11 and requests the identified media segment using conventional HTTP requests. In response,media server 11 streams the media segment toclient 15 for presentation to the user. This presentation allows, for example, the user to verify the portion of the media stream to which his or her annotation will correspond. -
Dialog box 260 also includes anannotation set identifier 272, anemail field 275, and asummary 276.Annotation set identifier 272 allows the user to identify a named set to which the new annotation will belong. This set can be a previously defined set, or a new set being created by the user. Selection of the particular set can be made from a drop-down menu activated by selection of abutton 273, or alternatively can be directly input by the user (e.g., typed in using an alphanumeric keyboard). According to one embodiment of the invention,annotation server 10 ofFIG. 3 supports read and write access controls, allowing the creator of the set to identify which users are able to read and/or write to the annotation set. In this embodiment, only those sets for which the user has write access can be entered asset identifier 272. -
Email identifier 275 allows the user to input the email address of a recipient of the annotation. When an email address is included, the newly created annotation is electronically mailed to the recipient indicated inidentifier 275 in addition to being added to the annotation database. Furthermore, the recipient of the electronic mail message can reply to the message to create an additional annotation. To enable this, the original email message is configured withannotation server 10 as the sender. Because of this, a “reply to sender” request from the recipient will cause an email reply to be sent toannotation server 10. Upon receipt of such an electronic mail message reply,annotation server 10 creates a new annotation and uses the reply message content as the content of the new annotation. This new annotation identifies, as a related annotation, the original annotation that was created when the original mail message was sent byannotation server 10. In the illustrated embodiment, this related annotation identifier is stored infield 196 ofFIG. 4 . -
Summary 276 allows the user to provide a short summary or title of the annotation content. Although the summary is illustrated as being text, it could include any of a wide variety of characters, alphanumerics, graphics, etc. In the illustrated embodiment,summary 276 is stored in thetitle field 190 of the annotation entry ofFIG. 4 . -
Dialog box 260 further includesradio buttons audio display bar 279 can be included to provide visual progress feedback when the audio is playing or recording. - The exact nature of input controls 278 is dependent on the type of annotation content being provided. In the case of text content, input controls 278 may simply include a box into which text can be input by the user via an alphanumeric keyboard. Additionally, a keyboard layout may also be provided to the user, allowing him or her to “point and click” using a mouse and pointer to select particular characters for entry.
- Annotation and Media Segment Retrieval
-
FIG. 8 shows a “query annotations”dialog box 330 that results from a user selectingquery button 248 ofFIG. 6 . Many of the options presented to the user indialog box 330 are similar to those presented in the “add new annotation”dialog box 260 ofFIG. 7 , however, those indialog box 330 are used as search criteria rather than data for a new annotation. -
Dialog box 330 includes atarget display 332 that contains an identifier of the target stream. This identifier can be input in any of a variety of manners, such as by typing in a new identifier intarget display 332, or by selection of a “browse” button (not shown) that allows the user to browse through different directories of media streams. In the illustrated embodiment, the identifier is an URL. However, alternate embodiments can use different identifier formats. -
Dialog box 330 also includestarget information 334, which includes a time strip, thumb, “from” button, “to” button, “play” button, and begin and end times, which are analogous to the time strip, thumb, “from” button, “to” button, “play” button, begin and end times ofdialog box 260 ofFIG. 7 . The begin and end times intarget information 334 limit the query for annotations to only those annotations having a time range that corresponds to at least part of the media segment between the begin and end times oftarget information 334. -
Dialog box 330 also includes anannotation set list 336.Annotation set list 336 includes a listing of the various sets that correspond to the target media stream. According to one implementation, only those sets for which an annotation has been created are displayed inset list 336. According to one embodiment of the invention,annotation server 10 ofFIG. 3 supports read and write security, allowing the creator of the set to identify which users are able to read and/or write to the annotation set. In this embodiment, only those sets for which the user has read access are displayed inset list 336. - A user can select sets from annotation set
list 336 in a variety of manners. For example, using a mouse and pointer to “click” on a set inlist 336, which highlights the set to provide feedback to the user that the set has been selected. Clicking on the selected set again de-selects the set (leaving it no longer highlighted). Additionally, a “select all”button 338 allows a user to select all sets inset list 336, while a “deselect all”button 340 allows a user to de-select all sets inset list 336. - In the illustrated embodiment, the sets displayed as part of annotation set
list 336 contain annotations which correspond to the target identifier intarget display 332. However, in alternate embodiments the sets inselection list 338 need not necessarily contain annotations which correspond to the target identifier intarget display 332.Interface module 150 allows a user to select different target streams during the querying process. Thus, a user may identify a first target stream and select one or more sets to query annotations from for the first target stream, and then identify a second target stream and select one or more sets to query annotations from for the second target stream. - Additional search criteria can also be input by the user. As illustrated, a particular creation date and
time identifier 342 can be input, along with a relation 344 (e.g., “after” or “before”). Similarly, particular words, phrases, characters, graphics, etc. that must appear in the summary can be input in a summarykeyword search identifier 346. A maximum number of annotations to retrieve in response to the query can also be included as amax identifier 348. Furthermore, the query can be limited to only annotations that correspond to the target identifier intarget display 332 by selectingcheck box 360. - A level of
detail 350 to retrieve can also be selected by the user. Examples of different levels that could be retrieved include the “full level” (that is, all content of the annotation), or a “deferred download” where only an identifier of the annotations (e.g., a summary or title) is downloaded. In the illustrated example, selection ofcheckbox 354 selects the deferred download level, whereas ifcheckbox 354 is not selected then the full level of detail is implicitly selected. - A
server identifier 356 identifies the annotation server with whichclient 15 is currently configured to communicate. Different annotation servers can be selected by the user by inputting the appropriate identifier asserver identifier 356. This input can be provided in any of a variety of manners, such as by typing in a new identifier inserver identifier 356 or by selection of a “browse” button (not shown) that allows the user to browse through different directories of annotation servers. - A user can request automatic display of the retrieved annotations by selecting a “display retrieved annotations”
checkbox 358. Selection of “advanced”button 362 reduces the number of options available to the user, simplifyingdialog box 330. For example, the simplified dialog box may not includefields - The user can then complete the query process by selecting a
query button 364. Upon selection of thequery button 364,interface 150 closes thequery dialog box 330 and forwards the search criteria toannotation server 10. Additionally, ifcheckbox 358 is selected then interface 150 displays a “view annotations”dialog box 400 ofFIG. 9 . Alternatively, a user can provide a view request, causinginterface 150 to displaydialog box 400, by selectingshow annotations button 252 inannotation toolbar 240 ofFIG. 6 . -
FIG. 9 shows adialog box 400 that identifies annotations corresponding to a playlist of media segments. The playlist is a result of the query input by the user as discussed above with reference toFIG. 8 . In the illustration ofFIG. 9 , annotation identifiers in the form ofuser identifiers 406 andsummaries 408 are displayed within anannotation listing box 402. The user can scroll through annotation identifiers in a conventional manner viascroll bars annotation listing box 402 according to a default criteria, such as chronological by creation time/date, by user, alphabetical by summaries, etc. - Related annotations are displayed in an
annotation listing 402 in a hierarchical, horizontally offset manner. The identifier of an annotation that is related to a previous annotation is “indented” from that previous annotation's identifier and a connecting line between the two identifiers is shown. -
Dialog box 400 can be displayed concurrently with a multimedia player that is presenting multimedia content that corresponds to the annotations in annotation listing 402 (e.g., as illustrated inFIG. 10 below).Interface module 150 can have the annotations “track” the corresponding multimedia content being played back, so that the user is presented with an indication (e.g., an arrow) as to which annotation(s) correspond to the current temporal position of the multimedia content. Such tracking can be enabled by selectingcheckbox 422, or disabled by de-selectingcheckbox 422. -
Dialog box 400 also includes a merge annotation sets checkbox 424. Selection of merge annotation sets checkbox 424 causesinterface module 150 to present annotation identifiers inlisting box 402 in a chronological order regardless of what set(s) the annotations inannotation listing 402 belong to. Ifcheckbox 424 is not selected, then annotations from different sets are grouped and displayed together in annotation listing 402 (e.g., under the same tree item). Thus, whencheckbox 424 is not selected,interface 150 displays one playlist for each annotation set that has been retrieved fromannotation server 10. -
Dialog box 400 also includes arefresh button 428, aclose button 430, and anadvanced button 432. Selection ofrefresh button 428 causesinterface module 150 to communicate with annotationback end 151 to accessannotation server 10 and obtain any additional annotations that correspond to the query that resulted inlisting box 402. - Selection of
close button 430 causes interface 150 to terminate the display ofdialog box 400. Selection ofadvanced button 432 causes interface 150 to display a different view annotations box having additional details, such as annotation target information (analogous to targetdisplay 332 discussed below with reference toFIG. 8 ), user-selectable preferences for information displayed as annotation identifiers inlisting box 402, etc. - Upon user selection of a particular annotation identifier from listing box 402 (e.g., “single clicking” on the summary), preview information is presented in a
preview section 416, and a selection box ormenu 410 is provided. The exact nature of the preview information is dependent on the data type and amount of information that was requested (e.g., as identified in level ofdetail 350 ofFIG. 8 ). -
Menu 410 includes the following options: play, export ASX playlist, export annotations, time order, custom order, save, and reset. Selection of the “play” option causes playback of the multimedia content to begin starting with the selected annotation inannotation list 402. Selection of the “export ASX playlist” option causesannotation backend 151 to output a record (e.g., create a file) that identifies the temporal segments of multimedia content that the annotations identified inlist 402 correspond to, as determined by the begin and end times of the annotations. Selection of the “export annotations” option causesannotation backend 151 to output a record (e.g., create a file) that includes the annotation content of each annotation identified inlist 402. - Selection of the “time order” option causes
interface module 150 to display the identifiers inlist 402 in chronological order based on the begin time for each annotation. Selection of the “custom order” option allows the user to identify some other criteria to be used in determining the order of the identifiers in list 402 (e.g., identifiers can be re-ordered in a conventional drag and drop manner). Re-ordering annotation identifiers causes the sequence numbers 204 (ofFIG. 4 ) of the annotations to be re-ordered accordingly. Selection of the “save” option causesinterface module 150 to save the current custom ordering toannotation server 10 ofFIG. 3 by saving the current sequence numbers of the annotations. Selection of the “reset” option causesinterface module 150 to ignore any changes that have been made since the last saved custom ordering and revert to the last saved custom ordering. - Transfer of the corresponding media segments (and/or the annotations) to
client 15 is initiated when the “play” option ofmenu 410 is selected. Upon selection of the play option,interface 150 ofFIG. 3 provides the list of annotation identifiers being displayed to web browser 153 (or other multimedia presentation application) in the order of their display, including the target identifier and temporal range information. Thus,web browser 153 receives a list of multimedia segments that it is to present to the user in a particular order.Web browser 153 then accessesmedia server 11 to stream the multimedia segments toclient 15 for presentation in that order. By use of the play option inmenu 410, a user is able to review the information regarding the annotations that satisfy his or her search criteria and then modify the annotation playlist (e.g., by deleting or reordering annotation identifiers) before the corresponding media segments (and/or the annotations) are presented to him or her. - Alternatively, transfer of the media segments may be initiated in other manners rather than by selection of the play option in
menu 410. For example, a “start” button may be included as part ofdialog box 400, selection of which initiates transfer of the media segments toclient 15. - The annotations and/or corresponding media segments are presented to the user “back to back” with very little or no noticeable gap between different annotations and between different segments. Thus, the presentation of the annotations and/or media segments is “seamless”.
- A user is able to reorder the media segments of the playlist and thereby alter their order of presentation. In the illustrated embodiment, media segments are reordered by changing the ordering of the annotation identifiers in
annotation listing 402 in a drag and drop manner. For example, using a mouse and pointer a user can select a particular annotation identifier (e.g., identifier 420) and drag it to a different location within the dialog box (e.g., betweenidentifiers 419 and 421), thereby changing when the media segment corresponding to the annotation identified byidentifier 420 is presented relative to the other annotations. - As discussed above, information regarding the media stream as well as the particular media segment within that stream to which an annotation corresponds is maintained in each annotation. At the appropriate time,
web browser 153 sends a message to theappropriate media server 11 ofFIG. 1 to begin streaming the appropriate segment toclient computer 15.Web browser 153, knowing the duration of each of the segments being provided toclient computer 15, forwards additional messages tomedia server 11 to continue with the provision of the next segment, according to the playlist, when appropriate. By managing the delivery of the media segments toclient computer 15 in such a manner,web browser 153 can keep the media segments being provided to the user in a seamless manner. - According to an alternate embodiment, the media segments could be streamed to
annotation server 10 for temporary buffering and subsequent streaming toclient computer 15. According to another alternate embodiment, identifying information (e.g., source, start time, and end time) for the media segment could be provided tomedia server 11 fromannotation server 10 for streaming toclient computer 15. - Additionally, according to one embodiment the collection of media segments identified by the playlist can be stored as an additional media stream by selecting “export ASF playlist” option in
menu 410 ofFIG. 9 . By saving the collection of media segments as a single media stream, the collection can be retrieved by the user (or other users) at a later time without having to go through another querying process. Furthermore, the collection of segments, stored as a media stream, can itself be annotated. - The collection of segments can be stored as a media stream in any of a variety of different locations and formats. The media stream can be stored in an additional data store (not shown) managed by
annotation server 10 ofFIG. 3 , or alternatively stored atmedia server 11 of FIG. I or another media server (not shown) ofFIG. 1 . According to one embodiment, the media stream includes the source information, start time, and end time for each of the segments in the playlist. Thus, little storage space is required and the identifying information for each of the segments is independent of the annotations. Alternatively, the media stream includes pointers to each of the annotations. For subsequent retrieval of the media segments, the stored pointers can be used to retrieve each of the appropriate annotations, from which the corresponding media segments can be retrieved. According to another alternate embodiment, the media segments themselves could be copied frommedia server 11 of FIG. I and those segments stored as the media stream. -
FIG. 10 shows one implementation of a graphicaluser interface window 450 that concurrently displays annotations and corresponding media segments. ThisUI window 450 has anannotation screen 454, amedia screen 456, and atoolbar 240. -
Media screen 456 is the region of the UI within which the multimedia content is rendered. For video content, the video is displayed onscreen 456. For non-visual content,screen 456 displays static or dynamic images representing the content. For audio content, for example, a dynamically changing frequency wave that represents an audio signal is displayed inmedia screen 456. -
Annotation screen 454 is the region of the UI within which the annotation identifiers and/or annotation content are rendered. For example,dialog box 400 ofFIG. 9 can beannotation screen 454. -
FIG. 11 illustrates methodological aspects of one embodiment of the invention in retrieving and presenting annotations and media segments to a user. - A
step 500 comprises displaying aquery dialog box 330 ofFIG. 8 . Interface 150 ofFIG. 3 providesdialog box 330 in response to a query request from a user, allowing the user to search for annotations that satisfy various user-definable criteria. - A
step 502 comprises receiving query input from the user. Interface 150 ofFIG. 3 receives the user's input(s) to the query dialog box and provides the inputs toannotation server 10 ofFIG. 3 . - A
step 504 comprises generating an annotation list.ABE 132 ofFIG. 3 uses the user inputs to the query dialog box to select annotations fromstores ABE 132 searches through annotationmeta data store 18 for the annotations that satisfy the criteria provided by the user. The annotations that satisfy that criteria then become part of the annotation list and identifying information, such as the annotation titles or summaries, are provided toclient 15 byannotation server 10. - A
step 506 comprises displaying a viewannotations dialog box 400 ofFIG. 9 that contains the annotation identifying information from the annotation list generated instep 504. -
Steps step 506.Steps - A
step 512 comprises retrieving the selected annotations and corresponding media segments.ABE 132 inannotation server 10 ofFIG. 3 is responsible for retrieving the selected annotations fromstores - A
step 514 comprises presenting the selected annotations and corresponding media segments to the user in a seamless manner. - In the illustrated embodiment, both the selected annotations as well as the corresponding media segments are provided to the user. In one alternate embodiment, only the media segments corresponding to the annotations (and not the annotations themselves) are provided to the user. In another alternate embodiment only the annotations (and not the corresponding segments of the media stream) are provided to the user. In another embodiment, the annotations are downloaded to the client computer first, and the media segments are downloaded to the client computer later in an on-demand manner.
- In the illustrated embodiment, annotation data is buffered in
annotation server 10 ofFIG. 1 for provision toclient 15 and media stream data is buffered inmedia server 11 for provision toclient 15. Sufficient buffering is provided to allow the annotation and media stream data to be provided to the client seamlessly. For example, when streaming two media segments toclient 15, as the end of the first media segment draws nearmedia server 11 is working on obtaining and streaming the beginning of the second media segment toclient 15. By doing so, there is little or no noticeable gap between the first and second media segments as presented to the user. Alternatively, rather than providing such buffering in theservers client 15 to allow the seamless presentation of the data. - The invention described above provides interactive playlist generation using annotations. Annotations correspond to media segments of one or more multimedia streams. Identifiers of the annotations can be presented to a user and their order advantageously changed by the user to suit his or her desires. These annotation identifiers correspond to a playlist of media segments. The media segments (and/or the annotations themselves) of the playlist are then advantageously presented to the user in a seamless, contiguous manner.
- Although the invention has been described in language specific to structural features and/or methodological steps, it is to be understood that the invention defined in the appended claims is not necessarily limited to the specific features or steps described. Rather, the specific features and steps are disclosed as preferred forms of implementing the claimed invention.
Claims (18)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US11/522,160 US7631015B2 (en) | 1997-03-14 | 2006-09-14 | Interactive playlist generation using annotations |
Applications Claiming Priority (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US08/818,804 US6006241A (en) | 1997-03-14 | 1997-03-14 | Production of a video stream with synchronized annotations over a computer network |
US10045298P | 1998-09-15 | 1998-09-15 | |
US09/396,702 US7111009B1 (en) | 1997-03-14 | 1999-09-15 | Interactive playlist generation using annotations |
US11/522,160 US7631015B2 (en) | 1997-03-14 | 2006-09-14 | Interactive playlist generation using annotations |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US09/396,702 Continuation US7111009B1 (en) | 1997-03-14 | 1999-09-15 | Interactive playlist generation using annotations |
Publications (2)
Publication Number | Publication Date |
---|---|
US20070011206A1 true US20070011206A1 (en) | 2007-01-11 |
US7631015B2 US7631015B2 (en) | 2009-12-08 |
Family
ID=36974623
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US09/396,702 Expired - Fee Related US7111009B1 (en) | 1997-03-14 | 1999-09-15 | Interactive playlist generation using annotations |
US11/522,160 Expired - Fee Related US7631015B2 (en) | 1997-03-14 | 2006-09-14 | Interactive playlist generation using annotations |
Family Applications Before (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US09/396,702 Expired - Fee Related US7111009B1 (en) | 1997-03-14 | 1999-09-15 | Interactive playlist generation using annotations |
Country Status (1)
Country | Link |
---|---|
US (2) | US7111009B1 (en) |
Cited By (26)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20050081159A1 (en) * | 1998-09-15 | 2005-04-14 | Microsoft Corporation | User interface for creating viewing and temporally positioning annotations for media content |
US20050120127A1 (en) * | 2000-04-07 | 2005-06-02 | Janette Bradley | Review and approval system |
US20060161838A1 (en) * | 2005-01-14 | 2006-07-20 | Ronald Nydam | Review of signature based content |
US20060195790A1 (en) * | 2005-02-28 | 2006-08-31 | Yahoo! Inc. | Method and system for exploring similarities |
US20060265477A1 (en) * | 2000-11-10 | 2006-11-23 | Alan Bartholomew | Method and apparatus for creating and posting media |
US20070136656A1 (en) * | 2005-12-09 | 2007-06-14 | Adobe Systems Incorporated | Review of signature based content |
US20070276852A1 (en) * | 2006-05-25 | 2007-11-29 | Microsoft Corporation | Downloading portions of media files |
US20080235589A1 (en) * | 2007-03-19 | 2008-09-25 | Yahoo! Inc. | Identifying popular segments of media objects |
US7448550B1 (en) | 2004-06-30 | 2008-11-11 | Adobe Systems Incorporated | Enabling the use of machine-readable codes |
US20080313227A1 (en) * | 2007-06-14 | 2008-12-18 | Yahoo! Inc. | Method and system for media-based event generation |
US20090187825A1 (en) * | 2008-01-23 | 2009-07-23 | Microsoft Corporation | Annotating and Sharing Content |
US7631015B2 (en) | 1997-03-14 | 2009-12-08 | Microsoft Corporation | Interactive playlist generation using annotations |
US7685522B1 (en) * | 2003-11-03 | 2010-03-23 | Adobe Systems Incorporated | Self-describing forms |
US7954049B2 (en) | 2006-05-15 | 2011-05-31 | Microsoft Corporation | Annotating multimedia files along a timeline |
US20120159324A1 (en) * | 2010-12-17 | 2012-06-21 | General Electric Company | Systems and methods for software state capture and playback |
US20130097145A1 (en) * | 1998-11-30 | 2013-04-18 | Gemstar Development Corporation | Search engine for video and graphics |
US9043403B2 (en) * | 2013-05-10 | 2015-05-26 | Google Inc. | System and method for providing visual representations of email to enable efficient email processing |
US9125169B2 (en) | 2011-12-23 | 2015-09-01 | Rovi Guides, Inc. | Methods and systems for performing actions based on location-based rules |
US9294799B2 (en) | 2000-10-11 | 2016-03-22 | Rovi Guides, Inc. | Systems and methods for providing storage of data on servers in an on-demand media delivery system |
WO2016196692A1 (en) * | 2015-06-01 | 2016-12-08 | Miller Benjamin Aaron | Break state detection in content management systems |
US20160364103A1 (en) * | 2015-06-11 | 2016-12-15 | Yaron Galant | Method and apparatus for using gestures during video playback |
US10068568B2 (en) | 2015-06-01 | 2018-09-04 | Sinclair Broadcast Group, Inc. | Content segmentation and time reconciliation |
US10394942B1 (en) * | 2008-07-01 | 2019-08-27 | Google Llc | Method and system for contextually placed chat-like annotations |
US10855765B2 (en) | 2016-05-20 | 2020-12-01 | Sinclair Broadcast Group, Inc. | Content atomization |
US20200397346A1 (en) * | 2018-03-13 | 2020-12-24 | Omron Corporation | Annotation method, annotation device, storage medium, and identification system |
US10909975B2 (en) | 2015-06-01 | 2021-02-02 | Sinclair Broadcast Group, Inc. | Content segmentation and time reconciliation |
Families Citing this family (118)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7747325B2 (en) | 1998-08-05 | 2010-06-29 | Neurovista Corporation | Systems and methods for monitoring a patient's neurological disease state |
US9375573B2 (en) | 1998-08-05 | 2016-06-28 | Cyberonics, Inc. | Systems and methods for monitoring a patient's neurological disease state |
US7209787B2 (en) | 1998-08-05 | 2007-04-24 | Bioneuronics Corporation | Apparatus and method for closed-loop intracranial stimulation for optimal control of neurological disease |
US9113801B2 (en) | 1998-08-05 | 2015-08-25 | Cyberonics, Inc. | Methods and systems for continuous EEG monitoring |
US7346920B2 (en) * | 2000-07-07 | 2008-03-18 | Sonic Solutions, A California Corporation | System, method and article of manufacture for a common cross platform framework for development of DVD-Video content integrated with ROM content |
US20050166232A1 (en) * | 1999-04-21 | 2005-07-28 | Lamkin Allan B... | Presentation of media content from multiple media sources |
US20060041639A1 (en) * | 1999-04-21 | 2006-02-23 | Interactual Technologies, Inc. | Platform detection |
US20050182828A1 (en) * | 1999-04-21 | 2005-08-18 | Interactual Technologies, Inc. | Platform specific execution |
US6529949B1 (en) * | 2000-02-07 | 2003-03-04 | Interactual Technologies, Inc. | System, method and article of manufacture for remote unlocking of local content located on a client device |
US20040220926A1 (en) * | 2000-01-03 | 2004-11-04 | Interactual Technologies, Inc., A California Cpr[P | Personalization services for entities from multiple sources |
US7392481B2 (en) * | 2001-07-02 | 2008-06-24 | Sonic Solutions, A California Corporation | Method and apparatus for providing content-owner control in a networked device |
US6859909B1 (en) * | 2000-03-07 | 2005-02-22 | Microsoft Corporation | System and method for annotating web-based documents |
US7398312B1 (en) * | 2000-03-29 | 2008-07-08 | Lucent Technologies Inc. | Method and system for caching streaming multimedia on the internet |
US7212727B2 (en) * | 2000-04-21 | 2007-05-01 | Matsushita Electric Industrial Co., Ltd. | Trick play method for digital storage medium |
US7779097B2 (en) * | 2000-09-07 | 2010-08-17 | Sonic Solutions | Methods and systems for use in network management of content |
US7689510B2 (en) * | 2000-09-07 | 2010-03-30 | Sonic Solutions | Methods and system for use in network management of content |
US9292516B2 (en) * | 2005-02-16 | 2016-03-22 | Sonic Solutions Llc | Generation, organization and/or playing back of content based on incorporated parameter identifiers |
US8078967B2 (en) * | 2000-12-01 | 2011-12-13 | Jlb Ventures Llc | Method and apparatus for virtual editing of multimedia presentations |
US20020147634A1 (en) * | 2001-01-31 | 2002-10-10 | Ronald Jacoby | System for dynamic generation of online streaming media advertisements |
US20020156912A1 (en) * | 2001-02-15 | 2002-10-24 | Hurst John T. | Programming content distribution |
US6897880B2 (en) * | 2001-02-22 | 2005-05-24 | Sony Corporation | User interface for generating parameter values in media presentations based on selected presentation instances |
US20030088687A1 (en) | 2001-12-28 | 2003-05-08 | Lee Begeja | Method and apparatus for automatically converting source video into electronic mail messages |
GB2387001B (en) | 2001-10-22 | 2005-02-02 | Apple Computer | Intelligent interaction between media player and host computer |
US20030167318A1 (en) * | 2001-10-22 | 2003-09-04 | Apple Computer, Inc. | Intelligent synchronization of media player with host computer |
US7739601B1 (en) | 2002-01-23 | 2010-06-15 | Microsoft Corporation | Media authoring and presentation |
US20120087637A1 (en) * | 2002-01-29 | 2012-04-12 | Logan James D | Methods and apparatus for recording and replaying video broadcasts |
US7797446B2 (en) | 2002-07-16 | 2010-09-14 | Apple Inc. | Method and system for updating playlists |
US7680849B2 (en) * | 2004-10-25 | 2010-03-16 | Apple Inc. | Multiple media type synchronization between host computer and media device |
US9412417B2 (en) | 2002-04-05 | 2016-08-09 | Apple Inc. | Persistent group of media items for a media device |
US9715500B2 (en) | 2004-04-27 | 2017-07-25 | Apple Inc. | Method and system for sharing playlists |
US7827259B2 (en) | 2004-04-27 | 2010-11-02 | Apple Inc. | Method and system for configurable automatic media selection |
US8150937B2 (en) | 2004-10-25 | 2012-04-03 | Apple Inc. | Wireless synchronization between media player and host device |
US6987221B2 (en) * | 2002-05-30 | 2006-01-17 | Microsoft Corporation | Auto playlist generation with multiple seed songs |
US7166791B2 (en) | 2002-07-30 | 2007-01-23 | Apple Computer, Inc. | Graphical user interface and methods of use thereof in a multimedia player |
US7956272B2 (en) * | 2002-07-30 | 2011-06-07 | Apple Inc. | Management of files in a personal communication device |
US7536713B1 (en) * | 2002-12-11 | 2009-05-19 | Alan Bartholomew | Knowledge broadcasting and classification system |
US20040123231A1 (en) * | 2002-12-20 | 2004-06-24 | Adams Hugh W. | System and method for annotating multi-modal characteristics in multimedia documents |
EP1616333A2 (en) * | 2003-04-24 | 2006-01-18 | Thomson Licensing | Creation of playlists using audio identification |
US20040237051A1 (en) * | 2003-05-23 | 2004-11-25 | Clauson Todd A. | Dynamic menu reordering |
JP3966515B2 (en) * | 2003-06-27 | 2007-08-29 | 日本アイ・ビー・エム株式会社 | Network system, server, data recording / reproducing apparatus, method and program thereof |
US7254593B2 (en) * | 2004-01-16 | 2007-08-07 | International Business Machines Corporation | System and method for tracking annotations of data sources |
US20070110074A1 (en) | 2004-06-04 | 2007-05-17 | Bob Bradley | System and Method for Synchronizing Media Presentation at Multiple Recipients |
US8797926B2 (en) | 2004-06-04 | 2014-08-05 | Apple Inc. | Networked media station |
US10972536B2 (en) | 2004-06-04 | 2021-04-06 | Apple Inc. | System and method for synchronizing media presentation at multiple recipients |
US8443038B2 (en) | 2004-06-04 | 2013-05-14 | Apple Inc. | Network media device |
US8438157B2 (en) * | 2004-06-28 | 2013-05-07 | International Business Machines Corporation | System and method for previewing relevance of streaming data |
US8261246B1 (en) | 2004-09-07 | 2012-09-04 | Apple Inc. | Method and system for dynamically populating groups in a developer environment |
US20060212472A1 (en) * | 2004-11-30 | 2006-09-21 | Wavenetworx Inc. | Method and system for creating a rich media content portal using third-party commercial portal application software |
US7958441B2 (en) | 2005-01-07 | 2011-06-07 | Apple Inc. | Media management for groups of media items |
US11314378B2 (en) | 2005-01-07 | 2022-04-26 | Apple Inc. | Persistent group of media items for a media device |
US7801910B2 (en) * | 2005-11-09 | 2010-09-21 | Ramp Holdings, Inc. | Method and apparatus for timed tagging of media content |
US9697231B2 (en) * | 2005-11-09 | 2017-07-04 | Cxense Asa | Methods and apparatus for providing virtual media channels based on media search |
US20070118873A1 (en) * | 2005-11-09 | 2007-05-24 | Bbnt Solutions Llc | Methods and apparatus for merging media content |
US9697230B2 (en) * | 2005-11-09 | 2017-07-04 | Cxense Asa | Methods and apparatus for dynamic presentation of advertising, factual, and informational content using enhanced metadata in search-driven media applications |
US20070106685A1 (en) * | 2005-11-09 | 2007-05-10 | Podzinger Corp. | Method and apparatus for updating speech recognition databases and reindexing audio and video content using the same |
US8868172B2 (en) | 2005-12-28 | 2014-10-21 | Cyberonics, Inc. | Methods and systems for recommending an appropriate action to a patient for managing epilepsy and other neurological disorders |
US20110107369A1 (en) * | 2006-03-28 | 2011-05-05 | O'brien Christopher J | System and method for enabling social browsing of networked time-based media |
US9812169B2 (en) * | 2006-03-28 | 2017-11-07 | Hewlett-Packard Development Company, L.P. | Operational system and architectural model for improved manipulation of video and time media data from networked time-based media |
WO2008060655A2 (en) * | 2006-03-29 | 2008-05-22 | Motionbox, Inc. | A system, method, and apparatus for visual browsing, deep tagging, and synchronized commenting |
US8082571B2 (en) * | 2006-06-05 | 2011-12-20 | Palo Alto Research Center Incorporated | Methods, apparatus, and program products to close interaction loops for social tv |
US8074251B2 (en) * | 2006-06-05 | 2011-12-06 | Palo Alto Research Center Incorporated | Limited social TV apparatus |
US20080027347A1 (en) | 2006-06-23 | 2008-01-31 | Neuro Vista Corporation, A Delaware Corporation | Minimally Invasive Monitoring Methods |
US8295934B2 (en) | 2006-11-14 | 2012-10-23 | Neurovista Corporation | Systems and methods of reducing artifact in neurological stimulation systems |
US8756333B2 (en) * | 2006-11-22 | 2014-06-17 | Myspace Music Llc | Interactive multicast media service |
US8176191B2 (en) * | 2006-11-30 | 2012-05-08 | Red Hat, Inc. | Automated identification of high/low value content based on social feedback |
AU2006249239B2 (en) * | 2006-12-07 | 2010-02-18 | Canon Kabushiki Kaisha | A method of ordering and presenting images with smooth metadata transitions |
US7559017B2 (en) * | 2006-12-22 | 2009-07-07 | Google Inc. | Annotation framework for video |
US8850140B2 (en) | 2007-01-07 | 2014-09-30 | Apple Inc. | Data backup for mobile device |
US8631088B2 (en) | 2007-01-07 | 2014-01-14 | Apple Inc. | Prioritized data synchronization with host device |
US9898656B2 (en) | 2007-01-25 | 2018-02-20 | Cyberonics, Inc. | Systems and methods for identifying a contra-ictal condition in a subject |
EP2124734A2 (en) | 2007-01-25 | 2009-12-02 | NeuroVista Corporation | Methods and systems for measuring a subject's susceptibility to a seizure |
US8036736B2 (en) | 2007-03-21 | 2011-10-11 | Neuro Vista Corporation | Implantable systems and methods for identifying a contra-ictal condition in a subject |
JP4833147B2 (en) * | 2007-04-27 | 2011-12-07 | 株式会社ドワンゴ | Terminal device, comment output method, and program |
US9788744B2 (en) | 2007-07-27 | 2017-10-17 | Cyberonics, Inc. | Systems for monitoring brain activity and patient advisory device |
US8156118B2 (en) * | 2007-08-20 | 2012-04-10 | Samsung Electronics Co., Ltd. | Method and system for generating playlists for content items |
US8046369B2 (en) | 2007-09-04 | 2011-10-25 | Apple Inc. | Media asset rating system |
KR20090063528A (en) * | 2007-12-14 | 2009-06-18 | 엘지전자 주식회사 | Mobile terminal and method of palying back data therein |
US9259591B2 (en) | 2007-12-28 | 2016-02-16 | Cyberonics, Inc. | Housing for an implantable medical device |
US20090171168A1 (en) | 2007-12-28 | 2009-07-02 | Leyde Kent W | Systems and Method for Recording Clinical Manifestations of a Seizure |
US8325800B2 (en) | 2008-05-07 | 2012-12-04 | Microsoft Corporation | Encoding streaming media as a high bit rate layer, a low bit rate layer, and one or more intermediate bit rate layers |
US8379851B2 (en) | 2008-05-12 | 2013-02-19 | Microsoft Corporation | Optimized client side rate control and indexed file layout for streaming media |
US8370887B2 (en) | 2008-05-30 | 2013-02-05 | Microsoft Corporation | Media streaming with enhanced seek operation |
US20090307199A1 (en) * | 2008-06-10 | 2009-12-10 | Goodwin James P | Method and apparatus for generating voice annotations for playlists of digital media |
US8265140B2 (en) | 2008-09-30 | 2012-09-11 | Microsoft Corporation | Fine-grained client-side control of scalable media delivery |
US8866698B2 (en) * | 2008-10-01 | 2014-10-21 | Pleiades Publishing Ltd. | Multi-display handheld device and supporting system |
WO2010039193A2 (en) * | 2008-10-01 | 2010-04-08 | Entourage Systems, Inc. | Multi-display handheld device and supporting system |
US20100145176A1 (en) * | 2008-12-04 | 2010-06-10 | Himes David M | Universal Electrode Array for Monitoring Brain Activity |
US8849390B2 (en) * | 2008-12-29 | 2014-09-30 | Cyberonics, Inc. | Processing for multi-channel signals |
US8588933B2 (en) | 2009-01-09 | 2013-11-19 | Cyberonics, Inc. | Medical lead termination sleeve for implantable medical devices |
US8769589B2 (en) * | 2009-03-31 | 2014-07-01 | At&T Intellectual Property I, L.P. | System and method to create a media content summary based on viewer annotations |
US8786624B2 (en) * | 2009-06-02 | 2014-07-22 | Cyberonics, Inc. | Processing for multi-channel signals |
KR20110047768A (en) * | 2009-10-30 | 2011-05-09 | 삼성전자주식회사 | Apparatus and method for displaying multimedia contents |
US8438131B2 (en) | 2009-11-06 | 2013-05-07 | Altus365, Inc. | Synchronization of media resources in a media archive |
US20110125560A1 (en) * | 2009-11-25 | 2011-05-26 | Altus Learning Systems, Inc. | Augmenting a synchronized media archive with additional media resources |
WO2011072890A1 (en) * | 2009-12-15 | 2011-06-23 | International Business Machines Corporation | Electronic document annotation |
EP2354970A1 (en) | 2010-01-05 | 2011-08-10 | Deutsche Telekom AG | Method, device and system for selecting data items |
US9643019B2 (en) * | 2010-02-12 | 2017-05-09 | Cyberonics, Inc. | Neurological monitoring and alerts |
US20110219325A1 (en) * | 2010-03-02 | 2011-09-08 | Himes David M | Displaying and Manipulating Brain Function Data Including Enhanced Data Scrolling Functionality |
JP5742378B2 (en) * | 2011-03-30 | 2015-07-01 | ソニー株式会社 | Information processing apparatus, playlist generation method, and playlist generation program |
US20120271823A1 (en) * | 2011-04-25 | 2012-10-25 | Rovi Technologies Corporation | Automated discovery of content and metadata |
US11611595B2 (en) | 2011-05-06 | 2023-03-21 | David H. Sitrick | Systems and methodologies providing collaboration among a plurality of computing appliances, utilizing a plurality of areas of memory to store user input as associated with an associated computing appliance providing the input |
US10402485B2 (en) | 2011-05-06 | 2019-09-03 | David H. Sitrick | Systems and methodologies providing controlled collaboration among a plurality of users |
US20130145248A1 (en) * | 2011-12-05 | 2013-06-06 | Sony Corporation | System and method for presenting comments with media |
US9245020B2 (en) | 2011-12-14 | 2016-01-26 | Microsoft Technology Licensing, Llc | Collaborative media sharing |
US20130238993A1 (en) * | 2012-03-12 | 2013-09-12 | Scool T.V. Ltd. | Apparatus and method for adding content using a media player |
US10620797B2 (en) * | 2012-06-29 | 2020-04-14 | Spotify Ab | Systems and methods for multi-context media control and playback |
US20140082635A1 (en) * | 2012-09-14 | 2014-03-20 | David H. Sitrick | Systems and methodologies for event processing for a plurality of users |
US9372833B2 (en) | 2012-09-14 | 2016-06-21 | David H. Sitrick | Systems and methodologies for document processing and interacting with a user, providing storing of events representative of document edits relative to a document; selection of a selected set of document edits; generating presentation data responsive to said selected set of documents edits and the stored events; and providing a display presentation responsive to the presentation data |
US20150254512A1 (en) * | 2014-03-05 | 2015-09-10 | Lockheed Martin Corporation | Knowledge-based application of processes to media |
US9794211B2 (en) * | 2014-08-20 | 2017-10-17 | Google Inc. | Event-based comment grouping for content items |
KR20160083759A (en) * | 2015-01-02 | 2016-07-12 | 삼성전자주식회사 | Method for providing an annotation and apparatus thereof |
US10783929B2 (en) | 2018-03-30 | 2020-09-22 | Apple Inc. | Managing playback groups |
US11297369B2 (en) | 2018-03-30 | 2022-04-05 | Apple Inc. | Remotely controlling playback devices |
US10993274B2 (en) | 2018-03-30 | 2021-04-27 | Apple Inc. | Pairing devices by proxy |
US10885092B2 (en) | 2018-04-17 | 2021-01-05 | International Business Machines Corporation | Media selection based on learning past behaviors |
US10390097B1 (en) | 2018-05-30 | 2019-08-20 | Rovi Guides, Inc. | Systems and methods for creating an asynchronous social watching experience among users |
US10614857B2 (en) | 2018-07-02 | 2020-04-07 | Apple Inc. | Calibrating media playback channels for synchronized presentation |
US10681402B2 (en) | 2018-10-09 | 2020-06-09 | International Business Machines Corporation | Providing relevant and authentic channel content to users based on user persona and interest |
Citations (21)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5333266A (en) * | 1992-03-27 | 1994-07-26 | International Business Machines Corporation | Method and apparatus for message handling in computer systems |
US5524193A (en) * | 1991-10-15 | 1996-06-04 | And Communications | Interactive multimedia annotation method and apparatus |
US5526407A (en) * | 1991-09-30 | 1996-06-11 | Riverrun Technology | Method and apparatus for managing information |
US5600775A (en) * | 1994-08-26 | 1997-02-04 | Emotion, Inc. | Method and apparatus for annotating full motion video and other indexed data structures |
US5633916A (en) * | 1994-12-30 | 1997-05-27 | Unisys Corporation | Universal messaging service using single voice grade telephone line within a client/server architecture |
US5699089A (en) * | 1994-03-03 | 1997-12-16 | Applied Voice Technology | Central control for sequential-playback objects |
US5732216A (en) * | 1996-10-02 | 1998-03-24 | Internet Angles, Inc. | Audio message exchange system |
US5838313A (en) * | 1995-11-20 | 1998-11-17 | Siemens Corporate Research, Inc. | Multimedia-based reporting system with recording and playback of dynamic annotation |
US5893087A (en) * | 1995-03-28 | 1999-04-06 | Dex Information Systems, Inc. | Method and apparatus for improved information storage and retrieval system |
US5903892A (en) * | 1996-05-24 | 1999-05-11 | Magnifi, Inc. | Indexing of media content on a network |
US5923848A (en) * | 1996-05-31 | 1999-07-13 | Microsoft Corporation | System and method for resolving names in an electronic messaging environment |
US5991365A (en) * | 1997-03-12 | 1999-11-23 | Siemens Corporate Research, Inc. | Remote phone-based access to a universal multimedia mailbox |
US6006241A (en) * | 1997-03-14 | 1999-12-21 | Microsoft Corporation | Production of a video stream with synchronized annotations over a computer network |
US6009462A (en) * | 1997-06-16 | 1999-12-28 | Digital Equipment Corporation | Replacing large bit component of electronic mail (e-mail) message with hot-link in distributed computer system |
US6081829A (en) * | 1996-01-31 | 2000-06-27 | Silicon Graphics, Inc. | General purpose web annotations without modifying browser |
US6105055A (en) * | 1998-03-13 | 2000-08-15 | Siemens Corporate Research, Inc. | Method and apparatus for asynchronous multimedia collaboration |
US6144375A (en) * | 1998-08-14 | 2000-11-07 | Praja Inc. | Multi-perspective viewer for content-based interactivity |
US6173317B1 (en) * | 1997-03-14 | 2001-01-09 | Microsoft Corporation | Streaming and displaying a video stream with synchronized annotations over a computer network |
US6173287B1 (en) * | 1998-03-11 | 2001-01-09 | Digital Equipment Corporation | Technique for ranking multimedia annotations of interest |
US6452615B1 (en) * | 1999-03-24 | 2002-09-17 | Fuji Xerox Co., Ltd. | System and apparatus for notetaking with digital video and ink |
US6484156B1 (en) * | 1998-09-15 | 2002-11-19 | Microsoft Corporation | Accessing annotations across multiple target media streams |
Family Cites Families (43)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US4649499A (en) | 1984-03-07 | 1987-03-10 | Hewlett-Packard Company | Touchscreen two-dimensional emulation of three-dimensional objects |
GB2254725A (en) * | 1991-04-12 | 1992-10-14 | Jeremy John Watts | Language learning tool |
EP0633539B1 (en) | 1993-06-30 | 2001-02-21 | Canon Kabushiki Kaisha | Document processing method and apparatus |
EP0650126A1 (en) | 1993-10-21 | 1995-04-26 | BULL HN INFORMATION SYSTEMS ITALIA S.p.A. | Annotation data processing system with hypermedia processable and active annotations |
US5583980A (en) | 1993-12-22 | 1996-12-10 | Knowledge Media Inc. | Time-synchronized annotation method |
GB2301260A (en) | 1995-05-26 | 1996-11-27 | Ibm | Voice mail system |
US5572643A (en) | 1995-10-19 | 1996-11-05 | Judson; David H. | Web browser with dynamic display of information objects during linking |
JP3441873B2 (en) | 1996-02-15 | 2003-09-02 | 富士重工業株式会社 | Pulley structure of belt-type continuously variable transmission and method of coupling movable sheave and hydraulic actuator member of the pulley |
AU725370C (en) | 1996-06-18 | 2003-01-02 | Cranberry Properties, Llc | Integrated voice, facsimile and electronic mail messaging system |
JPH1021261A (en) | 1996-07-05 | 1998-01-23 | Hitachi Ltd | Method and system for multimedia data base retrieval |
US5969716A (en) | 1996-08-06 | 1999-10-19 | Interval Research Corporation | Time-based media processing system |
US5893110A (en) | 1996-08-16 | 1999-04-06 | Silicon Graphics, Inc. | Browser driven user interface to a media asset database |
US5809250A (en) | 1996-10-23 | 1998-09-15 | Intel Corporation | Methods for creating and sharing replayable modules representive of Web browsing session |
US7111009B1 (en) | 1997-03-14 | 2006-09-19 | Microsoft Corporation | Interactive playlist generation using annotations |
US6449653B2 (en) | 1997-03-25 | 2002-09-10 | Microsoft Corporation | Interleaved multiple multimedia stream for synchronized transmission over a computer network |
US6259445B1 (en) | 1997-07-07 | 2001-07-10 | Informix, Inc. | Computer-based documentation and instruction |
US6360234B2 (en) | 1997-08-14 | 2002-03-19 | Virage, Inc. | Video cataloger system with synchronized encoders |
EP0899737A3 (en) | 1997-08-18 | 1999-08-25 | Tektronix, Inc. | Script recognition using speech recognition |
JPH11289512A (en) | 1998-04-03 | 1999-10-19 | Sony Corp | Editing list preparing device |
US6584479B2 (en) | 1998-06-17 | 2003-06-24 | Xerox Corporation | Overlay presentation of textual and graphical annotations |
US6163510A (en) | 1998-06-30 | 2000-12-19 | International Business Machines Corporation | Multimedia search and indexing system and method of operation using audio cues with signal thresholds |
US6833865B1 (en) | 1998-09-01 | 2004-12-21 | Virage, Inc. | Embedded metadata engines in digital capture devices |
US6366296B1 (en) | 1998-09-11 | 2002-04-02 | Xerox Corporation | Media browser using multimodal analysis |
US6956593B1 (en) | 1998-09-15 | 2005-10-18 | Microsoft Corporation | User interface for creating, viewing and temporally positioning annotations for media content |
US6317141B1 (en) | 1998-12-31 | 2001-11-13 | Flashpoint Technology, Inc. | Method and apparatus for editing heterogeneous media objects in a digital imaging device |
US6424370B1 (en) | 1999-10-08 | 2002-07-23 | Texas Instruments Incorporated | Motion based event detection system and method |
US6718308B1 (en) | 2000-02-22 | 2004-04-06 | Daniel L. Nolting | Media presentation system controlled by voice to text commands |
US6901207B1 (en) | 2000-03-30 | 2005-05-31 | Lsi Logic Corporation | Audio/visual device for capturing, searching and/or displaying audio/visual material |
US7003513B2 (en) | 2000-07-04 | 2006-02-21 | International Business Machines Corporation | Method and system of weighted context feedback for result improvement in information retrieval |
US6922702B1 (en) | 2000-08-31 | 2005-07-26 | Interactive Video Technologies, Inc. | System and method for assembling discrete data files into an executable file and for processing the executable file |
US20060015904A1 (en) | 2000-09-08 | 2006-01-19 | Dwight Marcus | Method and apparatus for creation, distribution, assembly and verification of media |
US6842761B2 (en) | 2000-11-21 | 2005-01-11 | America Online, Inc. | Full-text relevancy ranking |
US6701311B2 (en) | 2001-02-07 | 2004-03-02 | International Business Machines Corporation | Customer self service system for resource search and selection |
GB0103130D0 (en) | 2001-02-08 | 2001-03-28 | Newsplayer Ltd | Media editing method and software thereof |
US7085842B2 (en) | 2001-02-12 | 2006-08-01 | Open Text Corporation | Line navigation conferencing system |
US20030078973A1 (en) | 2001-09-25 | 2003-04-24 | Przekop Michael V. | Web-enabled system and method for on-demand distribution of transcript-synchronized video/audio records of legal proceedings to collaborative workgroups |
US20050160080A1 (en) | 2004-01-16 | 2005-07-21 | The Regents Of The University Of California | System and method of context-specific searching in an electronic database |
US20050198193A1 (en) | 2004-02-12 | 2005-09-08 | Jaakko Halme | System, method, and apparatus for creating metadata enhanced media files from broadcast media |
US20050198006A1 (en) | 2004-02-24 | 2005-09-08 | Dna13 Inc. | System and method for real-time media searching and alerting |
US7409384B2 (en) | 2004-04-19 | 2008-08-05 | Yahoo! Inc. | Integration of instant messaging with internet searching |
US7958115B2 (en) | 2004-07-29 | 2011-06-07 | Yahoo! Inc. | Search systems and methods using in-line contextual queries |
US7954049B2 (en) | 2006-05-15 | 2011-05-31 | Microsoft Corporation | Annotating multimedia files along a timeline |
US20070276852A1 (en) | 2006-05-25 | 2007-11-29 | Microsoft Corporation | Downloading portions of media files |
-
1999
- 1999-09-15 US US09/396,702 patent/US7111009B1/en not_active Expired - Fee Related
-
2006
- 2006-09-14 US US11/522,160 patent/US7631015B2/en not_active Expired - Fee Related
Patent Citations (23)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5526407A (en) * | 1991-09-30 | 1996-06-11 | Riverrun Technology | Method and apparatus for managing information |
US5524193A (en) * | 1991-10-15 | 1996-06-04 | And Communications | Interactive multimedia annotation method and apparatus |
US5333266A (en) * | 1992-03-27 | 1994-07-26 | International Business Machines Corporation | Method and apparatus for message handling in computer systems |
US5699089A (en) * | 1994-03-03 | 1997-12-16 | Applied Voice Technology | Central control for sequential-playback objects |
US5600775A (en) * | 1994-08-26 | 1997-02-04 | Emotion, Inc. | Method and apparatus for annotating full motion video and other indexed data structures |
US5633916A (en) * | 1994-12-30 | 1997-05-27 | Unisys Corporation | Universal messaging service using single voice grade telephone line within a client/server architecture |
US5893087A (en) * | 1995-03-28 | 1999-04-06 | Dex Information Systems, Inc. | Method and apparatus for improved information storage and retrieval system |
US5838313A (en) * | 1995-11-20 | 1998-11-17 | Siemens Corporate Research, Inc. | Multimedia-based reporting system with recording and playback of dynamic annotation |
US6081829A (en) * | 1996-01-31 | 2000-06-27 | Silicon Graphics, Inc. | General purpose web annotations without modifying browser |
US5903892A (en) * | 1996-05-24 | 1999-05-11 | Magnifi, Inc. | Indexing of media content on a network |
US5923848A (en) * | 1996-05-31 | 1999-07-13 | Microsoft Corporation | System and method for resolving names in an electronic messaging environment |
US5732216A (en) * | 1996-10-02 | 1998-03-24 | Internet Angles, Inc. | Audio message exchange system |
US6230172B1 (en) * | 1997-01-30 | 2001-05-08 | Microsoft Corporation | Production of a video stream with synchronized annotations over a computer network |
US5991365A (en) * | 1997-03-12 | 1999-11-23 | Siemens Corporate Research, Inc. | Remote phone-based access to a universal multimedia mailbox |
US6006241A (en) * | 1997-03-14 | 1999-12-21 | Microsoft Corporation | Production of a video stream with synchronized annotations over a computer network |
US6173317B1 (en) * | 1997-03-14 | 2001-01-09 | Microsoft Corporation | Streaming and displaying a video stream with synchronized annotations over a computer network |
US6009462A (en) * | 1997-06-16 | 1999-12-28 | Digital Equipment Corporation | Replacing large bit component of electronic mail (e-mail) message with hot-link in distributed computer system |
US6173287B1 (en) * | 1998-03-11 | 2001-01-09 | Digital Equipment Corporation | Technique for ranking multimedia annotations of interest |
US6332144B1 (en) * | 1998-03-11 | 2001-12-18 | Altavista Company | Technique for annotating media |
US6105055A (en) * | 1998-03-13 | 2000-08-15 | Siemens Corporate Research, Inc. | Method and apparatus for asynchronous multimedia collaboration |
US6144375A (en) * | 1998-08-14 | 2000-11-07 | Praja Inc. | Multi-perspective viewer for content-based interactivity |
US6484156B1 (en) * | 1998-09-15 | 2002-11-19 | Microsoft Corporation | Accessing annotations across multiple target media streams |
US6452615B1 (en) * | 1999-03-24 | 2002-09-17 | Fuji Xerox Co., Ltd. | System and apparatus for notetaking with digital video and ink |
Cited By (77)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7631015B2 (en) | 1997-03-14 | 2009-12-08 | Microsoft Corporation | Interactive playlist generation using annotations |
US7506262B2 (en) | 1998-09-15 | 2009-03-17 | Microsoft Corporation | User interface for creating viewing and temporally positioning annotations for media content |
US20050081159A1 (en) * | 1998-09-15 | 2005-04-14 | Microsoft Corporation | User interface for creating viewing and temporally positioning annotations for media content |
US9311405B2 (en) * | 1998-11-30 | 2016-04-12 | Rovi Guides, Inc. | Search engine for video and graphics |
US20130097145A1 (en) * | 1998-11-30 | 2013-04-18 | Gemstar Development Corporation | Search engine for video and graphics |
US20050120127A1 (en) * | 2000-04-07 | 2005-06-02 | Janette Bradley | Review and approval system |
US7555557B2 (en) * | 2000-04-07 | 2009-06-30 | Avid Technology, Inc. | Review and approval system |
US9462317B2 (en) | 2000-10-11 | 2016-10-04 | Rovi Guides, Inc. | Systems and methods for providing storage of data on servers in an on-demand media delivery system |
US9294799B2 (en) | 2000-10-11 | 2016-03-22 | Rovi Guides, Inc. | Systems and methods for providing storage of data on servers in an on-demand media delivery system |
US8725832B2 (en) | 2000-11-10 | 2014-05-13 | Trio Systems, Llc | System and method for creating and posting media lists for purposes of subsequent playback |
US7472175B2 (en) * | 2000-11-10 | 2008-12-30 | Trio Systems, Llc | System for creating and posting media for sharing on a communication network |
US8015263B2 (en) * | 2000-11-10 | 2011-09-06 | Trio Systems, Llc | System and method for creating and posting media lists for purposes of subsequent playback |
US20060265477A1 (en) * | 2000-11-10 | 2006-11-23 | Alan Bartholomew | Method and apparatus for creating and posting media |
US20090070438A1 (en) * | 2000-11-10 | 2009-03-12 | Trio Systems, Llc | System and method for creating and posting media lists for purposes of subsequent playback |
US7685522B1 (en) * | 2003-11-03 | 2010-03-23 | Adobe Systems Incorporated | Self-describing forms |
US7717340B1 (en) | 2004-06-30 | 2010-05-18 | Adobe Systems Incorporated | Enabling the use of machine-readable codes |
US7448550B1 (en) | 2004-06-30 | 2008-11-11 | Adobe Systems Incorporated | Enabling the use of machine-readable codes |
US20060161838A1 (en) * | 2005-01-14 | 2006-07-20 | Ronald Nydam | Review of signature based content |
US20060195790A1 (en) * | 2005-02-28 | 2006-08-31 | Yahoo! Inc. | Method and system for exploring similarities |
US20060195479A1 (en) * | 2005-02-28 | 2006-08-31 | Michael Spiegelman | Method for sharing and searching playlists |
US11048724B2 (en) | 2005-02-28 | 2021-06-29 | Huawei Technologies Co., Ltd. | Method and system for exploring similarities |
US20060195513A1 (en) * | 2005-02-28 | 2006-08-31 | Yahoo! Inc. | System and method for networked media access |
US20060195864A1 (en) * | 2005-02-28 | 2006-08-31 | Yahoo!, Inc. | Portable media device interoperability |
US10614097B2 (en) | 2005-02-28 | 2020-04-07 | Huawei Technologies Co., Ltd. | Method for sharing a media collection in a network environment |
US20060195789A1 (en) * | 2005-02-28 | 2006-08-31 | Yahoo! Inc. | Media engine user interface |
US7685204B2 (en) | 2005-02-28 | 2010-03-23 | Yahoo! Inc. | System and method for enhanced media distribution |
US10521452B2 (en) | 2005-02-28 | 2019-12-31 | Huawei Technologies Co., Ltd. | Method and system for exploring similarities |
US11468092B2 (en) | 2005-02-28 | 2022-10-11 | Huawei Technologies Co., Ltd. | Method and system for exploring similarities |
US7725494B2 (en) | 2005-02-28 | 2010-05-25 | Yahoo! Inc. | System and method for networked media access |
US7739723B2 (en) | 2005-02-28 | 2010-06-15 | Yahoo! Inc. | Media engine user interface for managing media |
US7747620B2 (en) | 2005-02-28 | 2010-06-29 | Yahoo! Inc. | Method and system for generating affinity based playlists |
US7818350B2 (en) | 2005-02-28 | 2010-10-19 | Yahoo! Inc. | System and method for creating a collaborative playlist |
US11573979B2 (en) | 2005-02-28 | 2023-02-07 | Huawei Technologies Co., Ltd. | Method for sharing and searching playlists |
US20060195521A1 (en) * | 2005-02-28 | 2006-08-31 | Yahoo! Inc. | System and method for creating a collaborative playlist |
US11709865B2 (en) | 2005-02-28 | 2023-07-25 | Huawei Technologies Co., Ltd. | Method for sharing and searching playlists |
US10019500B2 (en) | 2005-02-28 | 2018-07-10 | Huawei Technologies Co., Ltd. | Method for sharing and searching playlists |
US8346798B2 (en) | 2005-02-28 | 2013-01-01 | Yahoo! Inc. | Method for sharing and searching playlists |
US10860611B2 (en) | 2005-02-28 | 2020-12-08 | Huawei Technologies Co., Ltd. | Method for sharing and searching playlists |
US8626670B2 (en) | 2005-02-28 | 2014-01-07 | Yahoo! Inc. | System and method for improved portable media file retention |
US20060195516A1 (en) * | 2005-02-28 | 2006-08-31 | Yahoo! Inc. | Method and system for generating affinity based playlists |
US11789975B2 (en) | 2005-02-28 | 2023-10-17 | Huawei Technologies Co., Ltd. | Method and system for exploring similarities |
US20060195480A1 (en) * | 2005-02-28 | 2006-08-31 | Michael Spiegelman | User interface for sharing and searching playlists |
US20060195512A1 (en) * | 2005-02-28 | 2006-08-31 | Yahoo! Inc. | System and method for playlist management and distribution |
US20060195462A1 (en) * | 2005-02-28 | 2006-08-31 | Yahoo! Inc. | System and method for enhanced media distribution |
US20070136656A1 (en) * | 2005-12-09 | 2007-06-14 | Adobe Systems Incorporated | Review of signature based content |
US9384178B2 (en) | 2005-12-09 | 2016-07-05 | Adobe Systems Incorporated | Review of signature based content |
US7954049B2 (en) | 2006-05-15 | 2011-05-31 | Microsoft Corporation | Annotating multimedia files along a timeline |
US20070276852A1 (en) * | 2006-05-25 | 2007-11-29 | Microsoft Corporation | Downloading portions of media files |
US20080235589A1 (en) * | 2007-03-19 | 2008-09-25 | Yahoo! Inc. | Identifying popular segments of media objects |
US10282425B2 (en) * | 2007-03-19 | 2019-05-07 | Excalibur Ip, Llc | Identifying popular segments of media objects |
US20080313227A1 (en) * | 2007-06-14 | 2008-12-18 | Yahoo! Inc. | Method and system for media-based event generation |
US9542394B2 (en) * | 2007-06-14 | 2017-01-10 | Excalibur Ip, Llc | Method and system for media-based event generation |
US8140973B2 (en) | 2008-01-23 | 2012-03-20 | Microsoft Corporation | Annotating and sharing content |
US20090187825A1 (en) * | 2008-01-23 | 2009-07-23 | Microsoft Corporation | Annotating and Sharing Content |
US10394942B1 (en) * | 2008-07-01 | 2019-08-27 | Google Llc | Method and system for contextually placed chat-like annotations |
US20120159324A1 (en) * | 2010-12-17 | 2012-06-21 | General Electric Company | Systems and methods for software state capture and playback |
US9125169B2 (en) | 2011-12-23 | 2015-09-01 | Rovi Guides, Inc. | Methods and systems for performing actions based on location-based rules |
US9043403B2 (en) * | 2013-05-10 | 2015-05-26 | Google Inc. | System and method for providing visual representations of email to enable efficient email processing |
WO2016196692A1 (en) * | 2015-06-01 | 2016-12-08 | Miller Benjamin Aaron | Break state detection in content management systems |
US11527239B2 (en) | 2015-06-01 | 2022-12-13 | Sinclair Broadcast Group, Inc. | Rights management and syndication of content |
US11955116B2 (en) | 2015-06-01 | 2024-04-09 | Sinclair Broadcast Group, Inc. | Organizing content for brands in a content management system |
US10909975B2 (en) | 2015-06-01 | 2021-02-02 | Sinclair Broadcast Group, Inc. | Content segmentation and time reconciliation |
US10909974B2 (en) | 2015-06-01 | 2021-02-02 | Sinclair Broadcast Group, Inc. | Content presentation analytics and optimization |
US10923116B2 (en) | 2015-06-01 | 2021-02-16 | Sinclair Broadcast Group, Inc. | Break state detection in content management systems |
US10971138B2 (en) | 2015-06-01 | 2021-04-06 | Sinclair Broadcast Group, Inc. | Break state detection for reduced capability devices |
US10796691B2 (en) | 2015-06-01 | 2020-10-06 | Sinclair Broadcast Group, Inc. | User interface for content and media management and distribution systems |
US10224028B2 (en) | 2015-06-01 | 2019-03-05 | Sinclair Broadcast Group, Inc. | Break state detection for reduced capability devices |
US11783816B2 (en) | 2015-06-01 | 2023-10-10 | Sinclair Broadcast Group, Inc. | User interface for content and media management and distribution systems |
US10224027B2 (en) | 2015-06-01 | 2019-03-05 | Sinclair Broadcast Group, Inc. | Rights management and syndication of content |
US11664019B2 (en) | 2015-06-01 | 2023-05-30 | Sinclair Broadcast Group, Inc. | Content presentation analytics and optimization |
US11676584B2 (en) | 2015-06-01 | 2023-06-13 | Sinclair Broadcast Group, Inc. | Rights management and syndication of content |
US10068568B2 (en) | 2015-06-01 | 2018-09-04 | Sinclair Broadcast Group, Inc. | Content segmentation and time reconciliation |
US11727924B2 (en) | 2015-06-01 | 2023-08-15 | Sinclair Broadcast Group, Inc. | Break state detection for reduced capability devices |
US20160364103A1 (en) * | 2015-06-11 | 2016-12-15 | Yaron Galant | Method and apparatus for using gestures during video playback |
US10855765B2 (en) | 2016-05-20 | 2020-12-01 | Sinclair Broadcast Group, Inc. | Content atomization |
US11895186B2 (en) | 2016-05-20 | 2024-02-06 | Sinclair Broadcast Group, Inc. | Content atomization |
US20200397346A1 (en) * | 2018-03-13 | 2020-12-24 | Omron Corporation | Annotation method, annotation device, storage medium, and identification system |
Also Published As
Publication number | Publication date |
---|---|
US7111009B1 (en) | 2006-09-19 |
US7631015B2 (en) | 2009-12-08 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US7111009B1 (en) | Interactive playlist generation using annotations | |
US6484156B1 (en) | Accessing annotations across multiple target media streams | |
US7506262B2 (en) | User interface for creating viewing and temporally positioning annotations for media content | |
US6546405B2 (en) | Annotating temporally-dimensioned multimedia content | |
US9448696B1 (en) | System and method for media content collaboration throughout a media production process | |
US6557042B1 (en) | Multimedia summary generation employing user feedback | |
US9092173B1 (en) | Reviewing and editing word processing documents | |
US6426778B1 (en) | System and method for providing interactive components in motion video | |
US7562302B1 (en) | System and method for automatic generation of visual representations and links in a hierarchical messaging system | |
US8091026B2 (en) | Methods and apparatuses for processing digital objects | |
Chiu et al. | LiteMinutes: an Internet-based system for multimedia meeting minutes | |
US20040268224A1 (en) | Authoring system for combining temporal and nontemporal digital media | |
US20060277457A1 (en) | Method and apparatus for integrating video into web logging | |
US20020026521A1 (en) | System and method for managing and distributing associated assets in various formats | |
US20030124502A1 (en) | Computer method and apparatus to digitize and simulate the classroom lecturing | |
US20070250899A1 (en) | Nondestructive self-publishing video editing system | |
JP2002057981A (en) | Interface to access data stream, generating method for retrieval for access to data stream, data stream access method and device to access video from note | |
CN101657814A (en) | Systems and methods for specifying frame-accurate images for media asset management | |
US9286309B2 (en) | Representation of last viewed or last modified portion of a document | |
US8418051B1 (en) | Reviewing and editing word processing documents | |
JP2004112372A (en) | Image processing system | |
US8296647B1 (en) | Reviewing and editing word processing documents | |
CN107066437B (en) | Method and device for labeling digital works | |
Mu et al. | Enriched video semantic metadata: Authorization, integration, and presentation | |
KR20220037256A (en) | Method for providing an editable folder and adding an advertisement link to a user's folder, and service server for the same |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
CC | Certificate of correction | ||
FPAY | Fee payment |
Year of fee payment: 4 |
|
AS | Assignment |
Owner name: MICROSOFT TECHNOLOGY LICENSING, LLC, WASHINGTON Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:MICROSOFT CORPORATION;REEL/FRAME:034542/0001 Effective date: 20141014 |
|
FPAY | Fee payment |
Year of fee payment: 8 |
|
FEPP | Fee payment procedure |
Free format text: MAINTENANCE FEE REMINDER MAILED (ORIGINAL EVENT CODE: REM.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
LAPS | Lapse for failure to pay maintenance fees |
Free format text: PATENT EXPIRED FOR FAILURE TO PAY MAINTENANCE FEES (ORIGINAL EVENT CODE: EXP.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
STCH | Information on status: patent discontinuation |
Free format text: PATENT EXPIRED DUE TO NONPAYMENT OF MAINTENANCE FEES UNDER 37 CFR 1.362 |
|
FP | Lapsed due to failure to pay maintenance fee |
Effective date: 20211208 |