WO2009128227A1

WO2009128227A1 - Encoding apparatus of video data and sound data, encoding method thereof, and video editing system

Info

Publication number: WO2009128227A1
Application number: PCT/JP2009/001660
Authority: WO
Inventors: Atsushi Tabuchi; Naoya Yamasaki
Original assignee: Thomson Licensing
Priority date: 2008-04-13
Filing date: 2009-04-09
Publication date: 2009-10-22
Also published as: JP2011077564A

Abstract

An encoding apparatus of video data and sound data is disclosed. An encoding apparatus 100 includes a hardware encoder 151 and a software encoder 152. The hardware encoder 151 is configured by hardware for encoding processing and encodes a part of AV data. The software encoder 152 encodes another part of the AV data using a CPU 10 in parallel with encoding processing of the hardware encoder 151. A data allocator 14 allocates the AV data to both the encoders 151 and 152. A synthesizer 16 synthesizes video data encoded by the respective encoders into a series of encoded video data by arranging the video data in a predetermined order. An output unit 17 outputs the series of encoded video data. The encoding apparatus 100 can perform encoding at a faster encoding processing speed than an encoding processing speed achieved only by the hardware encoder 151.

Description

ENCODING APPARATUS OF VIDEO DATA AND SOUND DATA, ENCODING METHOD THEREOF, AND VIDEO EDITING SYSTEM

The present invention relates to encoding processing of video data and sound data and a video editing system using the same.

A video editing system is a system for supporting a user's editing work on video data and is generally realized by making a computer terminal including a general-purpose CPU, such as a personal computer, execute a predetermined application program. The video editing system records one or more pieces of video data and sound data from a video camera or the like onto an HDD beforehand and edits one video data stream therefrom. Edited sound and video data stream which has been compressed by a system or edited sound and video data stream which is non-compressed data is recorded in an arbitrary recording medium, such as an HDD or a DVD, within the same system, is transmitted for broadcast and the like, or is output to the outside through an interface, such as IEEE 1394.

The video editing system uses encoding of various schemes for compression of the sound and video data stream. In recent years, the amount of video data per stream is huge as can be seen from, for example, an HDV scheme. Therefore, in recent video editing systems, the time required for encoding processing tends to increase.

As a conventional technique for performing encoding processing of sound and video data at high speed, there is a technique of increasing the number of CPUs that perform encoding processings to execute the encoding processings in parallel. Additionally, there is a technique of mounting a chip or a circuit dedicated to encoding processing in a system and making it perform the whole encoding processing of video data.

[PLT 1] Japanese Unexamined Patent Application, First Publication No. 2004-356851
[PLT 2] Japanese Unexamined Patent Application, First Publication No. H06-339018

In the chip or the circuit dedicated to encoding processing, there is an upper limit of the encoding processing speed. If the encoding processing speed does not reach a user's desired speed, it may be considered to further improve the processing speed of a single chip or a circuit itself dedicated to encoding processing or to perform encoding processings in parallel by using a plurality of chips dedicated to encoding processing.

However, in order to improve the processing speed of the single chip or the circuit dedicated to encoding processing, the circuit design should be changed. This raises the manufacturing cost of the single chip or the circuit dedicated to encoding processing. Particularly, even a single chip dedicated to encoding processing is generally large and expensive. Accordingly, if a plurality of chips are used, the size increases and the manufacturing cost significantly increases. Thus, it is difficult for the chip or the circuit dedicated to encoding processing to perform the encoding processing at a faster speed while suppressing an increase in the manufacturing cost.

An object of the present invention is to provide an encoding apparatus capable of raising the encoding processing speed of video data and/or sound data, an encoding method thereof, and a video editing system.

According to an aspect of the present invention, there is provided an encoding apparatus for encoding AV data including sound data and/or video data, the encoding apparatus including: a CPU (Central Processing Unit); a hardware encoder configured by hardware for encoding processing, the hardware encoder encoding a part of the AV data; a software encoder for encoding another part of the AV data using the CPU in parallel with encoding processing of the hardware encoder; a data allocator for allocating the AV data to the hardware encoder and the software encoder; a synthesizer for arranging AV data respectively encoded by the hardware encoder and the software encoder in a predetermined order to synthesize the encoded AV data into a series of encoded AV data; and an output unit for outputting the series of encoded AV data, wherein AV data is encoded at a faster encoding processing speed than an encoding processing speed only by the hardware encoder.

According to the present invention, since AV data including sound data and/or video data is encoded in parallel by the hardware encoder and the software encoder, the AV data can be encoded at a faster encoding processing speed than an encoding processing speed only by the hardware encoder, and thus the encoding processing speed can be raised.

In this specification and claims, the "hardware encoder" is (i) a circuit, such as an IC or a module, specifically designed to execute a part or all of the encoding processing of video data and/or sound data or (ii) an arithmetic circuit, such as a CPU or a DSP (Digital Signal Processor), which is capable of executing a part or all of the encoding processing of video data and/or sound data and processing other than the encoding processing, and performs exclusively a part or all of the encoding processing during the arithmetic circuit is executing the part or all of the encoding processing of video data and/or sound data.

In this specification and claims, the "software encoder" refers to an encoder which performs encoding processing of video data in parallel with other processing by executing software for encoding processing of video data in an arithmetic circuit, such as a general-purpose CPU or a DSP, which is also used for processing other than encoding processing of video data.

According to another aspect of the present invention, there is provided a video editing system including: a CPU; an editor for editing AV data including sound data and/or video data; a hardware encoder configured by a circuit for encoding processing, the hardware encoder encoding a part of the edited AV data; a software encoder for encoding another part of the edited AV data using the CPU in parallel with encoding processing of the hardware encoder; a data allocator for allocating the edited AV data to the hardware encoder and the software encoder; a synthesizer for arranging AV data respectively encoded by the hardware encoder and the software encoder in a predetermined order to synthesize the encoded AV data into a series of encoded AV data; and an output unit for outputting the series of encoded AV data, wherein AV data is encoded at a faster encoding processing speed than an encoding processing speed only by the hardware encoder.

According to still another aspect of the present invention, there is provided a method for encoding video data by using a hardware encoder that is configured by hardware for encoding processing and encodes a part of AV data and a software encoder that encodes another part of the AV data using a CPU, the method including the steps of: receiving the AV data; allocating the AV data to the hardware encoder and the software encoder; encoding the allocated AV data in the hardware encoder and the software encoder; arranging AV data respectively encoded by the hardware encoder and the software encoder in a predetermined order to synthesize the encoded AV data into a series of encoded AV data; and outputting the series of encoded AV data, wherein AV data is encoded at a faster encoding processing speed than an encoding processing speed only by the hardware encoder.

According to the present invention, since AV data including sound data and/or video data is encoded in parallel by the hardware encoder and the software encoder, the AV data can be encoded at a faster encoding processing speed than the encoding processing speed only by the hardware encoder, and thus the encoding processing speed can be raised.

According to the present invention, an encoding apparatus capable of raising the encoding processing speed of video data and/or sound data, an encoding method thereof, and a video editing system can be provided.

FIG. 1 is a block diagram showing the configuration of a video editing system and the configuration of an encoding apparatus according to a first embodiment of the present invention. FIG. 2 is a schematic plan view of a board on which a hardware encoder shown in FIG. 1 is mounted. FIG. 3 is a block diagram showing a functional configuration of the video editing system according to the embodiments of the present invention. FIG. 4 is a diagram showing an example of an edit window displayed on the video editing system according to the first embodiment of the present invention. FIG. 5 is a block diagram of an encoding unit according to the first embodiment of the present invention. FIG. 6 is a flow chart of allocation processing of video data by the encoding unit shown in FIG. 5. FIG. 7 is a block diagram of an encoding unit according to a second embodiment of the present invention. FIG. 8 is a block diagram of an encoding unit according to a third embodiment of the present invention. FIG. 9 is an example of CPU load indices indicated by a first database shown in FIG. 8. FIG. 10 is a flow chart of processing for setting the allocation ratio of video data by the encoding unit shown in FIG. 8. FIG. 11 is a flow chart of allocation processing of video data by the encoding unit shown in FIG. 8. FIG. 12 is a block diagram of an encoding unit according to a fourth embodiment of the present invention. FIG. 13 is a flow chart of allocation processing of video data by the encoding unit shown in FIG. 12. FIG. 14 is a block diagram of an encoding unit according to a fifth embodiment of the present invention. FIG. 15 is a flow chart of allocation processing of video data by the encoding unit shown in FIG. 14. FIG. 16 is a block diagram of an encoding unit according to a sixth embodiment of the present invention. FIG. 17 is a flow chart of allocation processing of video data by the encoding unit shown in FIG. 16. FIG. 18 is a block diagram of an encoding unit according to a seventh embodiment of the present invention. FIG. 19 is a flow chart of allocation processing of video data by the encoding unit shown in FIG. 18. FIG. 20 is a block diagram showing an encoding unit according to an eighth embodiment of the present invention. FIG. 21 is a flow chart of allocation processing of video data by the encoding unit shown in FIG. 20.

Hereinafter, preferred embodiments of the present invention will be described referring to the drawings. Noted that although encoding processing of both video data and sound data is possible in the present invention, the case where encoding processing of video data is performed will be described for the convenience of explanation.

<<First embodiment>>
FIG. 1 is a block diagram showing the configuration of an encoding apparatus 100 and a video editing system 200 according to a first embodiment of the present invention. Referring to FIG. 1, the encoding apparatus 100 according to the first embodiment of the present invention is built in the video editing system 200. The video editing system 200 according to the first embodiment is a non-linear video editing system and is realized by using a computer terminal, such as a personal computer. The video editing system 200 includes a CPU 10, a memory unit 20, an internal HDD 30A, an internal drive 40A, an input/output interface 50, a user interface 60, an AV unit 70, a hardware encoder 80, and an internal bus 90 that connects these components. Among the components included in the video editing system 200, the CPU 10 and the hardware encoder 80 are included in the encoding apparatus 100. The video editing system 200 may further include a network interface connectable to an external LAN or the Internet.

The CPU 10 executes a program stored in the memory unit 20 and functions as an editor 13, a data allocator 14, a software encoder 152, a synthesizer 16, and an output unit 17 shown in FIG. 3 mentioned later. Details of these units will be described later.

A program and data causing the CPU 10 to execute processing described later are stored in the memory unit 20. Moreover, an input buffer area BI, a first output buffer area BO1, and a second output buffer area BO2 are reserved in the memory unit 20. Details of these buffer areas will be described later.

The internal HDD 30A is built in the computer terminal for realizing the video editing system 200. Instead of the internal HDD 30A or in addition to the internal HDD 30A as shown in FIG. 1, an external HDD 30B may be connected to the internal bus 90 through the input/output interface 50.

The internal drive 40A is built in the computer terminal for realizing the video editing system 200. Instead of the internal drive 40A or in addition to the internal drive 40A as shown in FIG. 1, an external drive 40B may be connected to the internal bus 90 through the input/output interface 50.

The internal drive 40A and the external drive 40B input video data and/or sound data from removable media, such as a DVD 102, and output video data and/or sound data to the removable media. In addition to an optical disc, removable media may include a magnetic disc, a magneto-optical disc, a Blu-ray disc, a semiconductor memory, and the like.

The input/output interface 50 can connect the user interface 60 and storage media built in external apparatuses, such as a second camera 101B, in addition to the external HDD 30B and the external drive 40B, to the internal bus 90. For example, the input/output interface 50 includes an IEEE 1394 interface and inputs/outputs video data and/or sound data from/to the second camera 101B using the IEEE 1394 interface. In addition to the second camera 101B, the input/output interface 50 can input video data and/or sound data from various apparatuses that process video data and/or sound data, such as a VTR, a switcher, or a transmission server, and output video data and/or sound data to the various apparatuses.

The user interface 60 is connected to the internal bus 90 through the input/output interface 50. The user interface 60 includes, for example, a mouse 61, a keyboard 62, a display 63, and a speaker 64. The user interface 60 may include a touch panel (not shown) as another input device.

The AV unit 70 includes a video interface and an audio interface. The AV unit 70 inputs/outputs video data from/to an external apparatus, such as the first camera 101A, through these interfaces. In addition to the first camera 101A, the AV unit 70 can input video data and/or sound data from various apparatuses that process video data and/or sound data, such as a VTR, a switcher, or a transmission server, and output video data and/or sound data to the various apparatuses.

The hardware encoder 80 is (i) a circuit, such as an IC or a module, specifically designed to execute a part or all of the encoding processing of video data and/or sound data or (ii) an arithmetic circuit, such as a CPU or a DSP, which is capable of executing a part or all of the encoding processing of video data and/or sound data and processing other than the encoding processing, and performs exclusively a part or all of the encoding processing during the arithmetic circuit is executing the part or all of the encoding processing of the video data and/or the sound data.

The hardware encoder 80 compresses non-compressed video data by encoding processing. For example, in the case of a lossy encoding scheme, video data encoding processing executed by the hardware encoder 80 includes the following steps (i) to (iv). (i) A step of division into blocks, tiles, and the like,
(ii) A step of orthogonal transform, such as DCT transform or wavelet transform,
(iii) A quantization step, and
(iv) A lossless encoding step, such as entropy coding.

The video data encoding processing of the hardware encoder 80 may also include, for example, a sub-sampling step, a shuffling step, an intra-frame prediction step, or an inter-frame prediction step. Additionally, the video data encoding processing of the hardware encoder 80 may be compression processing based on a lossless encoding scheme or may be encoding processing which does not perform compression.

The hardware encoder 80 has a frame buffer and encodes video data stored therein. Alternatively, a predetermined storage area of the memory unit 20 may be used as a similar frame buffer. The encoding scheme of the hardware encoder 80 is, for example, a DV scheme, an MPEG scheme, a JPEG 2000 scheme, an intra-frame scheme (e.g., an AVC-Intra scheme), or a closed GOP scheme in the MPEG scheme, but is not particularly limited thereto. The encoding scheme of the hardware encoder 80 is selected beforehand according to a device of an output destination of encoded video data. The hardware encoder 80 encodes each data unit based on information included therein. For example, in the intra-frame scheme, the data unit is one frame. Moreover, in the closed GOP scheme in the MPEG scheme, the encoding data unit may be 1 GOP.

FIG. 2 is an example of a schematic plan view of a board 800, on which the hardware encoder 80 shown in FIG. 1 is mounted. This board 800 is mounted in the computer terminal for realizing the video editing system 200.

Referring to FIG. 2, the board 800 includes a connector 81, a hardware encoder 82, a frame buffer 83, and an FPGA (Field Programmable Gate Array) 84. The AV unit 70 may be further mounted on the board 800.

The connector 81 is in conformity with PCI-Express and connects a circuit on the board 800 to a PCI-Express bus of the computer terminal when inserted into a PCI-Express slot in the computer terminal. This bus is included in the internal bus 90.

The frame buffer 83 is an independent device of a DDR2-SDRAM (Double-Data-Rate 2-Synchronous-DRAM) and may store a predetermined number of frames, for example, four frames. For every frame, video data to be encoded is transferred from the input buffer area BI of the memory unit 20 to the frame buffer 83 through the connector 81 and the FPGA 84. The hardware encoder 82 encodes video data in order from the first address of the frame buffer 83 through the FPGA 84. The video data encoded by the hardware encoder 82 is written into the frame buffer 83 through the FPGA 84 for every frame and is transferred from the frame buffer 83 to the first output buffer area BO1 of the memory unit 20 through the FPGA 84 and the connector 81.

The FPGA 84 is a single chip and includes a PCI-Express interface 85, a memory interface 86, a DMA engine 87, and an encoder interface 88. The PCI-Express interface 85 connects other modules in the FPGA 84 to the PCI-Express bus of the computer terminal, which constitutes the video editing system 200, through the connector 81. The memory interface 86 controls access from the other modules in the FPGA 84 to the frame buffer 83. The DMA engine 87 controls transfer of video data between the memory unit 20 and the frame buffer 83 through the PCI-Express bus in the computer terminal by using the PCI-Express interface 85 and the memory interface 86. The encoder interface 88 connects the other modules in the FPGA 84 to the hardware encoder 82.

FIG. 3 is a block diagram showing the main functional configuration of the video editing system 200 of FIG. 1. Referring to FIG. 3, the video editing system 200 includes the editor 13 and an encoding unit 11 as the functional configuration. The encoding unit 11 includes the data allocator 14, a hardware encoder 151, the software encoder 152, the synthesizer 16, and the output unit 17. The editor 13, the data allocator 14, the software encoder 152, the synthesizer 16, and the output unit 17 are realized by the CPU 10 of the video editing system 200 of FIG. 1.

The editor 13 receives a selection of video data to be edited, reads the selected video data, and performs editing processing based on a user's editing operation to thereby editing a series of video data streams.

Specifically, the editor 13 first displays a list of files stored in resources, such as the DVD 102, the internal HDD 30A, or the external HDD 30B, on the display 63 included in the user interface 60. These files include video data, sound data, still images, text data, etc., as material data. A user operates the mouse 61 and/or the keyboard 62 to select a file to be edited from the list. The editor 13 receives the user's file selection and displays a clip corresponding to the selected file on the display 63. The clip is information which refers to some or all of material data along the time axis. The clip includes time information indicating the start position and the end position of a data portion referred by the clip.

FIG. 4 is an example of an edit window EW. The editor 13 displays the edit window EW on the display 63 and receives an editing operation from the user. Referring to FIG. 4, the edit window EW includes, for example, a material window BW, a timeline window TW, and a preview window PW.

The editor 13 displays an icon of a clip IC1 corresponding to the selected file to be edited on the material window BW.

The editor 13 displays a plurality of tracks TR on the timeline window TW and receives the arrangement of a clip CL onto each track TR. In the example of FIG. 4, each track TR is a long and narrow belt-like area extending in the horizontal direction of the screen. Each track TR indicates the positional information on the timeline, which is the time axis of a video data stream generated by the editor 13. In the example of FIG. 4, the position in the horizontal direction on each track is associated with the position on the timeline such that the position on the timeline advances in synchronization with the movement of the position on each track TR from the left to the right in the horizontal direction of the screen. For example, the editor 13 receives the arrangement of the clip CL from the material window BW to each track TR through a user's operation of the mouse 61.

The editor 13 may also display a timeline cursor TLC and a time axis scale on the timeline window TW. In the example of FIG. 4, the timeline cursor TLC is a straight line which extends in the vertical direction of the screen from the time axis scale and intersects each track TR. The timeline cursor TLC is movable in parallel to each track TR. The value of the time axis scale pointed by the end of the timeline cursor TLC indicates the position of intersection between the timeline cursor TLC and each track TR on the timeline.

The editor 13 receives the settings of an IN point IP, which is the start position of the clip CL disposed on each track TR on the timeline, and an OUT point OP, which is the end position of the clip CL disposed on each track TR on the timeline, and changes to be made to the IN point IP and the OUT point OP of the clip CL after the clip has been disposed on each track TR.

The editor 13 may further receive, for the clip CL disposed on each track TR, setting of effects processing, such as adjustment of color and/or brightness of corresponding video, special effects on the entire corresponding video, or synthesis of videos between a plurality of clips CL disposed on different tracks TR, from the user.

The editor 13 displays video which corresponds to the clip CL disposed at the position on the timeline that is pointed by the timeline cursor TLC, on the preview window PW. The editor 13 also displays a moving image which corresponds to a designated range of the clip CL disposed on the timeline window TW, on the preview window PW. The user can check the result of editing processing received by the editor 13 from the video displayed on the preview window PW.

The editor 13 generates edit information on a series of video data streams to be edited based on the arrangement of clips CL on the tracks TR within the timeline window TW and effects processing set for each clip CL. The edit information specifies the content of editing processing of the editor 13 together with a time code. The time code indicates the position of video data which has been subjected to corresponding editing processing on the timeline. The editor 13 generates a series of video data streams by connecting video data referred to by the respective clip CL in the order on the timeline according to the edit information. Additionally, the editor 13 outputs the video data streams to the data allocator 14 as video data to be encoded and writes them in the input buffer area BI of the memory unit 20.

Returning to FIG. 3, the data allocator 14 allocates the video data to be encoded, which has been written in the input buffer area BI of the memory unit 20, to the hardware encoder 151 and the software encoder 152. Specifically, the data allocator 14 first reserves output buffer areas BO1 and BO2 within the memory unit 20 for the

encoders

151 and 152 for writing encoded video data therein. Then, the data allocator 14 sequentially selects a predetermined data unit, for example, one frame, of the video data written in the input buffer area BI as a data unit which is to be encoded by the

respective encoders

151 and 152. Then, the data allocator 14 sequentially transfers data units selected for the hardware encoder 151 to a frame buffer of the hardware encoder 151, and designates addresses in the first output buffer area BO1, to which the respective transmitted data units are to be written after the respective transmitted data units have been encoded, for the hardware encoder 151. On the other hand, regarding data units selected for the software encoder 152, the data allocator 14 sequentially designates addresses of the data units in the input buffer area BI together with addresses in the second output buffer area BO2, to which the data units are to be written after the data units have been encoded, for the software encoder 152. Additionally, the data allocator 14 sequentially allocates serial numbers, such as frame numbers, to the data units allocated for both the

encoders

151 and 152, and transmits the addresses designated for the

respective encoders

151 and 152 to the synthesizer 16 as writing destinations after encoding of the data units of the respective numbers together with the numbers.

Noted that, when two or more hardware encoders exist and they perform encoding processings independently, the data allocator 14 allocates video data to be encoded, which is to be processed by the respective hardware encoders, for the respective hardware encoders.

The hardware encoder 151 is a device driver of the hardware encoder 80 shown in FIG. 1. The hardware encoder 151 sequentially encodes video data to be encoded, which has been transferred by the data allocator 14 from the input buffer area BI of the memory unit 20 to the frame buffer. The encoding scheme is set by the editor 13. The hardware encoder 151 encodes each data unit of video data based on information included therein. For example, the encoding data unit is one frame when the encoding scheme is the intra-frame scheme, and the encoding data unit is 1 GOP when the encoding scheme is the closed GOP scheme. The hardware encoder 151 transfers the encoded video data to the address in the first output buffer area BO1 designated by the data allocator 14. Additionally, the hardware encoder 151 notifies the synthesizer 16 of the address of the transfer destination whenever encoded video data is transferred to the first output buffer area BO1.

The software encoder 152 is an encoding processing module executed by the CPU 10, and encodes video data to be encoded which is stored in the input buffer area BI of the memory unit 20 in order of the addresses designated from the data allocator 14. The software encoder 152 executes encoding processing independently and in parallel with the hardware encoder 151. Similar to the hardware encoder 151, the software encoder 152 encodes each data unit based on information included therein.

The data unit and the encoding scheme of the software encoder 152 are the same as the data unit and the encoding scheme of the hardware encoder 151. That is, if the encoding data unit of the hardware encoder 151 is one frame, the encoding data unit of the software encoder 152 is also one frame. If the encoding data unit of the hardware encoder 151 is 1 GOP, the encoding data unit of the software encoder 152 is also 1 GOP.

Moreover, if the encoding scheme of the hardware encoder 151 is the intra-frame scheme, the encoding scheme of the software encoder 152 is also the intra-frame scheme. If the encoding scheme of the hardware encoder 151 is the closed GOP scheme, the encoding scheme of the software encoder 152 is also the closed GOP scheme.

The software encoder 152 writes the encoded video data in the address in the second output buffer area BO2 designated by the data allocator 14. Additionally, the software encoder 152 informs the synthesizer 16 of the address of the writing destination whenever encoded video data is written in the second output buffer area BO2.

The synthesizer 16 arranges the data units which have been encoded by the hardware encoder 151 and the software encoder 152 in order of the numbers assigned by the data allocator 14, and synthesizes them into one encoded video data stream. Specifically, the synthesizer 16 first identifies the address corresponding to the number assigned to the data unit which is to be synthesized next from among the addresses received from the data allocator 14. When the same address as the identified address is notified from one of the

encoders

151 and 152, the synthesizer 16 then reads the encoded data unit from the address and adds it to the end of the arrangement of other data units read before. In this way, the order of the series of encoded data units that have been synthesized matches the order written in the input buffer area BI of the memory unit 20 by the editor 13. Additionally, the synthesizer 16 outputs the synthesized video data stream to the output unit 17.

Noted that, when the hardware encoder 151 and the software encoder 152 encode video data in the GOP unit by using the closed GOP scheme as an encoding scheme, the synthesizer 16 can also arrange the encoded video data streams based on not only the numbers received from the data allocator 14 but also information, such as a GOP header, included in the encoded video data. Preferably, data units respectively encoded by the hardware encoder 151 and the software encoder 152 are not predictable from each other because processing of the synthesizer becomes easy.

The output unit 17 formats the synthesized video data stream in a predetermined file format or a predetermined transmission format. The output unit 17 adds information and/or parameters which are required for decoding of encoded video data and other specified information to the video data stream, and formats the entire data stream in the specified format. The file format or the transmission format used by the output unit 17 is set by the editor 13. The output unit 17 writes the synthesized video data stream in an arbitrary recording medium, such as the internal HDD 30A, the external HDD 30B, or the DVD 102 mounted in the internal drive 40A or the external drive 40B through the internal bus 90. The output unit 17 may also transmit the synthesized video data stream to a database or an information terminal connected through the network interface. The output unit 17 may also output the synthesized video data stream from the AV unit 70 or the input/output interface 50 to an external apparatus. Note that the output unit 17 can also generate and output not only video data streams but also audio contents.

FIG. 5 is a block diagram of the encoding unit 11 according to the first embodiment of the present invention. Referring to FIG. 5, the encoding unit 11 includes the data allocator 14, the hardware encoder 151, the software encoder 152, the synthesizer 16, the output unit 17, and a buffer 21. The buffer 21 is disposed between the data allocator 14 and the hardware encoder 151. The buffer 21 corresponds to a frame buffer built in the hardware encoder 80, such as a storage area of the frame buffer 83 shown in FIG. 2. In this case, the capacity of the buffer 21 is, for example, several frames. Alternatively, the buffer 21 may be a storage area in the memory unit 20. In this case, the capacity of the buffer 21 is, for example, tens of frames.

The data allocator 14 sequentially transfers video data allocated to the hardware encoder 151, to the buffer 21. The hardware encoder 151 encodes the video data sequentially from the first address of the buffer 21 in a predetermined data unit, for example, by one frame.

The data allocator 14 reads space information from the buffer 21 and checks the size of an empty area of the buffer 21, for example, the number of empty frame buffers, based on the space information. If the size exceeds a predetermined threshold, for example, one frame, the data allocator 14 allocates the video data to be encoded to the hardware encoder 80. In contrast, if the size of the empty area of the buffer 21 decreases to the predetermined threshold or less, for example, one frame or less, the data allocator 14 allocates the next frame subsequent to the frame allocated last to the hardware encoder 151 to the software encoder 152. That is, the data allocator 14 does not write the next frame in the buffer 21 and transmits the address in the input buffer area BI, in which the next frame is stored, to the software encoder 152. Thereafter, the data allocator 14 continuously allocates video data to be encoded to the software encoder 152 until the size of the empty area of the buffer 21 exceeds the threshold again.

FIG. 6 is a flow chart of allocation processing of video data by the encoding unit 11 shown in FIG. 5. Hereinafter, allocation processing of video data will be described referring to FIG. 6. Note that, in the following explanation, the encoding data unit is assumed to be one frame for convenience's sake. For example, when a video data stream to be encoded is output from the editor 13, the following processing starts.

First, in step S11, the data allocator 14 allocates a predetermined number of frames, for example, one frame, to the hardware encoder 151 from the head of the video data stream to be encoded, and transfers it to the buffer 21. The hardware encoder 151 begins to encode video data sequentially from the first address of the buffer 21.

Then, in step S12, the data allocator 14 determines whether or not the number of empty frame buffers has decreased to a predetermined threshold or less based on the space information of the buffer 21. If the number of empty frame buffers is larger than the threshold (in the case of "Yes" in step S12), the process proceeds to step S13. If the number of empty frame buffers is smaller than or equal to the threshold (in the case of "No" in step S12), the process proceeds to step S14.

In step S13, the data allocator 14 allocates the next frame of the video data stream to be encoded to the hardware encoder 151, and transfers the frame to the buffer 21. Then, the process proceeds to step S15.

In step S14, the data allocator 14 allocates the next frame of the video data stream to be encoded to the software encoder 152. Then, the process proceeds to step S15.

Subsequently, in step S15, the data allocator 14 determines whether or not the allocated frame is the last frame of the video data stream to be encoded. This determination is possible by, for example, referring to the frame header. If the frame is not the last (in the case of "No" in step S15), the process returns to step S12. If the frame is the last (in the case of "Yes" in step S15), the process ends.

In the encoding apparatus 100 of the first embodiment, the data allocator 14 allocates a part of video data to be encoded to the software encoder 152 when the empty area of the buffer 21 of the encoding unit 11 has decreased to the threshold or less. As a result, encoding processing by the hardware encoder 151 and encoding processing by the software encoder 152 are performed in parallel, and thus it is possible to perform encoding processing of the entire encoding apparatus 100 at a faster speed than the case where only the hardware encoder 151 performs encoding processing. Particularly, the data allocator 14 acquires the space information of the buffer 21, in which a frame waiting for encoding processing by the hardware encoder 151 is temporarily stored, and determines the allocation destination of the next frame based on the space information. Accordingly, the encoding processing of the software encoder 152 is performed in parallel while always operating the hardware encoder 151, and thus the encoding apparatus 100 can perform the encoding processing at a faster speed than the encoding processing speed of the hardware encoder 151. As a result, the encoding processing speed of the video editing system 200 can be improved.

Moreover, although a video data stream to be encoded may include a portion which causes a heavy load in the hardware encoder 151, since the software encoder 152 performs encoding processing for such a portion in parallel, the efficiency of encoding processing can be extremely improved compared with the case where there is no software encoder 152. Furthermore, in a portion of a video data stream which can be sufficiently encoded only by encoding processing of the hardware encoder 151, the CPU 10 functioning as the software encoder 152 can perform processing other than the encoding processing, so that the hardware resources of the non-linear video editing system 200 can be utilized more flexibly and effectively. These two advantageous effects can also be obtained in the following embodiments.

<<Second embodiment>>
An encoding apparatus 100 according to a second embodiment of the present invention is built in a non-linear video editing system 200, similar to that according to the first embodiment. The video editing system 200 is the same as that according to the first embodiment except for an encoding unit. Therefore, the explanation in the first embodiment is cited for details of the same components.

FIG. 7 is a block diagram of an encoding unit 11A according to the second embodiment of the present invention. Referring to FIG. 7, the encoding unit 11A includes a data allocator 14, a hardware encoder 151, a software encoder 152, a synthesizer 16, an output unit 17, and an allocation ratio setting unit 22A. The encoding unit 11A includes the same components as the components shown in FIG. 3 except for the allocation ratio setting unit 22A. In FIG. 7, the same reference symbols as the reference symbols shown in FIG. 3 are given to the same components, and the detailed explanation thereof is cited.

Referring to FIG. 7, the function of the allocation ratio setting unit 22A is realized by the CPU 10, which controls other components of a computer terminal according to a predetermined program. The allocation ratio setting unit 22A sets the ratio of the number of data units in a video data stream to be encoded which are allocated to the hardware encoder 151 and the number of data units in the stream which are allocated to the software encoder 152. Additionally, the allocation ratio setting unit 22A may make a user select the value of the ratio. That is, the allocation ratio setting unit 22A displays a predetermined input screen on a display of the computer terminal prior to encoding processing by the encoding unit 11A and receives a value of the ratio from the user through a mouse and/or a keyboard. The value of the ratio selected by the user is transmitted from the allocation ratio setting unit 22A to the data allocator 14. The data allocator 14 allocates the video data stream to be encoded to the hardware encoder 151 and the software encoder 152 according to the ratio.

For example, the case where the encoding data unit is one frame and the user sets the ratio of the number of frames allocated to the software encoder 152 to the number of frames allocated to the hardware encoder 151 to 1/2 is assumed. The data allocator 14 first allocates the first frame of the video data stream to be encoded to the software encoder 152. The data allocator 14 then allocates the second and third frames of the video data stream to the hardware encoder 151. Additionally, the data allocator 14 allocates the fourth frame to the software encoder 152 and allocates the fifth and sixth frames to the hardware encoder 151. Similarly, thereafter, whenever one frame is allocated to the software encoder 152, two frames subsequent thereto are allocated to the hardware encoder 151.

Alternatively, whenever the editor 13 writes video data to be encoded in the memory unit 20 by a predetermined number of frames, for example, 15 frames, the data allocator 14 divides them into three equal parts, allocates the first one-third thereof, for example, the first to fifth frames, to the software encoder 152, and allocates the remaining two-thirds thereof, for example, the sixth to fifteenth frames, to the hardware encoder 151. While the software encoder 152 is encoding the first to fifth frames, the hardware encoder 151 encodes the sixth to fifteenth frames.

The encoding apparatus 100 according to the second embodiment may make the user select the ratio of the number of data units in a video data stream to be encoded which are allocated to the hardware encoder 151 and the number of data units in the stream which are allocated to the software encoder 152 through the allocation ratio setting unit 22A. Accordingly, the user can adjust the ratio of the number of data units allocated to the

encoders

151 and 152 per unit time while observing the time taken for the video editing system 200, in which the encoding unit 11A is built, to perform the entirety of a series of processings. As a result, the user can make the encoding processing speed of the entire encoding apparatus 100 faster than the encoding processing speed only by the hardware encoder 151. Thus, the encoding processing speed of the video editing system 200 can be improved.

<<Third embodiment>>
An encoding apparatus 100 according to a third embodiment of the present invention is built in a non-linear video editing system 200, similar to that according to the first embodiment. The video editing system 200 is the same as that according to the first embodiment except for an encoding unit. Therefore, the explanation in the first embodiment is cited for details of the same components.

FIG. 8 is a block diagram of an encoding unit 11B according to the third embodiment of the present invention. Referring to FIG. 8, the encoding unit 11B includes a data allocator 14, a hardware encoder 151, a software encoder 152, a synthesizer 16, an output unit 17, a CPU load estimator 23, an allocation ratio setting unit 22B, and a first database 24. The encoding unit 11B includes the same components as the components shown in FIG. 3 except for the CPU load estimator 23, the allocation ratio setting unit 22B, and the first database 24. In FIG. 8, the same reference symbols as the reference symbols shown in FIG. 3 are given to the same components, and the detailed explanation thereof is cited.

Referring to FIG. 8, the function of the CPU load estimator 23 is realized by the CPU 10, which controls other components of a computer terminal according to a predetermined program. The CPU load estimator 23 predicts the load of the CPU 10 in the editing processing of the editor 13 from edit information that specifies the content of the editing processing.

The CPU load estimator 23 receives the edit information from the editor 13 prior to the editing processing of the editor 13. The edit information includes a time range of video data, the chronological order of the video data, the content of the effect, information on resolution conversion, information on frame rate conversion, and the like. The CPU load estimator 23 predicts the transition of a timeline processing index TE during an editing processing period of the editor 13 based on the edit information as follows. Here, the timeline processing index TE is defined as the throughput of the CPU 10 required for the editor 13 to edit video data of one data unit, that is, as a CPU load index. Hereinafter, the data unit is assumed to be one frame for the convenience of explanation.

The CPU load estimator 23 first divides the whole editing processing period of the editor 13 into periods each including a plurality of editing processings based on the edit information. Here, if the number of frames edited in one divided period is smaller than a predetermined threshold, the one divided period is integrated into either of the periods before or after the one divided period. Accordingly, frames, the number of which being more than or equal to the threshold, are edited in each divided period.

The CPU load estimator 23 then identifies types of concurrent editing processings for every divided period from the edit information, and calculates the CPU load index corresponding to each identified type. Here, for every type of editing processing, the throughput of the CPU 10 required for the editing processing, that is, the CPU load index, has been evaluated beforehand and is stored in the memory unit 20 as the first database 24.

FIG. 9 is a table showing an example of the CPU load indices. Referring to FIG. 9, assuming that the CPU load index in video data decoding processing based on the AVC-Intra scheme is 100, CPU load indices regarding other various kinds of processings of the CPU 10 are evaluated. The CPU load estimator 23 searches the first database 24 for CPU load indices corresponding to the respective identified types of the editing processings. Additionally, the CPU load estimator 23 adds the searched CPU load indices for each divided period and determines each sum as the timeline processing index TE of the divided period.

For example, the case where edit information expresses editing processing performed in a certain period like "two materials of AVC-Intra 50 are synthesized by picture in picture and one title is inserted" is assumed. The CPU load estimator 23 first identifies "decoding of materials of the AVC-Intra 50", "effects processing by picture in picture", and "addition of a title" as types of concurrent editing processings in the period. The CPU load estimator 23 then acquires 100, 5, and 5 from the first database 24 as CPU load indices corresponding to the respective identified types of the editing processings. Additionally, the CPU load estimator 23 determines the timeline processing index TE from the searched

CPU load indices

100, 5, and 5 by the following expression: TE = 100 x 2 + 5 + 5 = 210. Here, since the materials of the AVC-Intra 50 are two frames, the CPU load index for their decoding processings is estimated as twice the value 100 per frame.

The CPU load estimator 23 reads a time code which indicates the start position of a time range of a video data stream edited in each divided period from the edit information, and associates it with the timeline processing index TE in the divided period. In this way, the transition of the timeline processing index TE during the editing processing period of the editor 13 is predicted. Additionally, the CPU load estimator 23 supplies the combination of the timeline processing index TE and the time code to the allocation ratio setting unit 22B.

The function of the allocation ratio setting unit 22B is realized by the CPU 10, which controls other components of a computer terminal according to a predetermined program. The allocation ratio setting unit 22B sets the ratio of video data allocated to both the

encoders

151 and 152 per unit time by the data allocator 14, based on the transition of the load of the CPU 10 predicted by the CPU load estimator 23, that is, the transition of the timeline processing index TE. For example, when the encoding data unit is one frame, the allocation ratio setting unit 22B calculates the ratio of the number of frames HF allocated to the hardware encoder 151 per unit time and the number of frames SF allocated to the software encoder 152 per unit time according to each timeline processing index TE as follows.

The allocation ratio setting unit 22B first receives the combination of the timeline processing index TE and the time code from the CPU load estimator 23. The allocation ratio setting unit 22B then calculates the ratio CE/TE of a CPU throughput index CE to each timeline processing index TE. The CPU throughput index CE is defined as the throughput per unit time, which can be provided by the CPU 10, with respect to editing processing of the editor 13 or encoding processing of the software encoder 152, for example, the throughput per one frame cycle. Here, the CPU 10 also generally performs processing related to other modules, such as the OS of the computer terminal, the data allocator 14, and the synthesizer 16. A throughput obtained by subtracting the throughput related to their processing from the total throughput, which can be provided by the CPU 10, is the CPU throughput index CE. Accordingly, the ratio CE/TE indicates a maximum value of the editing processing speed obtained if the throughput which can be provided by the CPU 10 is all provided for the same editing processing during a period evaluated by means of the timeline processing index TE, that is, a maximum number MF of frames on which the same editing processing can be executed per unit time.

Additionally, the allocation ratio setting unit 22B compares each maximum value MF with the encoding processing speed of the hardware encoder 151, that is, the number of frames HF that the hardware encoder 151 can encode per unit time.

If the maximum value MF of the editing processing speed is smaller than or equal to the encoding processing speed HF of the hardware encoder 151, it can be considered that the editing processing evaluated by the maximum value MF is slower than the encoding processing of the hardware encoder 151. In this case, for the period of that editing processing, the allocation ratio setting unit 22B sets the number of frames SF allocated to the software encoder 152 to "0", to thereby prevent an increase in the load of the CPU 10.

If the maximum value MF of the editing processing speed exceeds the encoding processing speed HF of the hardware encoder 151, it can be considered that the editing processing evaluated by the maximum value MF is faster than the encoding processing of the hardware encoder 151. In this case, for the period of that editing processing, the allocation ratio setting unit 22B sets the number of frames SF allocated to the software encoder 152 to a value larger than "0" as follows. Thus, the encoding processing time in the entire encoding unit 11B can be shortened by parallel processing of both the

encoders

151 and 152.

The encoding processing speed HF of the hardware encoder 151 is constant regardless of the load variation of the CPU 10. Therefore, the allocation ratio setting unit 22B first sets the number of frames allocated to the hardware encoder 151 per unit time to the encoding processing speed HF. In this case, in order to operate the hardware encoder 151 in parallel with the editor 13 with no delay, it is preferable to maintain the throughput per unit time of the CPU 10 for the editor 13 at least a product TE x HF of the timeline processing index TE and the encoding processing speed HF of the hardware encoder 151. Therefore, the remainder RE = CE - TE x HF obtained by subtracting the product TE x HF from the CPU throughput index CE can be considered as the upper limit of the throughput of the CPU 10 which can be provided to the software encoder 152 per unit time. The allocation ratio setting unit 22B calculates the upper limit RE for every timeline processing index TE. Note that the upper limit RE is equal to a value obtained by multiplying the difference between the maximum value MF of the editing processing speed and the encoding processing speed HF of the hardware encoder 151 by the timeline processing index TE: i.e., RE = TE x (MF - HF). Accordingly, when the maximum value MF of the editing processing speed exceeds the encoding processing speed HF of the hardware encoder 151, the upper limit RE is larger than "0".

On the other hand, the allocation ratio setting unit 22B adds the throughput per unit time of the CPU 10 required for encoding processing of the software encoder 152, that is, a software encoding processing index SE, to each timeline processing index TE, thereby calculating the sum QE = SE + TE. Here, the software encoding processing index SE changes according to the encoding scheme. The software encoding processing index SE evaluated for every type of encoding scheme is stored in the first database 24. The table of FIG. 9 shows examples of CPU load indices for every type of encoding processing. These CPU load indices are used as the software encoding processing index SE. The allocation ratio setting unit 22B searches the first database 24 for the software encoding processing index SE corresponding to the type of encoding scheme which will be actually used, beforehand.

The sum QE indicates the throughput per unit time of the CPU 10 required for parallel processing of editing processing corresponding to the timeline processing index TE and encoding processing corresponding to the software encoding processing index SE. Therefore, it is preferable that the allocation ratio setting unit 22B divide the upper limit RE by the sum QE, which is obtained from each timeline processing index TE, and sets the quotient RE/QE as the number of frames SF allocated to the software encoder 152 per unit time during an editing processing period indicated by the time code corresponding to the timeline processing index TE. Thus, it can be expected that the software encoder 152 will operate in parallel with the editor 13 with no delay. Note that since the upper limit RE is larger than "0", the number of frames SF allocated to the software encoder 152 is also larger than "0".

Here, even when the unit time is equal to a frame cycle, the number of frames SF and the encoding processing speed HF of the hardware encoder 151 are not always an integer. Accordingly, the allocation ratio setting unit 22B approximates the ratio SF:HF of the number of frames SF, which has been obtained from each timeline processing index TE, and the encoding processing speed HF of the hardware encoder 151 as an integer ratio HFI:SFI. For example, the sum HFI + SFI of two integers HFI and SFI expressing the integer ratio HFI:SFI is set small compared with the total number of frames edited in an editing processing period indicated by the time code corresponding to the timeline processing index TE.

The allocation ratio setting unit 22B sets the number of frames SF = 0 or the integer ratio HFI:SFI as follows for the data allocator 14. First, whenever one frame to be encoded is written from the editor 13 into the memory unit 20, the data allocator 14 reads a time code from the frame and transmits it to the allocation ratio setting unit 22B. The allocation ratio setting unit 22B compares the time code with each time code received from the CPU load estimator 23. When the time code from the data allocator 14 reaches a value smaller by a predetermined amount than the time code from the CPU load estimator 23, the allocation ratio setting unit 22B gives to the data allocator 14 the number of frames SF = 0 or the integer ratio HFI:SFI obtained from the timeline processing index TE corresponding to the time code from the CPU load estimator 23. If the number of frames SF = 0 is given, the data allocator 14 thereafter allocates all frames to be encoded to the hardware encoder 151. In contrast, if the integer ratio HFI:SFI is given, thereafter, whenever the data allocator 14 successively allocates SFI frames to the software encoder 152, the data allocator 14 successively allocates HFI frames subsequent thereto to the hardware encoder 151. The data allocator 14 continues the above-described allocation operation until the number of frames SF = 0 or the integer ratio HFI:SFI is newly input from the allocation ratio setting unit 22B.

Here, the predetermined amount is mainly determined by a general difference between a time code of a frame to be encoded which has been written in the memory unit 20 by the editor 13 and a time code of a frame edited in parallel with the writing. That is, the predetermined amount is adjusted such that the latter time code is estimated from the former time code.

Separately from those described above, the allocation ratio setting unit 22B may give a ratio alpha = HF:SF between the number of frames SF obtained from each timeline processing index TE and the encoding processing speed HF of the hardware encoder 151 to the data allocator 14. In this case, whenever one frame to be encoded is written from the editor 13 into the input buffer area BI in the memory unit 20, the data allocator 14 allocates the frame to the hardware encoder 151 with the probability of alpha /(1 + alpha) and allocates the frame to the software encoder 152 with the probability of 1/(1 + alpha).

In this way, the allocation ratio setting unit 22B changes the ratio HF:SF of the number of frames per unit time which are allocated to the

encoders

151 and 152 by the data allocator 14, for every timeline processing index TE. Thus, even if the load of the CPU 10 changes according to the content of editing processing of the editor 13, the encoding processing speed of the entire encoding unit 11B increases up to a maximum (HF + SF) / HF times the encoding processing speed HF only by the hardware encoder 151 while the waiting time of each

encoder

151 and 152 is maintained sufficiently short. Therefore, the processing speed of the entire video editing system 200 is increased.

For example, in the case where the CPU throughput index CE when one frame cycle is set as the unit time is "400" and the software encoding processing index SE is "200", the encoding processing speed CE/QE only by the software encoder 152 during an editing processing period for which the timeline processing index TE is 200 is equal to "1". On the other hand, the maximum value of the editing processing speed MF = CE/TE is equal to "2". Accordingly, when the encoding processing speed HF of the hardware encoder 151 is "1", since this value is smaller than the maximum value MF, the data allocator 14 allocates video data to be encoded to both the

encoders

151 and 152 in the ratio of the number of frames HF:SF = 1:0.5 = 2:1. As a result, the encoding processing speed of the entire encoding unit 11B increases up to a maximum 1.5 times the encoding processing speed only by the hardware encoder 151.

Note that if the encoding processing speed HF of the hardware encoder 151 is "2", this value is equal to the maximum value MF of the editing processing speed. Accordingly, it is preferable that the data allocator 14 allocate video data to be encoded only to the hardware encoder 151. The entire video editing system 200 can be operated efficiently by maintaining the balance between the editing processing speed and the encoding processing speed.

On the other hand, in the editing processing period for which the timeline processing index TE is "100", the encoding processing speed CE/QE only by the software encoder 152 is "1.33". At this time, since the maximum value of the editing processing speed MF = CE/TE is equal to "4", when the encoding processing speed HF of the hardware encoder 151 is "1", this value is smaller than the maximum value MF. Therefore, the data allocator 14 allocates video data to be encoded to both the

encoders

151 and 152 in the ratio of the number of frames HF:SF = 1:1. As a result, the encoding processing speed of the entire encoding unit 11B increases up to a maximum twice the encoding processing speed only by the hardware encoder 151 and increases up to a maximum 1.5 times the encoding processing speed only by the software encoder 152.

As another example, in the case where the CPU throughput index CE is "400" and the software encoding processing index SE is "1000", the encoding processing speed CE/QE only by the software encoder 152 during an editing processing period for which the timeline processing index TE is "200" is equal to "0.33". At this time, since the maximum value MF = CE/TE of the editing processing speed is equal to "2", when the encoding processing speed HF of the hardware encoder 151 is "1", this value is smaller than the maximum value MF. Therefore, the data allocator 14 allocates video data to be encoded to both the

encoders

151 and 152 in the ratio of the number of frames HF:SF = 1:0.17 = 6:1. As a result, the encoding processing speed of the entire encoding unit 11B increases up to a maximum 1.7 times the encoding processing speed only by the hardware encoder 151 and increases up to a maximum 3.5 times the encoding processing speed only by the software encoder 152.

FIG. 10 is a flow chart of processing for setting the allocation ratio of video data by the encoding unit 11B. According to the flow chart of FIG. 10, the CPU load estimator 23 predicts the transition of the timeline processing index TE based on edit information, and the allocation ratio setting unit 22B sets the ratio of the number of frames allocated to both the

encoders

151 and 152 based on the transition. Hereinafter, the prediction processing and setting processing will be described referring to FIG. 10. The encoding unit 11B starts the following processing when it receives edit information from the editor 13.

First, in step S21, the CPU load estimator 23 receives edit information from the editor 13 and divides the whole editing processing period of the editor 13 into periods, for which types of concurrent editing processings are common, based on the edit information.

Subsequently, in step S22, the CPU load estimator 23 first identifies the types of editing processings from the edit information for every divided period. The CPU load estimator 23 then searches the first database 24 for the CPU load indices corresponding to the identified types. The CPU load estimator 23 then adds the searched CPU load indices for every divided period and determines the sum as the timeline processing index TE of the divided period. Finally, the CPU load estimator 23 reads the time code indicating the start position of each divided period from the edit information, associates the time code with the timeline processing index TE in each divided period, and transmits the combination of the timeline processing index TE and the time code to the allocation ratio setting unit 22B.

Then, in step S23, the allocation ratio setting unit 22B calculates the maximum value of the editing processing speed MF = CE/TE for every timeline processing index TE received from the CPU load estimator 23.

Then, in step S24, the allocation ratio setting unit 22B compares each maximum value MF with the encoding processing speed HF of the hardware encoder 151. If the maximum value MF is smaller than or equal to the encoding processing speed HF (in the case of "Yes" in step S24), the process proceeds to step S25. If the maximum value MF is larger than the encoding processing speed HF (in the case of "No" in step S24), the process proceeds to step S26.

In step S25, the allocation ratio setting unit 22B sets the number of frames SF allocated to the software encoder 152 to "0". The process then proceeds to step S28.

In step S26, the allocation ratio setting unit 22B first calculates the upper limit RE = CE - TE x HF of the throughput of the CPU 10 which can be provided to the software encoder 152 per unit time. The allocation ratio setting unit 22B then calculates a value RE/QE, which is obtained by dividing the upper limit RE by the sum QE = SE + TE of the software encoding processing index SE and the timeline processing index TE, and sets it as the number of frames SF = RE/QE allocated to the software encoder 152 per unit time.

Then, in step S27, the allocation ratio setting unit 22B calculates the ratio HF:SF of the number of frames SF which has been set and the encoding processing speed HF of the hardware encoder 151. In general, since the ratio HF:SF is not an integer ratio, the allocation ratio setting unit 22B approximates the ratio HF:SF as the integer ratio HFI:SFI. Then, the process proceeds to step S28.

In step S28, the allocation ratio setting unit 22B associates the number of frames SF = 0 set in step S25 or the integer ratio HFI:SFI set in step S27 with a time code corresponding to the timeline processing index TE used for calculation of the maximum value MF.

Then, in step S29, the allocation ratio setting unit 22B checks whether or not there is a timeline processing index TE which has not been subjected to the processing of steps S23 to S28 in the timeline processing indices TE received from the CPU load estimator 29. If the unprocessed timeline processing index TE remains (in the case of "Yes" in step S29), the process returns to step S23. If the unprocessed timeline processing index TE does not remain (in the case of "No" in step S29), the process ends.

FIG. 11 is a flow chart of allocation processing of video data by the encoding unit 11B. The allocation ratio setting unit 22B and the data allocator 14 allocate video data streams to be encoded to both the

encoders

151 and 152 according to the flow chart of FIG. 11. Hereinafter, the allocation processing will be described referring to FIG. 11. The following processing can be started when, for example, the editor 13 begins to output a video data stream to be encoded.

In the first step S31, the allocation ratio setting unit 22B sets the time code whose playback time is earliest among the time codes received from the CPU load estimator 23 as a target time code Tr.

Then, in step S32, when one frame to be encoded is written from the editor 13 into the input buffer area BI in the memory unit 20, the data allocator 14 reads a time code Tc from the frame and transmits it to the allocation ratio setting unit 22B.

Then, in step S33, the allocation ratio setting unit 22B compares the time code Tc from the data allocator 14 with a value smaller by a predetermined amount than the target time code Tr. If the time code Tc from the data allocator 14 is larger than or equal to the value (Tr - predetermined amount) (in the case of "Yes" in step S33), the process proceeds to step S34. If the time code Tc from the data allocator 14 is smaller than the value (Tr - predetermined amount) (in the case of "No" in step S33), the process returns to step S32.

Then, in step S34, the allocation ratio setting unit 22B gives the number of frames SF = 0 or the integer ratio HFI:SFI corresponding to the target time code Tr to the data allocator 14. If the frame number SF = 0 is given, the process proceeds to step S35. If the integer ratio HFI:SFI is given, the process proceeds to step S36.

In step S35, the data allocator 14 gives all the frames to be encoded, which are stored in the input buffer area BI of the memory unit 20, to the hardware encoder 151 until the number of frames SF = 0 or the integer ratio HFI:SFI is newly input from the allocation ratio setting unit 22B. Then, the process proceeds to step S37.

In step S36, the integer ratio HFI:SFI is given to the data allocator 14. Accordingly, thereafter, whenever the data allocator 14 successively allocates SFI frames to the software encoder 152, the data allocator successively allocates HFI frames subsequent thereto to the hardware encoder 151. The data allocator 14 continues the allocation operation until the number of frames SF = 0 or the integer ratio HFI:SFI is newly input from the allocation ratio setting unit 22B. Then, the process proceeds to step S37.

In step S37, the data allocator 14 determines whether or not the allocated frame is the last frame of the video data stream to be encoded. If the frame is not the last one (in the case of "No" in step S37), the process returns to step S31. If the frame is the last one (in the case of "Yes" in step S37), the process ends. When the process is repeated from step S31, the allocation ratio setting unit 22B sets a time code received from the CPU load estimator 23 as the target time code Tr in order from the time code of the earliest playback time, for every repetition.

In the encoding apparatus 100 of the third embodiment, the CPU load estimator 23 predicts the transition of the timeline processing index TE based on edit information, and the allocation ratio setting unit 22B sets the ratio HF:SF of the number of frames allocated to both the

encoders

151 and 152 per unit time, based on the predicted transition of the timeline processing index TE. Accordingly, even if the load of the CPU 10 changes according to the content of editing processing of the editor 13, the encoding apparatus 100 can make the encoding processings of both the

encoders

151 and 152 performed in parallel while shortening the time for which each of the

encoders

151 and 152 waits for the input of frames to be encoded, or suppressing occurrence of such a time. As a result, the encoding apparatus 100 can perform its whole encoding processing at a faster speed than the encoding processing only by the hardware encoder 151. Thus, the encoding processing speed of the video editing system 200 can be improved.

<<Fourth embodiment>>
An encoding apparatus 100 according to a fourth embodiment of the present invention is built in a non-linear video editing system 200, similar to that according to the first embodiment. The video editing system 200 is the same as that according to the first embodiment except for an encoding unit. Therefore, the explanation in the first embodiment is cited for details of the same components.

FIG. 12 is a block diagram of an encoding unit 11C according to the fourth embodiment of the present invention. Referring to FIG. 12, the encoding unit 11C includes a data allocator 14, a hardware encoder 151, a software encoder 152, a synthesizer 16, an output unit 17, a CPU load detector 25, an allocation ratio setting unit 22C, and a second database 26. The encoding unit 11C includes the same components as the components shown in FIG. 3 except for the CPU load detector 25, the allocation ratio setting unit 22C, and the second database 26. In FIG. 12, the same reference symbols as the reference symbols shown in FIG. 3 are given to the same components, and the detailed explanation thereof is cited.

The function of the CPU load detector 25 is realized by the CPU 10, which controls other components of a computer terminal according to a predetermined program. The CPU load detector 25 measures the usage rate of the CPU 10 at a predetermined time interval using API provided by an OS 27 of the computer terminal. Additionally, the CPU load detector 25 evaluates the load from the usage rate of the CPU 10 and notifies the allocation ratio setting unit 22C of the load in real time. For example, the CPU load detector 25 evaluates the load of the CPU 10 by dividing the total range 0 to 100%, to which the value of the usage rate can be set, into several steps and determining to which step the measured usage rate belongs. Additionally, if the step where the measured usage rate of the CPU 10 belongs has changed, the CPU load detector 25 notifies the allocation ratio setting unit 22C of the change and the type of step after the change. For example, the case where the usage rate of the CPU 10 are divided into three steps of 0 to 20%, 20 to 80%, and 80 to 100% is considered. If the measured usage rate of the CPU 10 has changed from the first step of 0 to 20% to the second step of 20 to 80%, the CPU load detector 25 notifies the allocation ratio setting unit 22C that the change has occurred and that the step after the change is the second step of 20 to 80%.

The allocation ratio setting unit 22C determines the ratio HF:SF of number of frames allocated to both the

encoders

151 and 152 per unit time according to the step after the change, which is notified from the CPU load detector 25. The ratio of the number of frames HF:SF is evaluated beforehand for every step of the usage rate and is stored in the memory unit 20 as the second database 26. In the second database 26, the integer ratio HFI:SFI is stored as the ratio of the number of frames HF:SF. For example, the sum HFI + SFI of two integers of each integer ratio HFI:SFI is smaller than the total number of frames which are edited in a period corresponding to the time interval during which the usage rate of the CPU 10 is measured. The higher the step of the usage rate of the CPU 10 is, the higher the ratio of the number of frames HF:SF is set. That is, the heavier the load of the CPU 10 is, the smaller the number of frames SF allocated to the software encoder 152 per unit time is. The allocation ratio setting unit 22C searches the second database 26 for the ratio of the number of frames corresponding to the step of the usage rate notified from the CPU load detector 25. Additionally, the allocation ratio setting unit 22C transmits the searched ratio of the number of frames to the data allocator 14. Thus, the data allocator 14 allocates video data to be encoded to both the

encoders

151 and 152 according to the new ratio of the number of frames.

FIG. 13 is a flow chart of allocation processing of video data by the encoding unit 11C. According to the flow chart of FIG. 13, the CPU load detector 25, the allocation ratio setting unit 22C, and the data allocator 14 change the ratio of the number of frames allocated to both the

encoders

151 and 152 per unit time according to the usage rate of the CPU 10. Hereinafter, processing for setting the ratio of the number of frames will be described referring to FIG. 13. The following processing can be started when, for example, the editor 13 begins to output a video data stream to be encoded.

First, in step S41, the CPU load detector 25 measures the usage rate of the CPU 10 using API of the OS 27.

Subsequently, in step S42, the CPU load detector 25 identifies the step where the measured usage rate of the CPU 10 belongs and determines whether it is the same as the former step. If the step has changed (in the case of "Yes" in step S42), the process proceeds to step S43. If the step is the same (in the case of "No" in step S42), the process returns to step S41.

Then, in step S43, the CPU load detector 25 notifies the allocation ratio setting unit 22C of the fact that the step where the usage rate of the CPU 10 belongs has changed together with the type of the step after the change. The allocation ratio setting unit 22C searches the second database 26 for the ratio of the number of frames corresponding to the notified step of the usage rate. Additionally, the allocation ratio setting unit 22C transmits the searched ratio of the number of frames to the data allocator 14.

Then, in step S44, the data allocator 14 allocates video data to be encoded to both the

encoders

151 and 152 according to the new ratio of the number of frames.

Then, in step S45, the data allocator 14 determines whether or not the allocated frame is the last frame of the video data stream to be encoded. If the frame is not the last one (in the case of "No" in step S45), the process returns to step S41. If the frame is the last one (in the case of "Yes" in step S45), the process ends.

The loop of steps S41 to S45 is repeated at a predetermined time interval. Accordingly, the ratio of video data allocated to both the

encoders

151 and 152 is dynamically adjusted according to the actual load variation of the CPU 10.

In the encoding apparatus 100 of the fourth embodiment, the CPU load detector 25 measures the step of the usage rate of the CPU 10 in real time, and the allocation ratio setting unit 22C sets the ratio HF:SF of number of frames allocated to both the

encoders

151 and 152 per unit time according to the measured step of the usage rate of the CPU 10. As a result, regardless of the usage ratio, that is, load variation, of the CPU 10, the encoding apparatus 100 can make the encoding processings of both the

encoders

151 and 152 waits for the input of video data to be encoded, or suppressing occurrence of such a time. Accordingly, the encoding apparatus 100 can perform its whole encoding processing at a faster speed than the encoding processing only by the hardware encoder 151. Thus, the encoding processing speed of the video editing system 200 can be improved.

<<Fifth embodiment>>
An encoding apparatus 100 according to a fifth embodiment of the present invention is built in a non-linear video editing system 200, similar to that according to the first embodiment. The video editing system 200 is the same as that according to the first embodiment except for an encoding unit. Therefore, the explanation in the first embodiment is cited for details of the same components.

FIG. 14 is a block diagram of an encoding unit 11D according to the fifth embodiment of the present invention. Referring to FIG. 14, the encoding unit 11D includes a data allocator 14, a hardware encoder 151, a software encoder 152, a synthesizer 16, an output unit 17, a CPU load estimator 23, an allocation ratio setting unit 22D, a first database 24, a CPU load detector 25, and a second database 26. The encoding unit 11D substantially has the components of the third embodiment of FIG. 8 and the components of the fourth embodiment of FIG. 12. In FIG. 14, the same reference symbols as the reference symbols shown in FIGS. 8 and 12 are given to the same components, and the explanations of FIGS. 3, 8, and 12 are cited for the details thereof.

Similar to the allocation ratio setting unit 22B according to the third embodiment, the allocation ratio setting unit 22D sets the ratio of video data allocated to both the

encoders

151 and 152 per unit time by the data allocator 14, based on the transition of the load of the CPU 10 predicted by the CPU load estimator 23, that is, the transition of the timeline processing index TE.

Moreover, similar to the allocation ratio setting unit 22B according to the third embodiment, whenever a time code is transmitted from the data allocator 14, the allocation ratio setting unit 22D compares the time code with each time code received from the CPU load estimator 23. On the other hand, the allocation ratio setting unit 22D monitors the usage rate of the CPU 10 through the CPU load detector 25. Whenever the step of the usage rate of the CPU 10 changes, the allocation ratio setting unit 22D determines the CPU load index corresponding to the step after the change. Here, the CPU load index is evaluated beforehand for every step of the usage rate and is recorded in the second database 26. The allocation ratio setting unit 22D searches the second database 26 for the CPU load index corresponding to the step of the usage rate after the change.

If the time code from the data allocator 14 reaches a value smaller by a predetermined amount than the time code from the CPU load estimator 23, the allocation ratio setting unit 22D compares the timeline processing index TE corresponding to the time code from the CPU load estimator 23 with a CPU load index LE corresponding to the step of the usage rate of the CPU 10 at that point of time. If the timeline processing index TE is larger than the CPU load index LE, the allocation ratio setting unit 22D gives the number of frames SF = 0 or the integer ratio HFI:SFI obtained from the timeline processing index TE to the data allocator 14, similar to the allocation ratio setting unit 22B according to the third embodiment. On the other hand, if the timeline processing index TE is smaller than or equal to the CPU load index LE, the allocation ratio setting unit 22D searches the second database 26 for the ratio of the number of frames corresponding to the step of the usage rate of the CPU 10 and transmits it to the data allocator 14, similar to the allocation ratio setting unit 22C according to the fourth embodiment. In this way, when the actual load of the CPU 10 is approximately equal to or larger than the load predicted from edit information, video data is allocated to both the

encoders

151 and 152 not in the ratio of the number of frames set based on the edit information but in the ratio of the number of frames suitable for the actual load. Thereby, a drop in the processing speed of the entire video editing system 200 resulting from the prediction error of the load of the CPU 10 is prevented.

FIG. 15 is a flow chart of allocation processing of video data by the encoding unit 11D. The CPU load estimator 23, the CPU load detector 25, the allocation ratio setting unit 22D, and the data allocator 14 change the ratio of the number of frames allocated to both the

encoders

151 and 152 per unit time according to the flow chart of FIG. 15. Hereinafter, processing for setting the ratio of the number of frames will be described referring to FIG. 15. Note that FIG. 15 includes the same steps as the steps shown in FIGS. 11 and 13. Therefore, in FIG. 15, the same reference symbols as the reference symbols shown in FIGS. 11 and 13 are given to the same steps. Additionally, the explanations of FIGS. 11 and 13 are cited for details of the same steps. The following processing can be started when, for example, the editor 13 begins to output a video data stream to be encoded.

First, in step S31, the allocation ratio setting unit 22D sets a time code which has been received from the CPU load estimator 23 as a target time code Tr in order from the earliest playback time.

Then, in step S32, the allocation ratio setting unit 22D reads the time code Tc from the data allocator 14.

In the next step S33, the allocation ratio setting unit 22D compares the time code Tc with a value smaller by a predetermined amount than the target time code Tr. If the time code Tc is larger than or equal to the value (Tr - predetermined amount) (in the case of "Yes" in step S33), the process proceeds to step S51. If the time code Tc is smaller than the value (Tr - predetermined amount) (in the case of "No" in step S33), the process is repeated from step S32.

In step S51, the allocation ratio setting unit 22D calculates the timeline processing index TE corresponding to the target time code Tr.

Then, in step S52, the CPU load detector 25 measures the usage rate of the CPU 10 at a predetermined time interval using API of the OS 27. Additionally, the CPU load detector 25 identifies the step where the measured usage rate of the CPU 10 belongs and determines whether it is the same as the former step. If the step has changed, the CPU load detector 25 notifies the allocation ratio setting unit 22D of the fact that the step where the usage rate of the CPU 10 belongs has changed together with the type of the step after the change. The allocation ratio setting unit 22D searches the second database 26 for the CPU load index LE corresponding to the notified step of the usage rate.

Then, in step S53, the allocation ratio setting unit 22D compares the timeline processing index TE with the CPU load index LE. If the timeline processing index TE is larger than the CPU load index LE (in the case of "Yes" in step S53), the process proceeds to step S34. Therefore, similar to the steps S34 to S37 according to the third embodiment, video data is allocated to both the

encoders

151 and 152 according to the number of frames SF = 0 or the integer ratio HFI:SFI obtained from the timeline processing index TE. On the other hand, if the timeline processing index TE is smaller than or equal to the CPU load index LE (in the case of "No" in step S53), the process proceeds to step S43. Thus, similar to the steps S43 to S45 according to the fourth embodiment, video data is allocated to both the

encoders

151 and 152 in the ratio of the number of frames corresponding to the step of the usage rate of the CPU 10.

In the encoding apparatus 100 of the fifth embodiment, first, the CPU load estimator 23 predicts the transition of the timeline processing index TE based on edit information, and the allocation ratio setting unit 22D sets the ratio HF:SF of the number of frames allocated to both the

encoders

151 and 152 per unit time based on the predicted transition of the timeline processing index TE. Then, in the encoding apparatus 100, the CPU load detector 25 measures the step of the usage rate of the CPU 10 in real time, and the allocation ratio setting unit 22D estimates the actual load LE of the CPU 10 from the measured step of the usage rate of the CPU 10 and compares it with the timeline processing index TE. If the actual load LE of the CPU 10 is smaller than the timeline processing index TE, the encoding apparatus 100 allocates video data to be encoded to both the

encoders

151 and 152 in the ratio of the number of frames HF:SF set based on the timeline processing index TE. On the other hand, if the actual load LE of the CPU 10 is approximately equal to or larger than the timeline processing index TE, the encoding apparatus 100 allocates the video data to be encoded to both the

encoders

151 and 152 in the ratio of the number of frames which is associated with the actual load LE of the CPU 10 beforehand. Accordingly, even if the load of the CPU 10 changes more than or equal to the load predicted from the content of editing processing of the editor 13, the encoding apparatus 100 can make the encoding processings of both the

encoders

<<Sixth embodiment>>
An encoding apparatus 100 according to a sixth embodiment of the present invention is built in a non-linear video editing system 200, similar to that according to the first embodiment. The video editing system 200 is the same as that according to the first embodiment except for an encoding unit. Therefore, the explanation in the first embodiment is cited for details of the same components.

FIG. 16 is a block diagram of an encoding unit 11E according to the sixth embodiment of the present invention. Referring to FIG. 16, the encoding unit 11E includes a data allocator 14, a hardware encoder 151, a software encoder 152, a synthesizer 16, an output unit 17, a processing speed measurer 28, and an allocation ratio setting unit 22E. The encoding unit 11E includes the same components as the components shown in FIG. 3 except for the processing speed measurer 28 and the allocation ratio setting unit 22E. In FIG. 16, the same reference symbols as the reference symbols shown in FIG. 3 are given to the same components, and the explanation of FIG. 3 is cited for the details thereof.

The function of the processing speed measurer 28 is realized by the CPU 10, which controls other components of a computer terminal according to a predetermined program. The processing speed measurer 28 measures a time from when a data unit to be encoded is allocated to each of the

encoders

151 and 152 by the data allocator 14 to when the data unit has been encoded and the encoded data unit is output from each of the

encoders

151 and 152 to the synthesizer 16. Specifically, the processing speed measurer 28 first starts measurement of the time from when the number assigned to a data unit which is allocated to each of the

encoders

151 and 152 and an address of a writing destination of the encoded data unit are transmitted from the data allocator 14 to the synthesizer 16. The processing speed measurer 28 then ends the measurement of the time when the same address as that address is output from one of the

encoders

151 and 152 to the synthesizer 16. Additionally, the processing speed measurer 28 calculates the encoding processing speed of each of the

encoders

151 and 152 from the measured time and transmits it to the allocation ratio setting unit 22E. Alternatively, the processing speed measurer 28 may measure the bit rate of video data output from each of the

encoders

151 and 152 and calculate the encoding processing speed of each of the

encoders

151 and 152 from the obtained bit rate.

The allocation ratio setting unit 22E calculates a ratio HF1/SF1 of encoding processing speed between both the

encoders

151 and 152 based on an encoding processing speed HF1 of the hardware encoder 151 and an encoding processing speed SF1 of the software encoder 152, and compares it with a ratio HF2/SF2 of the number of frames allocated to the hardware encoder 151 and the software encoder 152 per unit time at that point of time. If the ratio of encoding processing speed HF1/SF1 is different from the ratio of the number of frames HF2/SF2, the allocation ratio setting unit 22E transmits the ratio of encoding processing speed HF1/SF1 to the data allocator 14 as a ratio of the number of frames allocated to both the

encoders

151 and 152 per unit time. Here, since the ratio of encoding processing speed is generally not an integer ratio, the allocation ratio setting unit 22E approximates the ratio of encoding processing speed as an integer ratio beforehand.

FIG. 17 is a flow chart of allocation processing of video data by the encoding unit 11E. According to the flow chart of FIG. 17, the processing speed measurer 28, the allocation ratio setting unit 22E, and the data allocator 14 adjust the ratio of the number of frames allocated to both the

encoders

151 and 152 per unit time to the ratio of encoding processing speed of the

encoders

151 and 152. Hereinafter, the adjustment processing will be described referring to FIG. 17. The following processing can be started when, for example, the editor 13 begins to output a video data stream to be encoded.

First, in step S61, the processing speed measurer 28 measures a time from when a data unit to be encoded is allocated to each of the

encoders

151 and 152 from the measured time and transmits it to the allocation rate setting unit 22E.

Then, in step S62, the allocation ratio setting unit 22E calculates the ratio of encoding processing speed between both the

encoders

151 and 152 based on the encoding processing speeds received from the processing speed measurer 28, and approximates the ratio as an integer ratio. Additionally, the allocation ratio setting unit 22E compares the ratio of encoding processing speed with the ratio of the number of frames allocated to both the

encoders

151 and 152 per unit time at that point of time. If the ratio of encoding processing speed HF1/SF1 is different from the ratio of the number of frames HF2/SF2 (in the case of "Yes" in step S62), the process proceeds to step S63. If the ratio of encoding processing speed HF1/SF1 is equal to the ratio of the number of frames HF2/SF2 (in the case of "No" in step S62), the process returns to step S61.

In step S63, the allocation ratio setting unit 22E transmits the ratio of encoding processing speed to the data allocator 14 as a new value of the ratio of the number of frames allocated to both the

encoders

151 and 152 per unit time.

Then, in step S64, the data allocator 14 allocates video data to be encoded to both the

encoders

151 and 152 in the newly set ratio of the number of frames.

Then, in step S65, the data allocator 14 determines whether or not the allocated frame is the last frame of the video data stream to be encoded. If the frame is not the last one (in the case of "No" in step S65), the process returns to step S61. If the frame is the last one (in the case of "Yes" in step S65), the process ends. The loop of steps S61 to S65 is repeated at a predetermined time interval whenever video data is output from each of the

encoders

151 and 152 to the synthesizer 16.

In the encoding apparatus 100 according to the sixth embodiment, the processing speed measurer 28 measures the actual encoding processing speed of each of the

encoders

151 and 152, and the allocation ratio setting unit 22E adjusts the ratio of the number of frames allocated to both the

encoders

151 and 152 to the ratio of actual encoding processing speed between both the

encoders

151 and 152. Accordingly, regardless of a variation in the encoding processing speed of the software encoder 152 caused by a variation in the load of the CPU 10, the encoding apparatus 100 can make the encoding processings of both the

encoders

151 and 152 performed in parallel while shortening the time for which the synthesizer 16 waits for the input of frames encoded by each of the

encoders

151 and 152, or suppressing occurrence of such a time. As a result, the encoding apparatus 100 can perform its whole encoding processing at a faster speed than the encoding processing only by the hardware encoder 151. Thus, the encoding processing speed of the video editing system 200 can be improved.

<<Seventh embodiment>>
An encoding apparatus 100 according to a seventh embodiment of the present invention is built in a non-linear video editing system 200, similar to that according to the first embodiment. The video editing system 200 is the same as that according to the first embodiment except for an encoding unit. Therefore, the explanation in the first embodiment is cited for details of the same components.

FIG. 18 is a block diagram of an encoding unit 11F according to the seventh embodiment of the present invention. Referring to FIG. 18, the encoding unit 11F includes a data allocator 14, a hardware encoder 151, a software encoder 152, a synthesizer 16, an output unit 17, a CPU load detector 25, an allocation rate setting unit 22F, a processing speed measurer 28, and a second database 26. The encoding unit 11F substantially has the components of the fourth embodiment of FIG. 12 and the components of the sixth embodiment of FIG. 16. In FIG. 18, the same reference symbols as the reference symbols shown in FIGS. 12 and 16 are given to the same components, and the explanations of FIGS. 3, 12, and 16 are cited for the details thereof.

Similar to the allocation ratio setting unit 22E according to the sixth embodiment, the allocation rate setting unit 22F calculates the ratio HF1/SF1 of encoding processing speed between both the

encoders

151 and 152 based on the encoding processing speeds received from the processing speed measurer 28 and compares it with the ratio HF2/SF2 of number of frames allocated to both the

encoders

151 and 152 per unit time at that point of time. On the other hand, the allocation rate setting unit 22F monitors the usage rate of the CPU 10 through the CPU load detector 25.

If the ratio of encoding processing speed HF1/SF1 is different from the ratio of the number of frames HF2/SF2, the allocation rate setting unit 22F searches the second database 26 for a ratio of the number of frames HF3/SF3 corresponding to the step of the usage rate of the CPU 10. Additionally, the allocation ratio setting unit 22F compares the searched ratio of the number of frames HF3/SF3 with the ratio of encoding processing speed HF1/SF1. If the ratio of encoding processing speed HF1/SF1 is larger than the searched ratio of the number of frames HF3/SF3, the allocation rate setting unit 22F transmits the ratio of encoding processing speed HF1/SF1 to the data allocator 14 as a ratio of the number of frames allocated to both the

encoders

151 and 152 per unit time, similar to the allocation ratio setting unit 22E according to the sixth embodiment. In contrast, if the ratio of encoding processing speed HF1/SF1 is smaller than or equal to the searched ratio of the number of frames HF3/SF3, the allocation rate setting unit 22F transmits the ratio of the number of frames HF3/SF3 corresponding to the step of the usage rate of the CPU 10 to the data allocator 14 as a ratio of the number of frames allocated to both the

encoders

151 and 152 per unit time, similar to the allocation ratio setting unit 22C according to the fourth embodiment.

In this way, if the load indicated by the usage rate of the CPU 10 is approximately equal to or larger than the load indicated by the ratio HF1/SF1 of encoding processing speed between both the

encoders

151 and 152, the allocation ratio setting unit 22F allocates video data to be encoded to both the

encoders

151 and 152 not in the ratio of encoding processing speed HF1/SF1 but in the ratio of the number of frames HF3/SF3 suitable for the load indicated by the usage rate.

FIG. 19 is a flow chart of allocation processing of video data by the encoding unit 11F. The processing speed measurer 28, the CPU load detector 25, the allocation rate setting unit 22F, and the data allocator 14 change the ratio of the number of frames allocated to both the

encoders

151 and 152 per unit time according to the flow chart of FIG. 19. Hereinafter, processing for setting the number of frames will be described referring to FIG. 19. Note that FIG. 19 includes the same steps as the steps shown in FIG. 17. Therefore, in FIG. 19, the same reference symbols as the reference symbols shown in FIG. 17 are given to the same steps. Additionally, the explanation of FIG. 17 is cited for details of the same steps. The following processing can be started when, for example, the editor 13 begins to output a video data stream to be encoded.

First, in step S61, the processing speed measurer 28 calculates the encoding processing speed of each of the

encoders

151 and 152 based on a time from when a data unit to be encoded is allocated to each of the

encoders

151 and 152 to when the encoded data unit is output to the synthesizer 16, and transmits it to the allocation ratio setting unit 22F.

Then, in step S62, the allocation ratio setting unit 22F calculates the ratio HF1/SF1 of encoding processing speed between both the

encoders

151 and 152 based on the encoding processing speeds of the

respective encoders

151 and 152 and approximates the ratio as an integer ratio. Additionally, if the ratio HF1/SF1 of encoding processing speed is different from the ratio HF2/SF2 of number of frames allocated to both the

encoders

151 and 152 per unit time at that point of time (in the case of "Yes" in step S62), the allocation ratio setting unit 22F moves the processing to step S71, and if the ratio of encoding processing speed HF1/SF1 is equal to the ratio of the number of frames HF2/SF2 (in the case of "No" in step S62), the allocation ratio setting unit 22F returns the processing to step S61.

Then, in step S71, the CPU load detector 25 measures the usage rate of the CPU 10 at a predetermined time interval using API of the OS 27. Additionally, the CPU load detector 25 identifies the step where the measured usage rate of the CPU 10 belongs and determines whether it is the same as the former step. If the step has changed, the CPU load detector 25 notifies the allocation rate setting unit 22F of the fact that the step where the usage rate of the CPU 10 belongs has changed together with the type of the step after the change. The allocation rate setting unit 22F searches the second database 26 for the ratio of the number of frames HF3/SF3 corresponding to the notified step of the usage rate.

Then, in step S72, the allocation ratio setting unit 22F compares the searched ratio of the number of frames HF3/SF3 with the ratio of encoding processing speed HF1/SF1. If the ratio of encoding processing speed HF1/SF1 is larger than the searched ratio of the number of frames HF3/SF3 (in the case of "Yes" in step S72), the process proceeds to step S63. Thus, the ratio of encoding processing speed HF1/SF1 is transmitted to the data allocator 14. In contrast, if the ratio of encoding processing speed HF1/SF1 is smaller than or equal to the searched ratio of the number of frames HF3/SF3 (in the case of "No" in step S72), the process proceeds to step S73. Thus, the searched ratio of the number of frames HF3/SF3 is transmitted to the data allocator 14. Accordingly, in step S64, the data allocator 14 allocates video data to both the

encoders

151 and 152 in the newly set ratio of the number of frames, similar to that in the sixth embodiment.

In the encoding apparatus 100 according to the seventh embodiment, first, the processing speed measurer 28 measures the actual encoding processing speeds of the

respective encoders

151 and 152, and the allocation ratio setting unit 22F sets the ratio of the number of frames allocated to both the

encoders

151 and 152 to the ratio HF1/SF1 of actual encoding processing speed between both the

encoders

151 and 152. Then, in the encoding apparatus 100, the CPU load detector 25 measures the step of the usage rate of the CPU 10 in real time, and the allocation rate setting unit 22F compares the ratio of the number of frames HF3/SF3 corresponding to the measured step of the usage rate of the CPU 10 with the ratio of encoding processing speed HF1/SF1. If the ratio of the number of frames HF3/SF3 corresponding to the step of the usage rate of the CPU 10 is smaller than the ratio of encoding processing speed HF1/SF1, the encoding apparatus 100 allocates video data to be encoded to both the

encoders

151 and 152 in the ratio of encoding processing speed HF1/SF1. In contrast, if the ratio of the number of frames HF3/SF3 corresponding to the step of the usage rate of the CPU 10 is approximately equal to or larger than the ratio of encoding processing speed HF1/SF1, the encoding apparatus 100 allocates video data to be encoded to both the

encoders

151 and 152 in the ratio of the number of frames HF3/SF3 corresponding to the step of the usage rate. Accordingly, regardless of a variation in the encoding processing speed of the software encoder 152 caused by a variation in the load of the CPU 10, the encoding apparatus 100 can make the encoding processings of both the

encoders

151 and 152 performed in parallel while shortening the time for which the synthesizer 16 waits for the input of frames encoded by the

respective encoders

<<Eighth embodiment>>
An encoding apparatus 100 according to an eighth embodiment of the present invention is built in a non-linear video editing system 200, similar to that according to the first embodiment. The video editing system 200 is the same as that according to the first embodiment except for an encoding unit. Therefore, the explanation in the first embodiment is cited for details of the same components.

FIG. 20 is a block diagram of an encoding unit 11G according to the eighth embodiment of the present invention. Referring to FIG. 20, the encoding unit 11G includes a data allocator 14, a hardware encoder 151, a software encoder 152, a synthesizer 16, an output unit 17, and a switching position detector 29. The encoding unit 11G includes the same components as the components shown in FIG. 3 except for the switching position detector 29. In FIG. 20, the same reference symbols as the reference symbols shown in FIG. 3 are given to the same components, and the explanation of FIG. 3 is cited for the details thereof.

The function of the switching position detector 29 is realized by the CPU 10, which controls other components of a computer terminal according to a predetermined program. The switching position detector 29 detects a position at which a scene is switched or a position at which the type of editing processing changes from a video data stream to be encoded, which is written from the editor 13 into the memory unit 20, and generates a detection signal according to the detection.

The switching position detector 29 receives the edit information from the editor 13 prior to the editing processing of the editor 13. Based on the edit information, the switching position detector 29 sets a time code of a frame in which a scene is switched in the video data stream edited by the editor 13 as a switching position, and sets a time code of a frame in which the content of editing processing, such as a decoding scheme of video data to be edited and/or the content of effects processing, is switched in the video data stream edited by the editor 13 as a change position.

Whenever one frame to be encoded is written from the editor 13 into the memory unit 20, the data allocator 14 reads a time code from the frame and transmits it to the switching position detector 29. The switching position detector 29 compares the time code with time codes indicating respective switching positions and respective change positions. If the time code from the data allocator 14 matches a time code indicating either one of the switching positions or the change positions (hereinafter abbreviated as "switching/change positions"), the switching position detector 29 outputs a detection signal to the data allocator 14. According to the detection signal, the data allocator 14 switches the output destination of the frame and frames subsequent thereto to another encoder, which is different from the encoder to which the immediately preceding frame is output.

Note that when a series of video data included between two consecutive switching/change positions is large, the switching position detector 29 may set one of boundaries between frames in the video data, as a new switching position. Accordingly, buffer areas for video data allocated to the

respective encoders

151 and 152 can be maintained small. Furthermore, in order to more reliably increase the encoding processing speed by parallel use of both the

encoders

151 and 152, it is preferable that the switching position detector 29 adjust the ratio of video data allocated to both the

encoders

151 and 152.

FIG. 21 is a flow chart of allocation processing of video data by the encoding unit 11G. The switching position detector 29 and the data allocator 14 allocate video data to be encoded to both the

encoders

151 and 152 at the switching/change positions according to the flow chart of FIG. 21. Hereinafter, the allocation processing will be described referring to FIG. 21. The following processing can be started when, for example, the editor 13 begins to output a video data stream to be encoded.

First, in step S81, the switching position detector 29 receives edit information from the editor 13 and sets time codes indicating switching/change positions for the video data stream edited by the editor 13 based on the edit information. Thereafter, the numbers sw = 0, 1, 2, ... are allocated to the respective time codes Tc in order from the earlier playback time.

Then, in step S82, the switching position detector 29 initializes the number sw to "0".

Then, in step S83, the data allocator 14 initializes the output destination of video data to be encoded to the hardware encoder 151.

Then, in step S84, whenever the editor 13 writes one frame to be encoded into the memory unit 20, the data allocator 14 reads the time code Tc from the frame and transmits it to the switching position detector 29.

Then, in step S85, the switching position detector 29 compares the time code Tc received from the data allocator 14 with a time code T(sw) of the sw-th switching/change position. If the time code Tc from the data allocator 14 is larger than or equal to the time code T(sw) of the sw-th switching/change position (in the case of "Yes" in step S85), the process proceeds to step S86. If the time code Tc is smaller than the time code T(sw) of the sw-th switching/change position (in the case of "No" in step S85), the process returns to step S84. In this case, the data allocator 14 allocates the frame having the time code Tc to the hardware encoder 151.

Then, in step S86, the switching position detector 29 generates a detection signal. According to the detection signal, the data allocator 14 allocates the frame having the time code Tc and frames subsequent thereto to the software encoder 152.

Then, in step S87, whenever the editor 13 writes one frame to be encoded into the memory unit 20, the data allocator 14 reads the time code Tc from the frame and transmits it to the switching position detector 29.

Then, in step S88, the switching position detector 29 compares the time code Tc received from the data allocator 14 with a time code T(sw+1) of the (sw+1)-th switching/change position. If the time code Tc from the data allocator 14 is larger than or equal to the time code T(sw+1) of the (sw+1)-th switching/change position (in the case of "Yes" in step S88), the process proceeds to step S89. If the time code Tc is smaller than the time code T(sw+1) of the (sw+1)-th switching/change position (in the case of "No" in step S88), the process returns to step S87. In this case, the data allocator 14 allocates the frame having the time code Tc to the software encoder 152.

Then, in step S89, the switching position detector 29 generates a detection signal. According to the detection signal, the data allocator 14 allocates the frame having the time code Tc and frames subsequent thereto to the hardware encoder 151.

Then, in step S90, the switching position detector 29 increments the number sw by "2".

Then, in step S91, the data allocator 14 determines whether or not the allocated frame is the last frame of the video data stream to be encoded. If the frame is not the last one (in the case of "No" in step S91), the process is repeated from step S84. If the frame is the last one (in the case of "Yes" in step S91), the process ends.

In the encoding apparatus 100 of the eighth embodiment, the switching position detector 29 sets the time code T(sw), which indicates the switching/change position for a video data stream to be encoded, based on edit information, and alternately switches the output destination of the frame to be encoded to both the

encoders

151 and 152 whenever the time code Tc of the frame reaches the time code T(sw) of a switching/change position. That is, the encoding apparatus 100 of the eighth embodiment allocates the video data stream to be encoded to both the

encoders

151 and 152 at the switching/change positions. Accordingly, since the encoding apparatus 100 encodes a series of scenes or the whole video data, on which the same editing processing continues, with the same encoder, the image quality can be made substantially uniform. Moreover, even if the change in the image quality resulting from switching between the

encoders

151 and 152 exists more or less, it is hard to be sensed to a viewer since it is restricted to the switching/change positions. As a result, the encoding apparatus 100 can perform encoding processing of the entire encoding apparatus 100 at a faster speed than the encoding processing only by the hardware encoder 151 by performing the encoding processings of both the

encoders

151 and 152 in parallel, while maintaining uniform image quality of the video data stream that the viewer feels. Thus, the encoding processing speed of the video editing system 200 can be improved.

Other than the first to eighth embodiments described above, the encoding unit according to an embodiment of the present invention may be formed by a combination of two or more of the first to eighth embodiments. For example, the fifth embodiment may be combined with the seventh embodiment, and the eighth embodiment may be combined with any one of the second to seventh embodiments.

The encoding apparatus 100 according to the embodiments of the present invention may be implemented as a real-time encoder in a video capture or a digital video camera, instead of the above-described video editing system 200. In this case, the encoding apparatus includes an input unit instead of the editor 13 in the configuration shown in FIG. 3. The input unit acquires video data from a removable disc, such as a DVD or a camera. Alternatively, the input unit may acquire video data from a resource on a network through a network interface. The input unit transmits the acquired video data to the data allocator 14. Note that a CPU of a computer terminal may be mounted on the same board on which the hardware encoder is mounted.

Note that the following items are further disclosed regarding the above description.
(Item 1)
An encoding apparatus for encoding AV data including sound data and/or video data, comprising:
a CPU;
a hardware encoder configured by hardware for encoding processing, the hardware encoder encoding a part of the AV data;
a software encoder for encoding another part of the AV data using the CPU in parallel with encoding processing of the hardware encoder;
a data allocator for allocating the AV data to the hardware encoder and the software encoder;
a synthesizer for arranging AV data respectively encoded by the hardware encoder and the software encoder in a predetermined order to synthesize the encoded AV data into a series of encoded AV data; and
an output unit for outputting the series of encoded AV data,
wherein AV data is encoded at a faster encoding processing speed than an encoding processing speed only by the hardware encoder.
(Item 2)
The encoding apparatus according to item 1, wherein an encoding scheme of the hardware encoder is the same as an encoding scheme of the software encoder.
(Item 3)
The encoding apparatus according to item 1, wherein AV data encoded by the hardware encoder and AV data encoded by the software encoder are not predictable from each other.
(Item 4)
The encoding apparatus according to item 1, further comprising a buffer connected between the data allocator and the hardware encoder, the buffer storing AV data allocated to the hardware encoder by the data allocator and supplying the AV data to the hardware encoder,
wherein the data allocator acquires space information of the buffer and allocates the AV data to the software encoder based on the space information.
(Item 5)
The encoding apparatus according to item 1,
wherein data indicating a ratio of the number of data units of AV data allocated to the hardware encoder per unit time and the number of data units of AV data allocated to the software encoder per unit time is supplied to the data allocator, and
the data allocator allocates AV data to the hardware encoder and the software encoder based on the data.
(Item 6)
The encoding apparatus according to item 1,
wherein the CPU performs another processing on the AV data other than encoding processing,
the encoding apparatus further comprises:
a CPU load estimator for predicting a load of the CPU caused by the other processing so as to correspond to the AV data and;
an allocation ratio setting unit for calculating a ratio of the number of data units of AV data allocated to the hardware encoder per unit time and the number of data units of AV data allocated to the software encoder per unit time according to the predicted load of the CPU, and supplying data indicating the ratio to the data allocator, and
the data allocator allocates AV data to the hardware encoder and the software encoder based on the data.
(Item 7)
The encoding apparatus according to item 6,
wherein the allocation ratio setting unit detects the load of the CPU and corrects the ratio based on the detected load.
(Item 8)
The encoding apparatus according to item 1, further comprising an allocation ratio setting unit for detecting a load of the CPU, calculating a ratio of the number of data units of AV data allocated to the hardware encoder per unit time and the number of data units of AV data allocated to the software encoder per unit time according to the detected load, and supplying data indicating the ratio to the data allocator,
wherein the data allocator allocates AV data to the hardware encoder and the software encoder based on the data.
(Item 9)
The encoding apparatus according to item 1, further comprising:
a processing speed measurer for measuring an encoding processing speed of the hardware encoder and an encoding processing speed of the software encoder; and
an allocation ratio setting unit for calculating a ratio of the number of data units of AV data allocated to the hardware encoder per unit time and the number of data units of AV data allocated to the software encoder per unit time according to the measured encoding processing speeds, and supplying data indicating the ratio to the data allocator,
wherein the data allocator allocates AV data to the hardware encoder and the software encoder based on the data.
(Item 10)
The encoding apparatus according to item 9,
wherein the allocation ratio setting unit detects a load of the CPU and corrects the ratio according to the detected load.
(Item 11)
The encoding apparatus according to item 1, further comprising a switching position detector for detecting a position at which a scene included in the AV data is switched or a position at which the type of editing processing changes, and generating a detection signal according to the detection,
wherein the data allocator switches an output destination of AV data between the hardware encoder and the software encoder according to the detection signal.
(Item 12)
A video editing system comprising:
a CPU;
an editor for editing AV data including sound data and/or video data;
a hardware encoder configured by a circuit for encoding processing, the hardware encoder encoding a part of the edited AV data;
a software encoder for encoding another part of the edited AV data using the CPU in parallel with encoding processing of the hardware encoder;
a data allocator for allocating the edited AV data to the hardware encoder and the software encoder;
a synthesizer for arranging AV data respectively encoded by the hardware encoder and the software encoder in a predetermined order to synthesize the encoded AV data into a series of encoded AV data; and
an output unit for outputting the series of encoded AV data,
wherein AV data is encoded at a faster encoding processing speed than an encoding processing speed only by the hardware encoder.
(Item 13)
The video editing system according to item 12, wherein an encoding scheme of the hardware encoder is the same as an encoding scheme of the software encoder.
(Item 14)
The video editing system to item 12, wherein AV data encoded by the hardware encoder and AV data encoded by the software encoder are not predictable from each other.
(Item 15)
The video editing system according to item 12, further comprising a buffer connected between the data allocator and the hardware encoder, the buffer storing AV data allocated to the hardware encoder by the data allocator and supplying the AV data to the hardware encoder,
wherein the data allocator acquires space information of the buffer and allocates the AV data to the software encoder based on the space information.
(Item 16)
The video editing system according to item 12,
wherein data indicating a ratio of the number of data units of AV data allocated to the hardware encoder per unit time and the number of data units of AV data allocated to the software encoder per unit time is supplied to the data allocator, and
the data allocator allocates AV data to the hardware encoder and the software encoder based on the data.
(Item 17)
The video editing system according to item 12,
wherein the editor includes edit information set by a user for AV data to be edited,
the video editing system further comprises:
a CPU load estimator for predicting a load of the CPU when the editor executes editing processing based on the edit information; and
an allocation ratio setting unit for calculating a ratio of the number of data units of AV data allocated to the hardware encoder per unit time and the number of data units of AV data allocated to the software encoder per unit time according to the predicted load of the CPU, and supplying data indicating the ratio to the data allocator, and
the data allocator allocates the AV data to the hardware encoder and the software encoder based on the data.
(Item 18)
The video editing system according to item 17,
wherein the allocation ratio setting unit detects the load of the CPU and corrects the ratio according to the detected load.
(Item 19)
The video editing system according to item 12, further comprising an allocation ratio setting unit for detecting a load of the CPU, calculating a ratio of the number of data units of AV data allocated to the hardware encoder per unit time and the number of data units of AV data allocated to the software encoder per unit time according to the detected load, and supplying data indicating the ratio to the data allocator,
wherein the data allocator allocates AV data to the hardware encoder and the software encoder based on the data.
(Item 20)
The video editing system according to item 12, further comprising:
a processing speed measurer for measuring an encoding processing speed of the hardware encoder and an encoding processing speed of the software encoder; and
an allocation ratio setting unit for calculating a ratio of the number of data units of AV data allocated to the hardware encoder per unit time and the number of data units of AV data allocated to the software encoder per unit time according to the measured encoding processing speeds, and supplying data indicating the ratio to the data allocator,
wherein the data allocator allocates AV data to the hardware encoder and the software encoder based on the data.
(Item 21)
The video editing system according to item 20,
wherein the allocation ratio setting unit detects a load of the CPU and corrects the ratio according to the detected load.
(Item 22)
The video editing system according to item 12, further comprising a frame switching position detector for detecting a position at which a scene changes or a position at which the type of editing processing changes, each position being included in AV data obtained by executing editing processing in the editor based on edit information, from the edit information to generate a predetermined detection signal,
wherein the data allocator switches an output destination of AV data between the hardware encoder and the software encoder according to the detection signal.
(Item 23)
A method for encoding video data by using a hardware encoder that is configured by hardware for encoding processing and encodes a part of AV data and a software encoder that encodes another part of the AV data using a CPU, the method comprising the steps of:
receiving the AV data;
allocating the AV data to the hardware encoder and the software encoder;
encoding the allocated AV data in the hardware encoder and the software encoder;
arranging AV data respectively encoded by the hardware encoder and the software encoder in a predetermined order to synthesize the encoded AV data into a series of encoded AV data; and
outputting the series of encoded AV data,
wherein AV data is encoded at a faster encoding processing speed than an encoding processing speed only by the hardware encoder.
(Item 24)
The method according to item 23, wherein the step of allocating the AV data comprises the steps of:
storing data allocated to the hardware encoder in a buffer;
supplying AV data from the buffer to the hardware encoder;
acquiring space information of the buffer; and
allocating the AV data to the software encoder based on the space information of the buffer.
(Item 25)
The method according to item 23, wherein the step of allocating the AV data comprises the steps of:
receiving data indicating a ratio of the number of data units of AV data allocated to the hardware encoder per unit time and the number of data units of AV data allocated to the software encoder per unit time; and
allocating AV data to the hardware encoder and the software encoder based on the data.
(Item 26)
The method according to item 23, wherein, in the step of encoding, the CPU performs another processing related to the AV data other than encoding processing of the software encoder,
the method further comprises the steps of:
predicting a load of the CPU caused by the other processing so as to correspond to the AV data; and
calculating a ratio of the number of data units of AV data allocated to the hardware encoder per unit time and the number of data units of AV data allocated to the software encoder per unit time according to the predicted load of the CPU, and outputting data indicating the ratio, and
in the step of allocating the AV data, AV data is allocated to the hardware encoder and the software encoder based on the data.
(Item 27)
The method according to item 26, wherein the step of allocating the AV data further comprises the step of detecting the load of the CPU and correcting the ratio based on the detected load.
(Item 28)
The method according to item 23, further comprising the steps of:
editing the AV data based on edit information set by a user for the AV data to generate a series of AV data to be encoded, the editing being executed in parallel with the step of encoding,
predicting a load of the CPU when executing editing processing based on the edit information in the step of editing so as to correspond to the series of AV data to be encoded; and
calculating a ratio of the number of data units of AV data allocated to the hardware encoder per unit time and the number of data units of AV data allocated to the software encoder per unit time according to the predicted load of the CPU, and outputting data indicating the ratio,
wherein, in the step of allocating the AV data, AV data is allocated to the hardware encoder and the software encoder based on the data.
(Item 29)
The method according to item 23, further comprising the step of detecting a load of the CPU, calculating a ratio of the number of data units of AV data allocated to the hardware encoder per unit time and the number of data units of AV data allocated to the software encoder per unit time according to the detected load, and supplying data indicating the ratio,
wherein, in the step of allocating the AV data, AV data is allocated to the hardware encoder and the software encoder based on the data.
(Item 30)
The method according to item 23, further comprising the steps of:
measuring an encoding processing speed of the hardware encoder and an encoding processing speed of the software encoder; and
calculating a ratio of the number of data units of AV data allocated to the hardware encoder per unit time and the number of data units of AV data allocated to the software encoder per unit time according to the measured encoding processing speeds, and supplying data indicating the ratio,
wherein the step of allocating the AV data comprises the step of allocating AV data to the hardware encoder and the software encoder based on the data.
(Item 31)
The method according to item 30, wherein the step of allocating the AV data further comprises the step of: detecting the load of the CPU and correcting the ratio based on the detected load.
(Item 32)
The method according to item 23, further comprising the step of detecting a position at which a scene is switched or a position at which the type of editing processing changes, each position being included in the AV data, to generate a predetermined detection signal,
wherein the step of allocating the AV data comprises the step of switching an output destination of AV data between the hardware encoder and the software encoder according to the detection signal.
(Item 33)
The method according to item 23, further comprising the steps of:
editing the AV data based on edit information set by a user for the AV data to generate a series of AV data to be encoded; and
detecting a position at which a scene changes or a position at which the type of editing processing changes, each position being included in the series of AV data to be encoded, to generate a predetermined detection signal,
wherein the step of allocating the AV data includes switching an output destination of AV data between the hardware encoder and the software encoder according to the detection signal.
(Item 34)
A program for encoding AV data including sound data and/or video data,
the program causing an apparatus comprising:
a CPU;
a hardware encoder configured by hardware dedicated to encoding processing, the hardware encoder encoding a part of the AV data; and
a software encoder for encoding another part of the AV data using the CPU
to execute the steps of:
receiving the AV data;
allocating the AV data to the hardware encoder and the software encoder;
making encoding processing on the AV data performed in parallel by the hardware encoder and the software encoder;
arranging AV data respectively encoded by the hardware encoder and the software encoder in a predetermined order to synthesize the encoded AV data into one encoded AV data; and
outputting the encoded AV data,
wherein AV data is encoded at a faster encoding processing speed than an encoding processing speed only by the hardware encoder.

An encoding apparatus capable of raising the encoding processing speed of video data and/or sound data, an encoding method thereof, and a video editing system can be provided.

10 CPU
11, 11A to 11G encoding unit
13 editor
14 data allocator
151 hardware encoder
152 software encoder
16 synthesizer
17 output unit
22A to 22F allocation ratio setting unit
23 CPU load estimator
25 CPU load detector
28 processing speed measurer
100 encoding apparatus
200 video editing system

Claims

An encoding apparatus for encoding AV data including sound data and/or video data, comprising:
a CPU;
a hardware encoder configured by hardware for encoding processing, the hardware encoder encoding a part of the AV data;
a software encoder for encoding another part of the AV data using the CPU in parallel with encoding processing of the hardware encoder;
a data allocator for allocating the AV data to the hardware encoder and the software encoder;
a synthesizer for arranging AV data respectively encoded by the hardware encoder and the software encoder in a predetermined order to synthesize the encoded AV data into a series of encoded AV data; and
an output unit for outputting the series of encoded AV data,
wherein AV data is encoded at a faster encoding processing speed than an encoding processing speed only by the hardware encoder.
The encoding apparatus according to claim 1, further comprising a buffer connected between the data allocator and the hardware encoder, the buffer storing AV data allocated to the hardware encoder by the data allocator and supplying the AV data to the hardware encoder,
wherein the data allocator acquires space information of the buffer and allocates the AV data to the software encoder based on the space information.
The encoding apparatus according to claim 1,
wherein data indicating a ratio of the number of data units of AV data allocated to the hardware encoder per unit time and the number of data units of AV data allocated to the software encoder per unit time is supplied to the data allocator, and
the data allocator allocates AV data to the hardware encoder and the software encoder based on the data.
The encoding apparatus according to claim 1,
wherein the CPU performs another processing on the AV data other than encoding processing,
the encoding apparatus further comprises:
a CPU load estimator for predicting a load of the CPU caused by the other processing so as to correspond to the AV data and;
an allocation ratio setting unit for calculating a ratio of the number of data units of AV data allocated to the hardware encoder per unit time and the number of data units of AV data allocated to the software encoder per unit time according to the predicted load of the CPU, and supplying data indicating the ratio to the data allocator, and
the data allocator allocates AV data to the hardware encoder and the software encoder based on the data.
The encoding apparatus according to claim 4,
wherein the allocation ratio setting unit detects the load of the CPU and corrects the ratio based on the detected load.
The encoding apparatus according to claim 1, further comprising an allocation ratio setting unit for detecting a load of the CPU, calculating a ratio of the number of data units of AV data allocated to the hardware encoder per unit time and the number of data units of AV data allocated to the software encoder per unit time according to the detected load, and supplying data indicating the ratio to the data allocator,
wherein the data allocator allocates AV data to the hardware encoder and the software encoder based on the data.
The encoding apparatus according to claim 1, further comprising:
a processing speed measurer for measuring an encoding processing speed of the hardware encoder and an encoding processing speed of the software encoder; and
an allocation ratio setting unit for calculating a ratio of the number of data units of AV data allocated to the hardware encoder per unit time and the number of data units of AV data allocated to the software encoder per unit time according to the measured encoding processing speeds, and supplying data indicating the ratio to the data allocator,
wherein the data allocator allocates AV data to the hardware encoder and the software encoder based on the data.
The encoding apparatus according to claim 7,
wherein the allocation ratio setting unit detects a load of the CPU and corrects the ratio according to the detected load.
The encoding apparatus according to claim 1, further comprising a switching position detector for detecting a position at which a scene included in the AV data is switched or a position at which the type of editing processing changes, and generating a detection signal according to the detection,
wherein the data allocator switches an output destination of AV data between the hardware encoder and the software encoder according to the detection signal.
A video editing system comprising:
a CPU;
an editor for editing AV data including sound data and/or video data;
a hardware encoder configured by a circuit for encoding processing, the hardware encoder encoding a part of the edited AV data;
a software encoder for encoding another part of the edited AV data using the CPU in parallel with encoding processing of the hardware encoder;
a data allocator for allocating the edited AV data to the hardware encoder and the software encoder;
a synthesizer for arranging AV data respectively encoded by the hardware encoder and the software encoder in a predetermined order to synthesize the encoded AV data into a series of encoded AV data; and
an output unit for outputting the series of encoded AV data,
wherein AV data is encoded at a faster encoding processing speed than an encoding processing speed only by the hardware encoder.
The video editing system according to claim 10,
wherein the editor includes edit information set by a user for AV data to be edited,
the video editing system further comprises:
a CPU load estimator for predicting a load of the CPU when the editor executes editing processing based on the edit information; and
an allocation ratio setting unit for calculating a ratio of the number of data units of AV data allocated to the hardware encoder per unit time and the number of data units of AV data allocated to the software encoder per unit time according to the predicted load of the CPU, and supplying data indicating the ratio to the data allocator, and
the data allocator allocates the AV data to the hardware encoder and the software encoder based on the data.
The video editing system according to claim 10,
wherein the allocation ratio setting unit detects the load of the CPU and corrects the ratio according to the detected load.
The video editing system according to claim 10, further comprising a frame switching position detector for detecting a position at which a scene changes or a position at which the type of editing processing changes, each position being included in AV data obtained by executing editing processing in the editor based on edit information, from the edit information to generate a predetermined detection signal,
wherein the data allocator switches an output destination of AV data between the hardware encoder and the software encoder according to the detection signal.
A method for encoding video data by using a hardware encoder that is configured by hardware for encoding processing and encodes a part of AV data and a software encoder that encodes another part of the AV data using a CPU, the method comprising the steps of:
receiving the AV data;
allocating the AV data to the hardware encoder and the software encoder;
encoding the allocated AV data in the hardware encoder and the software encoder;
arranging AV data respectively encoded by the hardware encoder and the software encoder in a predetermined order to synthesize the encoded AV data into a series of encoded AV data; and
outputting the series of encoded AV data,
wherein AV data is encoded at a faster encoding processing speed than an encoding processing speed only by the hardware encoder.
The method according to claim 14, further comprising the steps of:
editing the AV data based on edit information set by a user for the AV data to generate a series of AV data to be encoded, the editing being executed in parallel with the step of encoding,
predicting a load of the CPU when executing editing processing based on the edit information in the step of editing so as to correspond to the series of AV data to be encoded; and
calculating a ratio of the number of data units of AV data allocated to the hardware encoder per unit time and the number of data units of AV data allocated to the software encoder per unit time according to the predicted load of the CPU, and outputting data indicating the ratio,
wherein, in the step of allocating the AV data, AV data is allocated to the hardware encoder and the software encoder based on the data.
The method according to claim 14, further comprising the steps of:
editing the AV data based on edit information set by a user for the AV data to generate a series of AV data to be encoded; and
detecting a position at which a scene changes or a position at which the type of editing processing changes, each position being included in the series of AV data to be encoded, to generate a predetermined detection signal,
wherein the step of allocating the AV data includes switching an output destination of AV data between the hardware encoder and the software encoder according to the detection signal.