WO2002089486A2 - Method and system for video compression and distribution - Google Patents

Method and system for video compression and distribution Download PDF

Info

Publication number
WO2002089486A2
WO2002089486A2 PCT/CA2002/000641 CA0200641W WO02089486A2 WO 2002089486 A2 WO2002089486 A2 WO 2002089486A2 CA 0200641 W CA0200641 W CA 0200641W WO 02089486 A2 WO02089486 A2 WO 02089486A2
Authority
WO
WIPO (PCT)
Prior art keywords
video
frame
block
data
motion estimation
Prior art date
Application number
PCT/CA2002/000641
Other languages
French (fr)
Other versions
WO2002089486A3 (en
Inventor
Steve Vestergaard
Che-Wai Tsue (William)
Original Assignee
Destiny Software Productions Inc.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Destiny Software Productions Inc. filed Critical Destiny Software Productions Inc.
Priority to AU2002302214A priority Critical patent/AU2002302214A1/en
Publication of WO2002089486A2 publication Critical patent/WO2002089486A2/en
Publication of WO2002089486A3 publication Critical patent/WO2002089486A3/en

Links

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/80Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
    • H04N21/81Monomedia components thereof
    • H04N21/8166Monomedia components thereof involving executable data, e.g. software
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/102Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
    • H04N19/103Selection of coding mode or of prediction mode
    • H04N19/107Selection of coding mode or of prediction mode between spatial and temporal predictive coding, e.g. picture refresh
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/102Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
    • H04N19/124Quantisation
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/102Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
    • H04N19/132Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/134Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
    • H04N19/146Data rate or code amount at the encoder output
    • H04N19/149Data rate or code amount at the encoder output by estimating the code amount by means of a model, e.g. mathematical model or statistical model
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/134Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
    • H04N19/146Data rate or code amount at the encoder output
    • H04N19/15Data rate or code amount at the encoder output by monitoring actual compressed data size at the memory before deciding storage at the transmission buffer
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/134Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
    • H04N19/154Measured or subjectively estimated visual quality after decoding, e.g. measurement of distortion
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/169Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
    • H04N19/17Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object
    • H04N19/172Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object the region being a picture, frame or field
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/169Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
    • H04N19/17Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object
    • H04N19/176Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object the region being a block, e.g. a macroblock
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/50Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
    • H04N19/503Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal prediction
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/50Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
    • H04N19/503Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal prediction
    • H04N19/51Motion estimation or motion compensation
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/60Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding
    • H04N19/61Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding in combination with predictive coding

Definitions

  • the present invention relates generally to communications, and more specifically, to a method and system for compressing high quality video content for transfer over communication networks, and subsequently decompressing it on a computing device with minimal processing power.
  • Hard-wired telephone systems have evolved to include wireless telephone and pager networks based on satellite, cellular, wireless local loop, line of sight and other wireless technologies.
  • Data communication networks such as the Internet, Wide Area Networks (WANs) and Local Area Networks (LANs) have also become widespread, and support many different devices including personal and laptop computers, personal digital assistants (PDAs) and television set top boxes.
  • WANs Wide Area Networks
  • LANs Local Area Networks
  • PDAs personal digital assistants
  • These telephony and data networks are generally converging to a model in which data are transferred in multiple packets of digital symbols. These data packets may travel independently of one another both in terms of the routings they take, and the time they take to travel from the originator to the destination. This model has proven to be very successful and is the basis, for example, of the protocols used in the Internet.
  • An area of particular interest is the communication of video content between devices and over networks. Communicating high quality video content requires a great deal of bandwidth over the network that is transferring the content.
  • Full-motion video is essentially a sequence of still images that are replaced so fast that a human perceives them to be in continuous and smooth motion - 60 images per second being considered high quality and 20 images per second generally being accepted as the lower limit of continuous-motion.
  • a relatively poor quality image requires a great deal of bandwidth to view; for example a video sequence that has dimensions of 200 x 200 pixels, with 24 bits of colour resolution, at a rate of 10 frames per second, will require 9.6 megabits per second (MBPS) of data.
  • MBPS megabits per second
  • the most commonly used video compression standard is the MPEG - 1 standard (MPEG for Moving Pictures Experts Group), though there are other formats including the AVI, Real, and Quicktime formats. Other less common and proprietary formats are also known.
  • MPEG - 1 standard MPEG for Moving Pictures Experts Group
  • Other less common and proprietary formats are also known.
  • the MPEG formats are the most commonly used video compression formats, but they are very resource intensive.
  • the original objective of MPEG-1 was to deliver video and video at a data rate of 1.86 megabits per second (MBPS).
  • More advanced video compression techniques such as MPEG-4 and MPEG-7 offer better compression ratios but require much more computation in compression and/or decompression to provide higher quality (MPEG-4 was designed to work in low to high bitrate range).
  • MPEG videos which require very little bandwidth could be generated simply by limiting the dimensions of the display, or the resolution of the video content being supplied.
  • the end user will still require a fully function MPEG player on his personal computer to view the content.
  • the MPEG player must have the capacity to process the very high bandwidth signals that are part of the MPEG standards.
  • the second main problem with the current video compression and decompression software is that there is no optimisation of the data stream for the resources available.
  • many compression/decompression systems will allow the user to set certain parameters, such as the maximum bandwidth required to transmit the resulting video file, they do not take into account the original data being compressed and dynamically adjust the compression techniques being use.
  • One aspect of the invention is broadly defined as a method of compressing video comprising the steps of: a method of compressing video comprising the steps of: digitally sampling an video signal; setting the value of a quality parameter (Q) and the value of a targeted bit rate (TBR); calculating a predicted bit rate (PBR) for a video frame of the video signal; responding to the PBR being greater than the TBR by reducing the value of Q for the video frame; and compressing the video frame in accordance with the value of the Q parameter.
  • a method of compressing video comprising the steps of: digitally sampling an video signal; setting the value of a quality parameter (Q) and the value of a targeted bit rate (TBR); calculating a predicted bit rate (PBR) for a video frame of the video signal; responding to the PBR being greater than the TBR by reducing the value of Q for the video frame; and compressing the video frame in accordance with the value of the Q parameter.
  • Q quality parameter
  • TBR targeted bit rate
  • Figure 1 presents a flow chart of a method of compression in a broad embodiment of the invention
  • Figure 2 presents a block diagram of a system for compression, transfer and decompression of video files in a preferred embodiment of the invention
  • Figure 3 presents a flow chart of an overall method of handling video files and generating executable Java applets in a preferred embodiment of the invention
  • Figures 4A, 4B and 4C present a flow chart of a method of video file compression in a preferred embodiment of the invention
  • Figure 5 presents a flow chart of a method of video file decompression in a preferred embodiment of the invention.
  • FIG. 1 A methodology which addresses the objects outlined above, is presented as a flow chart in Figure 1.
  • This figure presents a method of compressing video data in which the compression technique and quality parameters are modified dynamically as compression proceeds, so that the quality is optimised in view of a fixed bandwidth that the content is to traverse.
  • This compression method is generally effected as follows: 1. first, by digitally sampling a video signal in some manner, at step 20; then 2. setting the value of a quality parameter (Q) and the value of a targeted bit rate (TBR) at step 22;
  • the video content being compressed can take on any form, analogue or digital. In most computer-based applications, the video content will already be in a digital form, either the product of a digital camera or scanner, or the output of a piece of digital imaging software. Thus, the step of digitally sampling the video signal at step 20 may already have been performed by another component in the system, rather than the core software.
  • the invention is not restricted by the value of the targeted bit rate (TBR), or the manner in which it is established.
  • TBR targeted bit rate
  • the software operator will generate several executable video packages which are optimized for different downloading rates.
  • these executable video packages would be optimised for the most common connection systems, such as 56.6 kbps (kilobit per second) dial-up modems, and various DSL (digital subscriber line) standards.
  • the end user can select an executable video package that his Internet connection can carry in real time. (Note that the TBR is not the actual connection speed, but the bandwidth that a video package will not be expected to exceed).
  • the step of adjusting the Q value to maximize the bit rate can also be effected in a number of manners.
  • digital video is generally implemented as a series of still images or frames.
  • the Q value could be determined by trail and error, extrapolated from two or more test points, determined in a feed-forward manner or in a similar manner which would be clear to one skilled in the art from the teachings herein.
  • the method of the invention keeps the quality of the compressed video as high as possible for the bandwidth that is available.
  • the compressed file or files will use as much of the available bandwidth as possible, without exceeding its capacity.
  • Figure 2 identifies the relevant parties in a transaction of the preferred embodiment of the invention, while the specific processing steps are presented in detail in Figures 3 through 5.
  • the invention is applied to an Internet and Web site environment.
  • the owner of a Web site (the "software operator") can purchase the software of the invention, use it to compress video clips, and post those video clips on his Web site.
  • These compressed video clips are packaged as executable Java applets, and are represented by an icon on the software operator's Web site.
  • the video file is streamed to the end user's computer or similar device, and immediately begins to play.
  • a compressed video clip actually consists of two software components: an executable Java applet, and a data file containing the compressed video content.
  • an executable Java applet can be downloaded and begin executing, while the data can be "streamed" separately. This allows the video content to be displayed as soon as sufficient data has been received. If the two files were combined into one, then streaming could not be done - one would have to wait for the entire file to download before it could be executed. This process is described in greater detail hereinafter.
  • the Java applet and compressed video data files are frequently referred to herein as a single file. This has been done in the interest of simplicity only. It is clearly preferable to implement the invention using separate files to take advantage of the streaming process.
  • FIG. 2 presents an exemplary layout of an Internet communications system 30 in a preferred embodiment of the invention.
  • the Internet 32 is described as a system of routers interconnected by an Internet backbone network, which allows two parties to communicate via whatever entities happen to be interconnected at any particular time.
  • the Internet 32 is far more complex, consisting of a vast interconnection of computers, servers, routers, computer networks and public telecommunication networks.
  • End users 34 may access the Internet 32 in a number of manners including modulating and demodulating data signals over a telephone line using audio frequencies, which requires a modem and connection to the Public Switched
  • Telephone Network which in turn connects to the Internet 32 via an Internet Service Provider 36.
  • Another manner of connection is the use of set top boxes or cable modems which modulate and demodulate data onto high frequencies which pass over existing telephone or television cable networks and are connected directly to the Internet 32 via Hi-Speed Internet Service Providers. Generally, these high frequency signals are transmitted outside the frequencies of existing services passing over these telephone or television cable networks.
  • An end user 34 may also obtain access to the Internet 32, using a digital cellular telephone, pager, or personal digital assistant.
  • Internet Service Providers (ISPs) 36 or Internet Access Providers (lAPs) are companies that provide access to the Internet.
  • ISPs 36 are considered by some to be distinguished from lAPs in that they also provide content and services to their subscribers, but in the context of this disclosure the distinction is irrelevant.
  • ISPs 36 For a monthly fee, ISPs 36 generally provider end users 34 with the necessary software, user name, password and physical access. Equipped with a telephone line modem, cable modem or set top box, one can then log on to the Internet 32 and browse the World Wide Web, and send and receive e-mail.
  • Web servers 38, 40 are computers which provide text, graphic or multimedia content, or software applications, to other parties over the Internet 32.
  • the software operator 42 will obtain software from the compressor/decompressor software server 38, and the operator's Web site 40 will provide executable, compressed video files and other content to the end users 34.
  • Such networks could include: wireless networks such as cellular telephone networks, the public switched telephone network, cable television networks, the Internet, ATM networks, frame relay networks, local area networks (LANs) and wide area networks (WANs).
  • wireless networks such as cellular telephone networks, the public switched telephone network, cable television networks, the Internet, ATM networks, frame relay networks, local area networks (LANs) and wide area networks (WANs).
  • LANs local area networks
  • WANs wide area networks
  • the owner of a Web site simply obtains a copy of the encoding software electronically, and uses it to generate executable video applets. These video applets can then be included with any Web page, linked to it using a graphic icon. End users 34 who visit these Web pages download and execute the video applets by clicking on these icons.
  • asymmetric compression and decompression is used, that is, only simple computations need to be performed during decompression on the end user's device, while more complex processing may be performed during compression.
  • the power and speed of the processing during compression is not limiting because it need not be performed in real time, unlike the decompression.
  • Figure 3 focuses on the handling of the original video data, and how the executable Java applet is generated, while Figure 4 presents the details of the compression routine itself.
  • the routine begins at step 130 of Figure 3, where, in response to the compression software being launched, the algorithm seeks out the video data file that is to be compressed.
  • the targeted file or files may be communicated to the compression software using any technique known in the art, including dragging files to an icon or identifying the file or files in a command line.
  • the software operator will now be queried to provide the available data rate that the compressed video is intended for, and the quality level that is desired, at step 132. It may also be desirable to query the software operator for other data, such as the name of the file, where it is to be stored, who generated it and other such details.
  • the software operator will generally choose a targeted bit rate that corresponds to commonly available bit rates that prospective end users may have. For example, the most common dial up modems have bit rates of 56.6 kbps (kilo bits per second), while common data rates for DSL (digital subscriber line) connections include 128, 256, 300 and 500 kbps. The software operator would therefore enter one of these values and identify the file as such, so that this targeted bit rate can be posted on the Web site along with the compressed file.
  • the quality level is adjusted automatically by the program, so the software operator can set a high value so that the algorithm attempts to maximize the quality at all times.
  • the default setting will also be to the highest level for the TBR.
  • step 134 the algorithm will consider each frame of the targeted file successively at step 134, and perform the compression routine of Figure 4 on that frame, at step 136.
  • all digital video data can ultimately be described as a series of individual frames which are successively displayed to the end user. While the invention will typically be applied to frames of digital RGB data
  • Red Green Blue data is a standard format for digital video data
  • MPEG format it may also be applied to other input formats.
  • the handling of the RGB MPEG data will be described in greater detail with respect to Figure 4.
  • the routine will have generated a Java-based executable, compressed video file which may be stored on the Web server or other storage device at step 140.
  • the Java-based executable, compressed video file may now be accessed from the Internet simply by placing an icon on a Web page, which is linked to stored Java code on the server (at step 142).
  • each frame of the input video data is considered individually.
  • each frame is prepared for the compression routine at step 150 by converting it from the input format (say, for example, from RGB format), to YUV format.
  • RGB format is a video format that is based on the hues red, green and blue. While this format may seem logical because it correlates intuitively with the images being displayed, it is not the most data efficient manner in which to define an image.
  • YUV is a colour encoding scheme for images in which luminance and chrominance are separate. The human eye is less sensitive to colour variations than to intensity variations. YUV is therefore advantageous because it allows the luminance (parameter Y) information to be handled at full bandwidth while the chrominance (parameters U and V) information, to which the human eye is less sensitive, can be compressed.
  • the YUV format is well known in the art of video imaging technology.
  • RGB format While there is no scientifically fixed relationship between the RGB format and the YUV format, the following is a standard relationship as used in the art:
  • blurred YUV For high contrast frames, "blurred YUV" data is also generated, where adjacent pixels are averaged together.
  • the blurred YUV frames are used in motion estimation only, and not for the final encoding of the pixels.
  • the algorithm calculates a predicted bit rate (PBR) for the frame being considered, by averaging the bit rate of the previous three seconds of data, at step 152.
  • PBR predicted bit rate
  • Other time periods could also be used to calculate this moving average, as could weighted averaging of the data, or similar techniques.
  • PBR is then compared to the TBR at step 154, and three cases are established:
  • Each 8 x 8 block in the prepared new frame is now compared with the corresponding block in a history frame at step 166, and a "motion estimate" is calculated (see step 194 for a discussion on how the history frame is generated).
  • a "motion estimate" is calculated (see step 194 for a discussion on how the history frame is generated).
  • the concept of motion estimation is well know in the art, and is used in various standards including the MPEG standard. The invention adds several improvements to the traditional process though.
  • Each frame is divided into 8 x 8 blocks of pixels.
  • the routine searches the surrounding pixels in the history frame for the closest match. When the closest match is found, an offset vector (x, y), is calculated, and an error level is assigned at step 168.
  • the error level for the block match is a relative measure comparing the closest matching block to the corresponding block in the history frame. It is only used to perform the decision in step 170, so clearly a number of different models could be used.
  • the invention employs a quick hierarchical search algorithm to narrow down the search range in 5 rounds.
  • the block is compared with the surrounding eight blocks with offsets (-31 , -31 ), (0, -31 ), (31 , 31 ), (-31 , 0), (0, 0), (31 , 0), (-31 , 31 ), (0, 31 ) and (31 , 31 ). Then the one with the smallest error is chosen to be the base for the next round.
  • an error level threshold will be set to classify each block as a moved block, a new block or a no-change block at step 170. If the error level of the block is higher than the threshold, the block is considered to be new, otherwise, it is classified as moved.
  • a no-change block is a special case of the moved block with offset (0,0).
  • the current block will be considered a moved block, and the vector for the block will be set to the offset (x, y) determined by the motion estimation analysis, at step 172;
  • step 164 detects that this has been completed, the frame will now be defined by a set of motion estimation vectors. These motion estimation data are compiled into a motion estimation table which is considered at step 178 of Figure 4C.
  • the routine determines whether it would be more efficient to record all of the block changes in the frame, or whether there are so many that it would be more efficient to simply save the entire frame as a JPEG (Joint Photographic Experts Group) image.
  • JPEG Joint Photographic Experts Group
  • the JPEG standard is a high-quality compression standard for still pictures, and is well known in the art.
  • the decision point was taken to be 70%; that is, if 70% or more of the blocks in the frame are new or moved, then the routine branches to step 180 where the entire frame is saved as a JPEG image.
  • the size of this JPEG image is then compared with the TBR at step 182, and if the JPEG image is small enough, control passes to step 194. If the image is too large, then remedial measures are taken at step 184.
  • a number of things can be done to reduce the size of the full-frame JPEG image, including the following: 1. full frames can be scaled down, for example, to half-frame resolution as indicated at step 184 in Figure 4C; or 2. full frames can be compressed with lower quality.
  • JPEG generating software has a variable quality setting, which can be linked to the Q value used by the routine of the invention.
  • the quality can be increased if the bitrate is lower than the TBR, or decreased if the bitrate is higher than the TBR.
  • step 178 If the number of new and moved blocks is determined to be small at step 178, then control passes to step 186 where the motion estimation table is compressed using "Huffman coding". Huffman coding is "lossless" in that no detail or information is lost in compressing and decompressing.
  • Huffman coding is performed by categorizing data points by the likelihood that they will occur. Then, the most common points are assigned shorter codes, and the less likely codes are assigned longer codes.
  • the motion estimation is recorded as an offset vector (x, y). All (x, y) pairs are sorted and counted, and popular pairs will be assigned a code in the table. Unpopular pairs will be assigned to one single code and will store the offset (x, y) uncompressed. Note again that a vector of (0, 0) means there as no change from the history frame to the new frame, and that an out of range pair is used for new blocks.
  • the invention uses predetermined Huffman coding tables within the compressor and decompressor.
  • Adaptive Huffman coding or dynamic Huffman tables could be used, but generally, adaptive Huffman would consume too many CPU cycles to decompress, and dynamic Huffman tables would increase the amount of data to download.
  • a series of JPEG images are then generated at step 188, corresponding to the blocks that are identified in the motion estimation table.
  • step 190 it is then determined whether the data for the current frame is less than the TBR. If the data for the frame still exceeds the TBR, in spite of dropping the quality level as described above, the frame is simply dropped at step 192. Otherwise, the image data is copied to the compressed video data file at step 194.
  • the compressed video data and the executable Java decompression code are stored separately.
  • the executable Java decompression code is stored in a zip file containing the Java Applet byte code, and the compressed video data is stored in a separate file which will be streamed down to the client machine during playback.
  • a new history frame is then constructed at step 196, simply by saving the current image in a buffer, so it can be compared with the next image when the routine is repeated.
  • the history frame will be null (i.e. the buffer will be empty), so when the first comparison is made to history frame at step 166, there will be no matches.
  • Compressed audio data can also be added to the compressed video data file at this point.
  • the audio and video are interleaved so that synchronization is implicit.
  • the data blocks could be stored as follows: V A A A V A A A V A A A V ("V" characters representing video data, and "A" characters representing audio data).
  • the interleaved blocks will be stored in this manner: V AAA V AAAA V AAA V AAA V AAA V AAA V AAA V AAA V AAAA ....
  • some of the A groups will have three blocks and some will have four. Note that even if there is an empty video frame (dropped or no change), there may still be an audio block associated with it, which must be stored.
  • null video blocks are inserted which have a size of 0 for the data portion. The decompression software recognizes these 0 blocks as null video blocks, thus ensuring synchronization.
  • the compression technique of the invention may be resource intensive itself, but it results in compressed video which may be decompressed using simple operations. Thus, decompression can be performed on the end user's computer or other device. These simple operations can be executed quickly enough, that an executable file can be created which can run in a Java environment.
  • Java is a code interpretor which is almost universally supported by Web browsers for computers and similar devices. More important, it is platform independent, so that Java code and applets may be executed on any platform.
  • JPEG imaging standard used in the invention does use some complex processing, but JPEG is supported by Java natively as one of its standard image formats. Thus, the code to produce the JPEG images is in machine code, and is inherent to Java.
  • the decompression routine of the invention can be applied in a Java environment, this allows Web sites, for example, to provide executable applets which can be downloaded and played by an end user with a Java-enabled browser or operating system. These executable applets can also be delivered to end users in other ways such as via Email or Banner Ads.
  • the end user does not have to obtain application software or browser plugins to listen to the content.
  • the end user does not have to address issues of compatibility with his platform and the format of the video content being downloaded, inconvenience of obtaining upgrades, and possibly requiring many different software packages to address different video formats.
  • decompression software is packaged with the video content so the end user simply downloads an executable file.
  • the preferred embodiment of the invention would generally be implemented: as an icon on a Web page, which is executed when an end user 34 clicks upon it, or as an applet which executes when anyone visits the Web page. In either case, the method presented in the flow chart of Figure 6 would be performed. Execution of this method generally requires that the end user 34 have the following:
  • Java compatible browser typically consisting of a video display terminal and a video card. It is also necessary to have a Java applet residing on an Internet Web server for the end user 34 to download.
  • the routine begins at step 198 of Figure 5, after the end user 34 has initiated the downloading or execution of an applet in one of the manners noted above.
  • the zipped decompression applet is then transmitted to the end user 34 and is unzipped for execution.
  • An auto detection function is performed by the Java applet, as known in the art, to determined the available bandwidth capability of the end user 34.
  • the applet uses this information to determine which of the previously stored compressed video data files should be downloaded, providing the best quality for the bandwidth available.
  • the compressed video data is then "streamed" to the end user, using the undocumented but commonly used streaming facility in Java.
  • “Streaming” is the process of beginning to play the content before all of the content has been downloaded.
  • content can be decompressed and played as soon as the zipped decompression applet and the first sub-block of the compressed video data have been received; meanwhile, the balance of the compressed video content can be downloaded and decompressed in the background.
  • the decompression software simply buffers the compressed video content as it is received and decompressed. As the video card on the end user's computer requires data, it simply calls the buffer for data.
  • Decoding of the compressed video file is performed on a frame by frame basis, in a manner that is complementary to the compression routine described above.
  • Each frame of the compressed video file is considered separately at step 200.
  • the motion estimate data for the frame is decoded using a predetermined Huffman table that was downloaded with the executable Java applet, at step 202.
  • the JPEG image or images associated with the frame are then decompressed at step 204, using a standard JPEG engine as known in the art. If the frame was stored as a full-frame JPEG image, then there will only be one image to decompress and copy to the new frame. If the frame was stored as an assembly of moved blocks, then these blocks are copied from the history frame to the new frame at step 206, being moved in accordance with their decoded motion vectors.
  • Blocks which are unchanged from the history frame are then copied from the history frame at step 208, to the new frame. If the JPEG image had been scaled down at step 184, it is now returned to full scale at step 210, and is copied to the new frame.
  • the completed image is now ready for display, and is transferred to the video card for display on the end user's video interface at step 212.
  • This new frame is then buffered as the new history frame at step 214, so that it can be used as the basis for the next frame (if required). Control then returns to step 200, so that the next frame can be considered.
  • the method of the invention provides a number of marketable advantages including the following: 1. compressed, high quality video can be streamed over the existing Internet using a standard 56.6 modem and telephone line, or higher speed connections; 2. asymmetric compression provides sufficiently fast decompression, making a platform independent, Java implementation possible; 3. no plugins or players are required on the end user's device, so there are no difficulties with system compatibility, having to obtain multiple media players, or having to purchase or upgrade software;
  • the decompression software can be run on any Java-enabled Web browser; 5. a regular Web server can be used - special servers are not required;
  • the method of the invention allows various sizes of compressed files to be created, so that the end user 34 can obtain an executable file that is optimal for the bandwidth of his network connection.
  • Most compression systems are static, and the end user 34 has no choice of which video file to download;
  • the efficient compression of the preferred embodiment provides for an extremely small download;
  • the software operator 42 does not have to perform any complex programming to compress his files or to generate executable applets which may be loaded onto his Web site. All programming code is generated automatically.
  • the invention is not limited by the nature of the Web page being transmitted.
  • the invention could be used to insert simple banners into Web pages, or more sophisticated multimedia advertisements.
  • these advertisements could be sent along with real audio, real video, telephone over Internet, video conferencing over Internet, or other data and software applications.
  • the invention offers a compression technique that is particularly effective in view of the current trade-off between available resources and video quality. It is expected that the bandwidth and speed of existing communication networks, and the available processing power on various computing platforms will continue to improve, thus the tradeoff curve will slowly shift.
  • This will allow the method of the invention to be implemented with more resource intensive functions.
  • the preferred embodiment of the invention avoids certain resource- intensive techniques such as sub-pixel motion estimation and delta/error encoding.
  • resource- intensive techniques such as sub-pixel motion estimation and delta/error encoding.
  • the method steps of the invention need not be implemented as Java code, but may be embodiment in sets of executable machine code stored in a variety of formats such as object code or source code.
  • the executable machine code may be integrated with the code of other programs, implemented as subroutines, by external program calls or by other techniques as known in the art.
  • the embodiments of the invention may be executed by a computer processor or similar device programmed in the manner of method steps, or may be executed by an electronic system which is provided with means for executing these steps.
  • an electronic memory medium such as computer diskettes, CD-Roms, Random Access Memory (RAM), Read Only Memory (ROM) or similar computer software storage media known in the art, may be programmed to execute such method steps.
  • electronic signals representing these method steps may also be transmitted via a communication network.
  • the invention could be applied to all manner of appliances having computer or processor control and communication capability, including computers, smart terminals, lap top computers, personal digital assistants, cellular telephones, Bluetooth devices, Internet-ready telephones, televisions, television set top units, and automobiles.
  • computers smart terminals, lap top computers, personal digital assistants, cellular telephones, Bluetooth devices, Internet-ready telephones, televisions, television set top units, and automobiles.
  • Such implementations would be clear to one skilled in the art, and do not take away from the invention.

Abstract

Use of the Internet is widespread, particularly using dial-up modems. However, there is currently no method of compressing high quality video content that optimises the quality for a given, available bandwidth. As well, there is no method for compressing video content for that can be decompressed on a computing device with minimal processing power. Typical video compression/decompression systems require a software application or browser plugin on the end user's computer because their decompression requires a great deal of processing. The invention provides a method and system in which video data can be decompressed in real time, in a Java environment, so executable video applets can be posted on Web sites, and be universally accessible. This end users do not have to download, configure and upgrade multiple software applications or plugins.

Description

Method and System for Video Compression and Distribution
The present invention relates generally to communications, and more specifically, to a method and system for compressing high quality video content for transfer over communication networks, and subsequently decompressing it on a computing device with minimal processing power.
Background of the Invention
Over the last two decades, tremendous advances have been made in the availability and capability of communication networks and devices. Hard-wired telephone systems have evolved to include wireless telephone and pager networks based on satellite, cellular, wireless local loop, line of sight and other wireless technologies. Data communication networks such as the Internet, Wide Area Networks (WANs) and Local Area Networks (LANs) have also become widespread, and support many different devices including personal and laptop computers, personal digital assistants (PDAs) and television set top boxes. There are also devices and networks which operate in both telephony and data environments, combining technologies of various types.
These telephony and data networks are generally converging to a model in which data are transferred in multiple packets of digital symbols. These data packets may travel independently of one another both in terms of the routings they take, and the time they take to travel from the originator to the destination. This model has proven to be very successful and is the basis, for example, of the protocols used in the Internet. An area of particular interest is the communication of video content between devices and over networks. Communicating high quality video content requires a great deal of bandwidth over the network that is transferring the content.
Full-motion video is essentially a sequence of still images that are replaced so fast that a human perceives them to be in continuous and smooth motion - 60 images per second being considered high quality and 20 images per second generally being accepted as the lower limit of continuous-motion. However, even a relatively poor quality image requires a great deal of bandwidth to view; for example a video sequence that has dimensions of 200 x 200 pixels, with 24 bits of colour resolution, at a rate of 10 frames per second, will require 9.6 megabits per second (MBPS) of data. Thus, it is clearly desirable to compress video data which is to be stored or transmitted.
Techniques are known for compressing video content to minimize the demands on the network resources, but these techniques have two fundamental problems:
1. they place large computational demands on the devices performing the decompression, so they must be implemented by dedicated media players or browser plugins on the computer playing the content. This causes logistical problems in the distribution of media files; and 2. they apply a certain set of compression techniques to the input content without regard for the bandwidth of the channel that is to carry the content.
The most commonly used video compression standard is the MPEG - 1 standard (MPEG for Moving Pictures Experts Group), though there are other formats including the AVI, Real, and Quicktime formats. Other less common and proprietary formats are also known. These and similar techniques generally use a software player which exists as an independent software application on the receiver's personal computer, or as a plugin to an Internet browser. Either way, the decompression software must be compatible with the end user's operating system and software platform, and the format of the video file being downloaded. Thus, the end user must obtain the correct software and configure it appropriately for his computer, making periodic upgrades as required. These downloading and configuration tasks can be slow and frustrating, which presents a barrier to access even if the software is available to the end user at no cost.
As there is no single format which has emerged as an industry standard, an end user must perform this exercise for multiple plugins and software applications.
Clearly, it is impractical to expect end users to maintain a large number of such formats on their computer. Therefore, a Web site that includes such video content could not be universally accessible.
The MPEG formats are the most commonly used video compression formats, but they are very resource intensive. The original objective of MPEG-1 , for example, was to deliver video and video at a data rate of 1.86 megabits per second (MBPS). More advanced video compression techniques such as MPEG-4 and MPEG-7 offer better compression ratios but require much more computation in compression and/or decompression to provide higher quality (MPEG-4 was designed to work in low to high bitrate range). MPEG videos which require very little bandwidth could be generated simply by limiting the dimensions of the display, or the resolution of the video content being supplied. However, the end user will still require a fully function MPEG player on his personal computer to view the content. Thus, the MPEG player must have the capacity to process the very high bandwidth signals that are part of the MPEG standards.
These very high bandwidth signals require a great deal of complex processing on the end user's computer. In order to decompress MPEG files quickly enough to appear in real time, executable machine code must therefore be used to view MPEG files. This explains why MPEG players are invariably provided as dedicated software applications on the end user's computer or other device: this is the only way they can process the data quickly enough to provide real-time video decompression.
As noted above, the second main problem with the current video compression and decompression software is that there is no optimisation of the data stream for the resources available. Although many compression/decompression systems will allow the user to set certain parameters, such as the maximum bandwidth required to transmit the resulting video file, they do not take into account the original data being compressed and dynamically adjust the compression techniques being use.
As a result, given a fixed set of parameters, low quality video content will be compressed using the same techniques as high quality original video content, when in fact, it may be possible to compress the low quality video content to fit the available bandwidth without any loss in quality. Overall, quality therefore suffers. There is therefore a need for a video compression and decompression system which provides high quality reproduction, optimised for the bandwidth available, without placing excessive processing demands on the end user's computer.
Summary of the Invention
It is therefore an object of the invention to provide a method and system which obviates or mitigates at least one of the disadvantages described above.
One aspect of the invention is broadly defined as a method of compressing video comprising the steps of: a method of compressing video comprising the steps of: digitally sampling an video signal; setting the value of a quality parameter (Q) and the value of a targeted bit rate (TBR); calculating a predicted bit rate (PBR) for a video frame of the video signal; responding to the PBR being greater than the TBR by reducing the value of Q for the video frame; and compressing the video frame in accordance with the value of the Q parameter.
Brief Description of the Drawings
These and other features of the invention will become more apparent from the following description in which reference is made to the appended drawings in which: Figure 1 presents a flow chart of a method of compression in a broad embodiment of the invention; Figure 2 presents a block diagram of a system for compression, transfer and decompression of video files in a preferred embodiment of the invention; Figure 3 presents a flow chart of an overall method of handling video files and generating executable Java applets in a preferred embodiment of the invention; Figures 4A, 4B and 4C present a flow chart of a method of video file compression in a preferred embodiment of the invention; and Figure 5 presents a flow chart of a method of video file decompression in a preferred embodiment of the invention.
Description of the Invention
A methodology which addresses the objects outlined above, is presented as a flow chart in Figure 1. This figure presents a method of compressing video data in which the compression technique and quality parameters are modified dynamically as compression proceeds, so that the quality is optimised in view of a fixed bandwidth that the content is to traverse.
This compression method is generally effected as follows: 1. first, by digitally sampling a video signal in some manner, at step 20; then 2. setting the value of a quality parameter (Q) and the value of a targeted bit rate (TBR) at step 22;
3. adjusting the value of the Q parameter so that the bit rate for a video frame of the video signal is maximized, but will not exceed the TBR at step 24; and
4. compressing the video frame in accordance with the value of the Q parameter at step 28. The video content being compressed can take on any form, analogue or digital. In most computer-based applications, the video content will already be in a digital form, either the product of a digital camera or scanner, or the output of a piece of digital imaging software. Thus, the step of digitally sampling the video signal at step 20 may already have been performed by another component in the system, rather than the core software.
The range and values of the quality parameter (Q) established at step 22, will depend on the specifics of the software application itself. An example is provided in the description of the preferred embodiment of the invention which follows, but clearly, any suitable framework could be used.
Similar, the invention is not restricted by the value of the targeted bit rate (TBR), or the manner in which it is established. Typically, the software operator will generate several executable video packages which are optimized for different downloading rates. As the bottleneck in the downloading process is generally the "last mile" (i.e. the end user's connection to his Internet Service Provider), these executable video packages would be optimised for the most common connection systems, such as 56.6 kbps (kilobit per second) dial-up modems, and various DSL (digital subscriber line) standards. Thus, the end user can select an executable video package that his Internet connection can carry in real time. (Note that the TBR is not the actual connection speed, but the bandwidth that a video package will not be expected to exceed).
The step of adjusting the Q value to maximize the bit rate can also be effected in a number of manners. As noted above, digital video is generally implemented as a series of still images or frames. Thus, the data required for a finite period of time can always be determined, regardless of whether the data are compressed. The Q value could be determined by trail and error, extrapolated from two or more test points, determined in a feed-forward manner or in a similar manner which would be clear to one skilled in the art from the teachings herein.
Finally, the actual compression of the video frame could be effected in any manner known in the art - most compression techniques allow the quality level to be determined in a manner which would be compatible with the invention. A particularly effective compression technique is described in the preferred embodiment hereinafter.
As noted in the Background, it is desirable to generate video files which use as much of the available bandwidth as allowable. Typical compression systems simply do not do this, but rather they allow the operator to set certain parameters governing the compression techniques themselves, or fix the quality level. This approach is misguided - by fixing the quality level, there is no certainty that the video will be able to stream over a given channel, in real time. Similarly, by fixing the compression technique without regard for the video data being fed into the software, the quality will inevitably be sub-optimal and the bandwidth the output data will require, will be unpredictable.
The method of the invention keeps the quality of the compressed video as high as possible for the bandwidth that is available. Thus, the compressed file or files will use as much of the available bandwidth as possible, without exceeding its capacity.
The preferred embodiment of the invention described hereinafter also provides further advantages over the prior art.
Detailed Description of Preferred Embodiments of the Invention
The preferred embodiment of the invention is presented by means of the block diagram in Figure 2, and the flow charts of Figures 3 through 5. Figure 2 identifies the relevant parties in a transaction of the preferred embodiment of the invention, while the specific processing steps are presented in detail in Figures 3 through 5.
In the preferred embodiment, the invention is applied to an Internet and Web site environment. The owner of a Web site (the "software operator") can purchase the software of the invention, use it to compress video clips, and post those video clips on his Web site. These compressed video clips are packaged as executable Java applets, and are represented by an icon on the software operator's Web site.
When an end user clicks on the icon, the video file is streamed to the end user's computer or similar device, and immediately begins to play.
In the preferred embodiment a compressed video clip actually consists of two software components: an executable Java applet, and a data file containing the compressed video content. This way, the executable Java applet can be downloaded and begin executing, while the data can be "streamed" separately. This allows the video content to be displayed as soon as sufficient data has been received. If the two files were combined into one, then streaming could not be done - one would have to wait for the entire file to download before it could be executed. This process is described in greater detail hereinafter. Note that the Java applet and compressed video data files are frequently referred to herein as a single file. This has been done in the interest of simplicity only. It is clearly preferable to implement the invention using separate files to take advantage of the streaming process. Figure 2 presents an exemplary layout of an Internet communications system 30 in a preferred embodiment of the invention. Generally, the Internet 32 is described as a system of routers interconnected by an Internet backbone network, which allows two parties to communicate via whatever entities happen to be interconnected at any particular time. However, it would be known to one skilled in the art that the Internet 32 is far more complex, consisting of a vast interconnection of computers, servers, routers, computer networks and public telecommunication networks.
End users 34 may access the Internet 32 in a number of manners including modulating and demodulating data signals over a telephone line using audio frequencies, which requires a modem and connection to the Public Switched
Telephone Network, which in turn connects to the Internet 32 via an Internet Service Provider 36. Another manner of connection is the use of set top boxes or cable modems which modulate and demodulate data onto high frequencies which pass over existing telephone or television cable networks and are connected directly to the Internet 32 via Hi-Speed Internet Service Providers. Generally, these high frequency signals are transmitted outside the frequencies of existing services passing over these telephone or television cable networks.
An end user 34 may also obtain access to the Internet 32, using a digital cellular telephone, pager, or personal digital assistant. Internet Service Providers (ISPs) 36 or Internet Access Providers (lAPs), are companies that provide access to the Internet. ISPs 36 are considered by some to be distinguished from lAPs in that they also provide content and services to their subscribers, but in the context of this disclosure the distinction is irrelevant. For a monthly fee, ISPs 36 generally provider end users 34 with the necessary software, user name, password and physical access. Equipped with a telephone line modem, cable modem or set top box, one can then log on to the Internet 32 and browse the World Wide Web, and send and receive e-mail.
Web servers 38, 40 are computers which provide text, graphic or multimedia content, or software applications, to other parties over the Internet 32. In the discussion of the invention which follows hereinafter, the software operator 42 will obtain software from the compressor/decompressor software server 38, and the operator's Web site 40 will provide executable, compressed video files and other content to the end users 34.
Of course, the invention may be applied to almost any communication network known in the art, and may be applied to a system of several different networks working together. Such networks could include: wireless networks such as cellular telephone networks, the public switched telephone network, cable television networks, the Internet, ATM networks, frame relay networks, local area networks (LANs) and wide area networks (WANs).
Compression
As explained above, the owner of a Web site simply obtains a copy of the encoding software electronically, and uses it to generate executable video applets. These video applets can then be included with any Web page, linked to it using a graphic icon. End users 34 who visit these Web pages download and execute the video applets by clicking on these icons.
In the preferred embodiment of the invention "asymmetric" compression and decompression is used, that is, only simple computations need to be performed during decompression on the end user's device, while more complex processing may be performed during compression. The power and speed of the processing during compression is not limiting because it need not be performed in real time, unlike the decompression.
The preferred embodiment of the routine for generating compressed video files is presented in the flow charts of Figures 3 and 4. Figure 3 focuses on the handling of the original video data, and how the executable Java applet is generated, while Figure 4 presents the details of the compression routine itself.
The routine begins at step 130 of Figure 3, where, in response to the compression software being launched, the algorithm seeks out the video data file that is to be compressed. The targeted file or files may be communicated to the compression software using any technique known in the art, including dragging files to an icon or identifying the file or files in a command line.
The software operator will now be queried to provide the available data rate that the compressed video is intended for, and the quality level that is desired, at step 132. It may also be desirable to query the software operator for other data, such as the name of the file, where it is to be stored, who generated it and other such details. The software operator will generally choose a targeted bit rate that corresponds to commonly available bit rates that prospective end users may have. For example, the most common dial up modems have bit rates of 56.6 kbps (kilo bits per second), while common data rates for DSL (digital subscriber line) connections include 128, 256, 300 and 500 kbps. The software operator would therefore enter one of these values and identify the file as such, so that this targeted bit rate can be posted on the Web site along with the compressed file.
The quality level is adjusted automatically by the program, so the software operator can set a high value so that the algorithm attempts to maximize the quality at all times. The default setting will also be to the highest level for the TBR.
Next, the algorithm will consider each frame of the targeted file successively at step 134, and perform the compression routine of Figure 4 on that frame, at step 136. As noted above, all digital video data can ultimately be described as a series of individual frames which are successively displayed to the end user. While the invention will typically be applied to frames of digital RGB data
(Red Green Blue data is a standard format for digital video data) in MPEG format, it may also be applied to other input formats. The handling of the RGB MPEG data will be described in greater detail with respect to Figure 4.
Once all of the frames have been compressed, the routine will have generated a Java-based executable, compressed video file which may be stored on the Web server or other storage device at step 140. The Java-based executable, compressed video file may now be accessed from the Internet simply by placing an icon on a Web page, which is linked to stored Java code on the server (at step 142).
The details of the compression routine will now be described with respect to the flow chart of Figure 5.
As noted at steps 134 and 136 above, each frame of the input video data is considered individually. First, each frame is prepared for the compression routine at step 150 by converting it from the input format (say, for example, from RGB format), to YUV format.
RGB format is a video format that is based on the hues red, green and blue. While this format may seem logical because it correlates intuitively with the images being displayed, it is not the most data efficient manner in which to define an image. YUV is a colour encoding scheme for images in which luminance and chrominance are separate. The human eye is less sensitive to colour variations than to intensity variations. YUV is therefore advantageous because it allows the luminance (parameter Y) information to be handled at full bandwidth while the chrominance (parameters U and V) information, to which the human eye is less sensitive, can be compressed. The YUV format is well known in the art of video imaging technology.
While there is no scientifically fixed relationship between the RGB format and the YUV format, the following is a standard relationship as used in the art:
Y = ( 0.299 * R) + (0.587 * G) + (0.114 * B)
V = (-0.169 * R) - (0.331 * G) + (0.500 * B) + 128 U = ( 0.500 * R) - (0.419 * G) - (0.081 * B) + 128
For high contrast frames, "blurred YUV" data is also generated, where adjacent pixels are averaged together. The blurred YUV frames are used in motion estimation only, and not for the final encoding of the pixels.
It has been found that motion estimation does not work very well on high contrast or noisy source video because the noise and the edges generate a lot of error. If the motion estimation error threshold is raised to accommodate, it increases the number of matches as well as false matches. However, by blurring the source video before the motion estimation, it enables the algorithm to detect more matches without increasing a lot any false matches. Next, the algorithm calculates a predicted bit rate (PBR) for the frame being considered, by averaging the bit rate of the previous three seconds of data, at step 152. Other time periods could also be used to calculate this moving average, as could weighted averaging of the data, or similar techniques. Similarly, one could look ahead to future frames and begin to adjust the quality ahead of time. The PBR is then compared to the TBR at step 154, and three cases are established:
1. if the PBR < TBR (that is, the predicted bit rate is less than the bandwidth assumed to be available), then the level of the quality parameter (Q) may be increased at step 156. While an assignment of Q = Q + 1 is used in the preferred embodiment, clearly another assignment could also be used;
2. if the PBR is approximately equal to the TBR but is still less than it, the Q should be decreased by a small amount at step 158. In the preferred embodiment, an assignment of Q = Q - 1 was used; and 3. if the PBR is greater than the TBR, the Q must be reduced so that the compressed data will be able to flow over the available channel at step 160.
An assignment of Q = Q + 1 was used in this case.
Control now passes to step 164 of Figure 4B, where the frame is now considered in terms of blocks of 8 x 8 pixels.
Each 8 x 8 block in the prepared new frame is now compared with the corresponding block in a history frame at step 166, and a "motion estimate" is calculated (see step 194 for a discussion on how the history frame is generated). In general, the concept of motion estimation is well know in the art, and is used in various standards including the MPEG standard. The invention adds several improvements to the traditional process though.
Each frame is divided into 8 x 8 blocks of pixels. For each 8 x 8 block, the routine searches the surrounding pixels in the history frame for the closest match. When the closest match is found, an offset vector (x, y), is calculated, and an error level is assigned at step 168. The error level for the block match is a relative measure comparing the closest matching block to the corresponding block in the history frame. It is only used to perform the decision in step 170, so clearly a number of different models could be used.
Specifically, the search range is +/- 31 pixels along the X and Y axes, thus there are 63 x 63 = 3969 comparisons in a full search (+0 and -0 are the same, hence 63). To speed up the search, the invention employs a quick hierarchical search algorithm to narrow down the search range in 5 rounds. In the first round, the block is compared with the surrounding eight blocks with offsets (-31 , -31 ), (0, -31 ), (31 , 31 ), (-31 , 0), (0, 0), (31 , 0), (-31 , 31 ), (0, 31 ) and (31 , 31 ). Then the one with the smallest error is chosen to be the base for the next round. In the next round, the range is dropped by half to 16. This process is repeated until a single block remains. The total comparisons required in the quick search is about 8 x 5 + 1 = 40 (1% of the full search). The quick search is able to find over 90% of matches found in a full search. Based on the current Q value, an error level threshold will be set to classify each block as a moved block, a new block or a no-change block at step 170. If the error level of the block is higher than the threshold, the block is considered to be new, otherwise, it is classified as moved. A no-change block is a special case of the moved block with offset (0,0). If the Q value is set to a high quality level, then a comparatively high percentage of blocks will be considered new, and a great deal of bandwidth will be consumed. If the Q value is low, a higher percentage of blocks will be considered "moved blocks", so much less bandwidth will be required. As shown in Figure 4B:
1. if the block error level is low relative to Q, then the current block will be considered a moved block, and the vector for the block will be set to the offset (x, y) determined by the motion estimation analysis, at step 172;
2. if the block error level is high relative to Q, then the current block will be considered a new block, and the vector for the block will be set out of range at step 174. This out of range value will be recognized by the decompression software as representing a new block; and
3. if the block error level is close to, or equal to zero, then the current block will be considered an unchanged block, and the vector for the block will be set to (0, 0) at step 176.
As shown, the steps of 164 - 176 are repeated until all of the blocks in the frame have been considered. When step 164 detects that this has been completed, the frame will now be defined by a set of motion estimation vectors. These motion estimation data are compiled into a motion estimation table which is considered at step 178 of Figure 4C.
At step 178, the routine determines whether it would be more efficient to record all of the block changes in the frame, or whether there are so many that it would be more efficient to simply save the entire frame as a JPEG (Joint Photographic Experts Group) image. The JPEG standard is a high-quality compression standard for still pictures, and is well known in the art.
In the preferred embodiment of the invention, the decision point was taken to be 70%; that is, if 70% or more of the blocks in the frame are new or moved, then the routine branches to step 180 where the entire frame is saved as a JPEG image. Clearly, other decision points could used. The size of this JPEG image is then compared with the TBR at step 182, and if the JPEG image is small enough, control passes to step 194. If the image is too large, then remedial measures are taken at step 184. A number of things can be done to reduce the size of the full-frame JPEG image, including the following: 1. full frames can be scaled down, for example, to half-frame resolution as indicated at step 184 in Figure 4C; or 2. full frames can be compressed with lower quality. Most JPEG generating software has a variable quality setting, which can be linked to the Q value used by the routine of the invention. The quality of course, can be increased if the bitrate is lower than the TBR, or decreased if the bitrate is higher than the TBR.
After the remedial measures are effected, control passes to step 190.
If the number of new and moved blocks is determined to be small at step 178, then control passes to step 186 where the motion estimation table is compressed using "Huffman coding". Huffman coding is "lossless" in that no detail or information is lost in compressing and decompressing.
Briefly, Huffman coding is performed by categorizing data points by the likelihood that they will occur. Then, the most common points are assigned shorter codes, and the less likely codes are assigned longer codes.
For example, given six data points A through F, with probabilities as shown below, one could generate a Huffman assignment table as shown:
Point Probability Huffman Code Bit Length
A 0.3 00 2
B 0.3 01 2
C 0.13 100 3 D D 0 0..1122 1 10011 3
E 0.1 110 3
F 0.05 111 3
In straight binary coding, a 3-bit word would be required to encoding the six data points A through F. With the Huffman coding, however, the more common data points have 2-bit words, and the longer ones, 3-bit words. Considering the probabilities as noted above, the average word will have 2.34 bits, which gives 2.34/3 = 78% compression, with no loss of data.
As noted above, the motion estimation is recorded as an offset vector (x, y). All (x, y) pairs are sorted and counted, and popular pairs will be assigned a code in the table. Unpopular pairs will be assigned to one single code and will store the offset (x, y) uncompressed. Note again that a vector of (0, 0) means there as no change from the history frame to the new frame, and that an out of range pair is used for new blocks.
The invention uses predetermined Huffman coding tables within the compressor and decompressor. Adaptive Huffman coding or dynamic Huffman tables could be used, but generally, adaptive Huffman would consume too many CPU cycles to decompress, and dynamic Huffman tables would increase the amount of data to download.
A series of JPEG images are then generated at step 188, corresponding to the blocks that are identified in the motion estimation table.
At step 190, it is then determined whether the data for the current frame is less than the TBR. If the data for the frame still exceeds the TBR, in spite of dropping the quality level as described above, the frame is simply dropped at step 192. Otherwise, the image data is copied to the compressed video data file at step 194.
In the preferred embodiment, the compressed video data and the executable Java decompression code are stored separately. The executable Java decompression code is stored in a zip file containing the Java Applet byte code, and the compressed video data is stored in a separate file which will be streamed down to the client machine during playback.
A new history frame is then constructed at step 196, simply by saving the current image in a buffer, so it can be compared with the next image when the routine is repeated. When the routine runs for the first time, the history frame will be null (i.e. the buffer will be empty), so when the first comparison is made to history frame at step 166, there will be no matches.
Compressed audio data can also be added to the compressed video data file at this point. In the preferred embodiment, the audio and video are interleaved so that synchronization is implicit. For example, the data blocks could be stored as follows: V A A A V A A A V A A A V ("V" characters representing video data, and "A" characters representing audio data).
The number of audio blocks associated with each video frame depends on the audio block size and frame rate, and it varies between frames as it may round up or down to the nearest block. Given, for example, 12 frames per second, 200 sample audio blocks, 8000 samples per second, this will yield: audio samples per frame = 8000 / 12 = 666.667 audio blocks per frame = 666.667 / 200 = 3.333 Thus, the interleaved blocks will be stored in this manner: V AAA V AAAA V AAA V AAA V AAAA V AAA V AAA V AAAA .... In other words, some of the A groups will have three blocks and some will have four. Note that even if there is an empty video frame (dropped or no change), there may still be an audio block associated with it, which must be stored. In this case, null video blocks are inserted which have a size of 0 for the data portion. The decompression software recognizes these 0 blocks as null video blocks, thus ensuring synchronization.
Downloading and Executing an Video File
The compression technique of the invention may be resource intensive itself, but it results in compressed video which may be decompressed using simple operations. Thus, decompression can be performed on the end user's computer or other device. These simple operations can be executed quickly enough, that an executable file can be created which can run in a Java environment.
Java is a code interpretor which is almost universally supported by Web browsers for computers and similar devices. More important, it is platform independent, so that Java code and applets may be executed on any platform.
Because it is an interpretor, Java executes applets much slower than executable machine code. Thus, Java cannot execute the processing intensive decompression techniques used by MPEG and similar video standards, fast enough to provide real time, high quality video. Thus, MPEG players must be implemented using browser plugins and external machine code players. This raises the problems noted in the Background: that MPEG and similar video formats lack universality, and that end users must download and update multiple video players on an ongoing basis.
The JPEG imaging standard used in the invention does use some complex processing, but JPEG is supported by Java natively as one of its standard image formats. Thus, the code to produce the JPEG images is in machine code, and is inherent to Java.
Since the decompression routine of the invention can be applied in a Java environment, this allows Web sites, for example, to provide executable applets which can be downloaded and played by an end user with a Java-enabled browser or operating system. These executable applets can also be delivered to end users in other ways such as via Email or Banner Ads.
Using executable applets, the end user does not have to obtain application software or browser plugins to listen to the content. Thus, the end user does not have to address issues of compatibility with his platform and the format of the video content being downloaded, inconvenience of obtaining upgrades, and possibly requiring many different software packages to address different video formats. In the preferred embodiment of the invention, decompression software is packaged with the video content so the end user simply downloads an executable file. As noted above, there are two main methods in which the preferred embodiment of the invention would generally be implemented: as an icon on a Web page, which is executed when an end user 34 clicks upon it, or as an applet which executes when anyone visits the Web page. In either case, the method presented in the flow chart of Figure 6 would be performed. Execution of this method generally requires that the end user 34 have the following:
1. a device capable of connecting to the Internet;
2. a Java compatible browser, email reader or other Java compatible software application; and 3. a video display system, typically consisting of a video display terminal and a video card. It is also necessary to have a Java applet residing on an Internet Web server for the end user 34 to download.
The routine begins at step 198 of Figure 5, after the end user 34 has initiated the downloading or execution of an applet in one of the manners noted above. The zipped decompression applet is then transmitted to the end user 34 and is unzipped for execution. An auto detection function is performed by the Java applet, as known in the art, to determined the available bandwidth capability of the end user 34. The applet uses this information to determine which of the previously stored compressed video data files should be downloaded, providing the best quality for the bandwidth available.
The compressed video data is then "streamed" to the end user, using the undocumented but commonly used streaming facility in Java. "Streaming" is the process of beginning to play the content before all of the content has been downloaded. In the method of the invention, content can be decompressed and played as soon as the zipped decompression applet and the first sub-block of the compressed video data have been received; meanwhile, the balance of the compressed video content can be downloaded and decompressed in the background. The decompression software simply buffers the compressed video content as it is received and decompressed. As the video card on the end user's computer requires data, it simply calls the buffer for data.
Decoding of the compressed video file is performed on a frame by frame basis, in a manner that is complementary to the compression routine described above.
Each frame of the compressed video file is considered separately at step 200. To begin with, the motion estimate data for the frame is decoded using a predetermined Huffman table that was downloaded with the executable Java applet, at step 202.
The JPEG image or images associated with the frame, are then decompressed at step 204, using a standard JPEG engine as known in the art. If the frame was stored as a full-frame JPEG image, then there will only be one image to decompress and copy to the new frame. If the frame was stored as an assembly of moved blocks, then these blocks are copied from the history frame to the new frame at step 206, being moved in accordance with their decoded motion vectors.
Blocks which are unchanged from the history frame are then copied from the history frame at step 208, to the new frame. If the JPEG image had been scaled down at step 184, it is now returned to full scale at step 210, and is copied to the new frame.
The completed image is now ready for display, and is transferred to the video card for display on the end user's video interface at step 212. This new frame is then buffered as the new history frame at step 214, so that it can be used as the basis for the next frame (if required). Control then returns to step 200, so that the next frame can be considered.
To summarize, the method of the invention provides a number of marketable advantages including the following: 1. compressed, high quality video can be streamed over the existing Internet using a standard 56.6 modem and telephone line, or higher speed connections; 2. asymmetric compression provides sufficiently fast decompression, making a platform independent, Java implementation possible; 3. no plugins or players are required on the end user's device, so there are no difficulties with system compatibility, having to obtain multiple media players, or having to purchase or upgrade software;
4. the decompression software can be run on any Java-enabled Web browser; 5. a regular Web server can be used - special servers are not required;
6. the method of the invention allows various sizes of compressed files to be created, so that the end user 34 can obtain an executable file that is optimal for the bandwidth of his network connection. Most compression systems are static, and the end user 34 has no choice of which video file to download; 7. the efficient compression of the preferred embodiment provides for an extremely small download; and 8. the software operator 42 does not have to perform any complex programming to compress his files or to generate executable applets which may be loaded onto his Web site. All programming code is generated automatically.
The invention is not limited by the nature of the Web page being transmitted. The invention could be used to insert simple banners into Web pages, or more sophisticated multimedia advertisements. As well, these advertisements could be sent along with real audio, real video, telephone over Internet, video conferencing over Internet, or other data and software applications.
While particular embodiments of the present invention have been shown and described, it is clear that changes and modifications may be made to such embodiments without departing from the true scope and spirit of the invention. Portions of the invention could be implemented in part, in different applications.
As well, the invention offers a compression technique that is particularly effective in view of the current trade-off between available resources and video quality. It is expected that the bandwidth and speed of existing communication networks, and the available processing power on various computing platforms will continue to improve, thus the tradeoff curve will slowly shift. This will allow the method of the invention to be implemented with more resource intensive functions. For example, the preferred embodiment of the invention avoids certain resource- intensive techniques such as sub-pixel motion estimation and delta/error encoding. However, as processing speeds increase, it may become practical to employ such techniques. The method steps of the invention need not be implemented as Java code, but may be embodiment in sets of executable machine code stored in a variety of formats such as object code or source code. Clearly, the executable machine code may be integrated with the code of other programs, implemented as subroutines, by external program calls or by other techniques as known in the art.
The embodiments of the invention may be executed by a computer processor or similar device programmed in the manner of method steps, or may be executed by an electronic system which is provided with means for executing these steps. Similarly, an electronic memory medium such computer diskettes, CD-Roms, Random Access Memory (RAM), Read Only Memory (ROM) or similar computer software storage media known in the art, may be programmed to execute such method steps. As well, electronic signals representing these method steps may also be transmitted via a communication network.
The invention could be applied to all manner of appliances having computer or processor control and communication capability, including computers, smart terminals, lap top computers, personal digital assistants, cellular telephones, Bluetooth devices, Internet-ready telephones, televisions, television set top units, and automobiles. Such implementations would be clear to one skilled in the art, and do not take away from the invention.

Claims

WHAT IS CLAIMED IS:
1. A method of compressing video comprising the steps of: digitally sampling an video signal; setting the value of a quality parameter (Q) and the value of a targeted bit rate
(TBR); adjusting said value of said Q parameter so that the bit rate for a video frame of said video signal is maximized, but will not exceed said TBR; and compressing said video frame in accordance with said value of said Q parameter.
2. The method of claim 1 , wherein said step of adjusting comprises the steps of: calculating a predicted bit rate (PBR) for a video frame of said video signal; and responding to said PBR being greater than said TBR by reducing the value of Q for said video frame.
3. The method of claim 1 , wherein said step of responding further comprises the step of: responding to said PBR being less than said TBR by increasing the value of Q for said frame.
4. The method of claim 1 , wherein said step of calculating a predicted bit rate (PBR) comprises the step of: averaging the bit rate of the previous three seconds of compressed video data.
5. The method of claim 1 , further comprising the step of: responding to a frame being high contrast, by performing motion estimation on "blurred YUV" data
6. The method of claim 1 , further comprising the step of: generating blurred YUV data by averaging adjacent pixels together; whereby motion estimation using blurred data will detect more matches without increasing the occurrence of false matches.
7. A method of compressing video comprising the steps of: digitally sampling a video signal; for each frame of said digitally sampled video signal: converting said frame to YUV format; dividing said YUV format frame into blocks; for each block: calculating a motion estimation vector between said frame and a previous frame; compressing said motion estimation vector using Huffman compression; and generating a JPEG image of said block; and merging said Huffman compressed motion estimation vectors and said JPEG images with a Java decompression program.
8. A method of compressing video comprising the steps of: digitally sampling a video signal; establishing an available bitrate; for each frame of said digitally sampled video signal: converting said frame to YUV format; calculating bit rate over the last three seconds; dividing said YUV format frame into blocks; for each block: calculating a motion estimation vector between said frame and a previous frame; classifying said block as moved, new or no-change, in response to said available bit-rate; compressing said motion estimation vector using Huffman compression; and generating a JPEG image of said block, the quality of said JPEG image being dependent on said bitrate; and merging said Huffman compressed motion estimation vectors and said JPEG images with a Java decompression program.
9. The method of claim 2, where said step of coding comprises the step of Huffman coding said residual values.
10. The method of claim 1 , wherein said step of digitally sampling further comprises the step of dividing said digital samples into blocks for said linear prediction operation.
11. The method of claim 9, further comprising the steps of: sub-dividing each said block into sub-blocks; searching past data to identify a prior sub-block which correlates to a current sub- block; calculating a scaling factor which best matches said prior sub-block to said current sub-block; recording the starting point of said prior sub-block and the scaling factor; and subtracting said scaled prior sub-block from said current sub-block.
12. The method of claim 1 , further comprising the step of: wrapping said residual data in a Java applet, including a decompression routine.
13. A method of decompressing a compressed video file comprising the steps of: receiving data corresponding to said compressed video file; sorting said data into blocks; and linearly expanding said data, reproducing said video file.
14. A system for executing the method of any one of claims 1 through 13.
15. A computer readable memory medium for storing software code executable to perform the method steps of any one of claims 1 through 13.
16. A carrier signal incorporating software code executable to perform the method steps of any one of claims 1 through 13.
PCT/CA2002/000641 2001-05-01 2002-05-01 Method and system for video compression and distribution WO2002089486A2 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
AU2002302214A AU2002302214A1 (en) 2001-05-01 2002-05-01 Method and system for video compression and distribution

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CA2,345,878 2001-05-01
CA002345878A CA2345878A1 (en) 2001-05-01 2001-05-01 Multi media distribution method and system

Publications (2)

Publication Number Publication Date
WO2002089486A2 true WO2002089486A2 (en) 2002-11-07
WO2002089486A3 WO2002089486A3 (en) 2003-02-20

Family

ID=4168949

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CA2002/000641 WO2002089486A2 (en) 2001-05-01 2002-05-01 Method and system for video compression and distribution

Country Status (3)

Country Link
AU (1) AU2002302214A1 (en)
CA (1) CA2345878A1 (en)
WO (1) WO2002089486A2 (en)

Cited By (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
GB2397432A (en) * 2003-01-17 2004-07-21 Kwok Ying Kerri Yu Linking self-executable multimedia files
EP1496656A1 (en) * 2003-05-27 2005-01-12 Fuji Photo Film Co., Ltd. Method and apparatus for moving image conversion, method and apparatus for moving image transmission, and programs therefor
EP1619900A1 (en) * 2004-07-22 2006-01-25 Samsung Electronics Co, Ltd Bit rate controller
CN103891303A (en) * 2011-08-16 2014-06-25 黛斯悌尼软件产品有限公司 Script-based video rendering
TWI488461B (en) * 2008-02-01 2015-06-11 Interdigital Patent Holdings Method and apparatus for prioritizing logical channels
EP2950531A1 (en) * 2014-05-30 2015-12-02 Axell Corporation Moving image reproduction method and moving image reproduction system
NL2016702A (en) * 2016-04-29 2017-11-06 E-Comvideo Web-based enriched video.
WO2021071637A1 (en) * 2019-10-09 2021-04-15 Microsoft Technology Licensing, Llc System and method for motion adaptive filtering as pre-process to video encoding

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116760988B (en) * 2023-08-18 2023-11-10 瀚博半导体(上海)有限公司 Video coding method and device based on human visual system

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5333012A (en) * 1991-12-16 1994-07-26 Bell Communications Research, Inc. Motion compensating coder employing an image coding control method
GB2316564A (en) * 1996-04-30 1998-02-25 Daewoo Electronics Co Ltd Adaptive quantizer for video signal transform encoding system
WO1998019450A2 (en) * 1996-10-31 1998-05-07 Sensormatic Electronics Corporation Intelligent video information management system
EP0961490A2 (en) * 1998-05-28 1999-12-01 International Business Machines Corporation Internet convolution audio/video server
GB2340327A (en) * 1998-07-29 2000-02-16 Nokia Mobile Phones Ltd Motion estimation in a video coding system
US6037987A (en) * 1997-12-31 2000-03-14 Sarnoff Corporation Apparatus and method for selecting a rate and distortion based coding mode for a coding system
EP1063851A2 (en) * 1999-06-22 2000-12-27 Victor Company Of Japan, Ltd. Apparatus and method of encoding moving picture signal
EP1075149A2 (en) * 1994-09-29 2001-02-07 Sony Corporation Video encoding

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5333012A (en) * 1991-12-16 1994-07-26 Bell Communications Research, Inc. Motion compensating coder employing an image coding control method
EP1075149A2 (en) * 1994-09-29 2001-02-07 Sony Corporation Video encoding
GB2316564A (en) * 1996-04-30 1998-02-25 Daewoo Electronics Co Ltd Adaptive quantizer for video signal transform encoding system
WO1998019450A2 (en) * 1996-10-31 1998-05-07 Sensormatic Electronics Corporation Intelligent video information management system
US6037987A (en) * 1997-12-31 2000-03-14 Sarnoff Corporation Apparatus and method for selecting a rate and distortion based coding mode for a coding system
EP0961490A2 (en) * 1998-05-28 1999-12-01 International Business Machines Corporation Internet convolution audio/video server
GB2340327A (en) * 1998-07-29 2000-02-16 Nokia Mobile Phones Ltd Motion estimation in a video coding system
EP1063851A2 (en) * 1999-06-22 2000-12-27 Victor Company Of Japan, Ltd. Apparatus and method of encoding moving picture signal

Non-Patent Citations (5)

* Cited by examiner, † Cited by third party
Title
ANDERSON C H ET AL: "Change detection and tracking using pyramid transform techniques" INTELLIGENT ROBOTS AND COMPUTER VISION, CAMBRIDGE, MA, USA, 16-20 SEPT. 1985, vol. 579, pages 72-78, XP008007320 Proceedings of the SPIE - The International Society for Optical Engineering, 1985, USA ISSN: 0277-786X *
FERGUSON T C ET AL: "FRACTAL TRANSFORM TECHNIQUES FOR VERY LOW BIT RATE VIDEO CODING" ISCAS '97. PROCEEDINGS OF THE 1997 IEEE INTERNATIONAL SYMPOSIUM ON CIRCUITS AND SYSTEMS. CIRCUITS AND SYSTEMS IN THE INFORMATION AGE. HONG KONG, JUNE 9 - 12, 1997, IEEE INTERNATIONAL SYMPOSIUM ON CIRCUITS AND SYSTEMS, NEW-YORK, NY: IEEE, US, vol. 2, 9 June 1997 (1997-06-09), pages 1456-1459, XP000832482 ISBN: 0-7803-3584-8 *
KARLEKAR J ET AL: "New multiresolution motion estimation and compensation scheme" CIRCUITS AND SYSTEMS, 1999. ISCAS '99. PROCEEDINGS OF THE 1999 IEEE INTERNATIONAL SYMPOSIUM ON ORLANDO, FL, USA 30 MAY-2 JUNE 1999, PISCATAWAY, NJ, USA,IEEE, US, 30 May 1999 (1999-05-30), pages 459-462, XP010341184 ISBN: 0-7803-5471-0 *
NOSRATINIA A ET AL: "Multi-resolution backward video coding" PROCEEDINGS OF THE INTERNATIONAL CONFERENCE ON IMAGE PROCESSING. (ICIP). WASHINGTON, OCT. 23 - 26, 1995, LOS ALAMITOS, IEEE COMP. SOC. PRESS, US, vol. 3, 23 October 1995 (1995-10-23), pages 563-566, XP010197031 ISBN: 0-7803-3122-2 *
SULLIVAN G J ET AL: "RATE-DISTORTION OPTIMIZATION FOR TREE-STRUCTURED SOURCE CODING WITH MULTI-WAY NODE DECISIONS" MULTIDIMENSIONAL SIGNAL PROCESSING. SAN FRANCISCO, MAR. 23 - 26, 1992, PROCEEDINGS OF THE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH AND SIGNAL PROCESSING (ICASSP), NEW YORK, IEEE, US, vol. 3 CONF. 17, 23 March 1992 (1992-03-23), pages 393-396, XP000378952 ISBN: 0-7803-0532-9 *

Cited By (29)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
GB2397432A (en) * 2003-01-17 2004-07-21 Kwok Ying Kerri Yu Linking self-executable multimedia files
EP1496656A1 (en) * 2003-05-27 2005-01-12 Fuji Photo Film Co., Ltd. Method and apparatus for moving image conversion, method and apparatus for moving image transmission, and programs therefor
US7647428B2 (en) 2003-05-27 2010-01-12 Fujifilm Corporation Method and apparatus for email relay of moving image conversion and transmission, and programs therefor
EP1619900A1 (en) * 2004-07-22 2006-01-25 Samsung Electronics Co, Ltd Bit rate controller
US7451080B2 (en) 2004-07-22 2008-11-11 Samsung Electronics Co., Ltd. Controlling apparatus and method for bit rate
CN100459706C (en) * 2004-07-22 2009-02-04 三星电子株式会社 Controlling apparatus and method for bit rate
KR101058524B1 (en) 2004-07-22 2011-08-23 삼성전자주식회사 Bit quantity control device and control method
US9603161B2 (en) 2008-02-01 2017-03-21 Interdigital Patent Holdings, Inc. Method and apparatus for prioritizing logical channels
TWI488461B (en) * 2008-02-01 2015-06-11 Interdigital Patent Holdings Method and apparatus for prioritizing logical channels
US9913286B2 (en) 2008-02-01 2018-03-06 Interdigital Patent Holdings, Inc. Method and apparatus for prioritizing logical channels
CN103891303A (en) * 2011-08-16 2014-06-25 黛斯悌尼软件产品有限公司 Script-based video rendering
US9137567B2 (en) 2011-08-16 2015-09-15 Destiny Software Productions Inc. Script-based video rendering
US10645405B2 (en) 2011-08-16 2020-05-05 Destiny Software Productions Inc. Script-based video rendering
US9215499B2 (en) 2011-08-16 2015-12-15 Destiny Software Productions Inc. Script based video rendering
US9143826B2 (en) 2011-08-16 2015-09-22 Steven Erik VESTERGAARD Script-based video rendering using alpha-blended images
US9571886B2 (en) 2011-08-16 2017-02-14 Destiny Software Productions Inc. Script-based video rendering
EP2745526A4 (en) * 2011-08-16 2016-04-06 Destiny Software Productions Inc Script-based video rendering
US9380338B2 (en) 2011-08-16 2016-06-28 Destiny Software Productions Inc. Script-based video rendering
US9432727B2 (en) 2011-08-16 2016-08-30 Destiny Software Productions Inc. Script-based video rendering
US9432726B2 (en) 2011-08-16 2016-08-30 Destiny Software Productions Inc. Script-based video rendering
CN105282562A (en) * 2014-05-30 2016-01-27 株式会社艾库塞尔 Moving image playing method and moving image playing system
JP2015228581A (en) * 2014-05-30 2015-12-17 株式会社アクセル Method and system for reproducing moving image
EP2950531A1 (en) * 2014-05-30 2015-12-02 Axell Corporation Moving image reproduction method and moving image reproduction system
US10356431B2 (en) 2014-05-30 2019-07-16 Axell Corporation Moving image reproduction method and moving image reproduction system
US20150350664A1 (en) * 2014-05-30 2015-12-03 Axell Corporation Moving image reproduction method and moving image reproduction system
NL2016702A (en) * 2016-04-29 2017-11-06 E-Comvideo Web-based enriched video.
WO2021071637A1 (en) * 2019-10-09 2021-04-15 Microsoft Technology Licensing, Llc System and method for motion adaptive filtering as pre-process to video encoding
US11062424B2 (en) 2019-10-09 2021-07-13 Microsoft Technology Licensing, Llc Systems and methods for motion adaptive filtering as pre-process to video encoding
CN114514746A (en) * 2019-10-09 2022-05-17 微软技术许可有限责任公司 System and method for motion adaptive filtering as a pre-process for video coding

Also Published As

Publication number Publication date
CA2345878A1 (en) 2002-11-01
WO2002089486A3 (en) 2003-02-20
AU2002302214A1 (en) 2002-11-11

Similar Documents

Publication Publication Date Title
US6961754B2 (en) Interactive access, manipulation, sharing and exchange of multimedia data
US6308222B1 (en) Transcoding of audio data
US20020065922A1 (en) Method and apparatus for selection and redirection of an existing client-server connection to an alternate data server hosted on a data packet network (DPN) based on performance comparisons
KR100869213B1 (en) Storage medium storing data compression program, data compression method, and data compression device
EP1623341B1 (en) Methods, data structures, and systems for processing media data streams
KR100540495B1 (en) A method and apparatus for compressing a continuous, indistinct data stream
EP1438673B1 (en) System and method for communicating media signals
US6832259B2 (en) Dynamic adjustment of transmitted data size for a subscriber device
US20020078241A1 (en) Method of accelerating media transfer
CN100349446C (en) Improved apparatus and method for processing multi-media and general internetwork
US20130010794A1 (en) Generating Multiple Data Steams From a Single Data Source
US7647549B2 (en) Method and device for processing a request for obtaining multimedia data
US7773632B2 (en) Header compress/decompress framework
US8572278B2 (en) Generating multiple data streams from a single data source
EP0961490A2 (en) Internet convolution audio/video server
KR20010071516A (en) Method and apparatus for a client-sever system with heterogeneous clients
EP1625706A2 (en) System for doing service location management taking into account the node and network characteristics
US20020165970A1 (en) System and method for intelligent bit rate and buffer selection
KR20010111380A (en) An internet service apparatus and service method
US7640362B2 (en) Adaptive compression in an edge router
EP1879353B1 (en) Contents distribution system, contents distribution server, contents reproduction terminal, and contents distribution method
WO2002089486A2 (en) Method and system for video compression and distribution
JP4970912B2 (en) Video segmentation server and control method thereof
US20050213610A1 (en) Compressor/decompressor selecting apparatus and method of the same
US20040107293A1 (en) Program obtainment method and packet transmission apparatus

Legal Events

Date Code Title Description
AK Designated states

Kind code of ref document: A2

Designated state(s): AE AG AL AM AT AU AZ BA BB BG BR BY BZ CA CH CN CO CR CU CZ DE DK DM DZ EC EE ES FI GB GD GE GH GM HR HU ID IL IN IS JP KE KG KP KR KZ LC LK LR LS LT LU LV MA MD MG MK MN MW MX MZ NO NZ OM PH PL PT RO RU SD SE SG SI SK SL TJ TM TN TR TT TZ UA UG US UZ VN YU ZA ZM ZW

AL Designated countries for regional patents

Kind code of ref document: A2

Designated state(s): GH GM KE LS MW MZ SD SL SZ TZ UG ZM ZW AM AZ BY KG KZ MD RU TJ TM AT BE CH CY DE DK ES FI FR GB GR IE IT LU MC NL PT SE TR BF BJ CF CG CI CM GA GN GQ GW ML MR NE SN TD TG

121 Ep: the epo has been informed by wipo that ep was designated in this application
AK Designated states

Kind code of ref document: A3

Designated state(s): AE AG AL AM AT AU AZ BA BB BG BR BY BZ CA CH CN CO CR CU CZ DE DK DM DZ EC EE ES FI GB GD GE GH GM HR HU ID IL IN IS JP KE KG KP KR KZ LC LK LR LS LT LU LV MA MD MG MK MN MW MX MZ NO NZ OM PH PL PT RO RU SD SE SG SI SK SL TJ TM TN TR TT TZ UA UG US UZ VN YU ZA ZM ZW

AL Designated countries for regional patents

Kind code of ref document: A3

Designated state(s): GH GM KE LS MW MZ SD SL SZ TZ UG ZM ZW AM AZ BY KG KZ MD RU TJ TM AT BE CH CY DE DK ES FI FR GB GR IE IT LU MC NL PT SE TR BF BJ CF CG CI CM GA GN GQ GW ML MR NE SN TD TG

REG Reference to national code

Ref country code: DE

Ref legal event code: 8642

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 69(1) EPC

122 Ep: pct application non-entry in european phase
NENP Non-entry into the national phase in:

Ref country code: JP

WWW Wipo information: withdrawn in national office

Country of ref document: JP