Motion-Compensated Hybrid Video Encoding

Loading...

En provenance du cours de Northwestern University

Fundamentals of Digital Image and Video Processing

666 notes

Northwestern University

Fundamentals of Digital Image and Video Processing

666 notes

In this class you will learn the basic principles and tools used to process images and videos, and how to apply them in solving practical problems of commercial and scientific interests. Digital images and videos are everywhere these days – in thousands of scientific (e.g., astronomical, bio-medical), consumer, industrial, and artistic applications. Moreover they come in a wide range of the electromagnetic spectrum - from visible light and infrared to gamma rays and beyond. The ability to process image and video signals is therefore an incredibly important skill to master for engineering/science students, software developers, and practicing scientists. Digital image and video processing continues to enable the multimedia technology revolution we are experiencing today. Some important examples of image and video processing include the removal of degradations images suffer during acquisition (e.g., removing blur from a picture of a fast moving car), and the compression and transmission of images and videos (if you watch videos online, or share photos via a social media website, you use this everyday!), for economical storage and efficient transmission. This course will cover the fundamentals of image and video processing. We will provide a mathematical framework to describe and analyze images and videos as two- and three-dimensional signals in the spatial, spatio-temporal, and frequency domains. In this class not only will you learn the theory behind fundamental processing tasks including image/video enhancement, recovery, and compression - but you will also learn how to perform these key processing tasks in practice using state-of-the-art techniques and tools. We will introduce and use a wide variety of such tools – from optimization toolboxes to statistical techniques. Emphasis on the special role sparsity plays in modern image and video processing will also be given. In all cases, example images and videos pertaining to specific application domains will be utilized.

À partir de la leçon

Video Compression

In this module we discus video compression with an emphasis on motion-compensated hybrid video encoding and video compression standards including H.261, H.263, H.264, H.265, MPEG-1, MPEG-2, and MPEG-4.

Motion-Compensated Hybrid Video Encoding 19:38

On Video Compression Standards 18:22

H.261, H.263, MPEG-1 and MPEG-2 28:32

Rencontrer les enseignants

Aggelos K. Katsaggelos
Joseph Cummings Professor
Department of Electrical Engineering and Computer Science

Welcome to week ten.

This is the beginning of the third and last week

covered in this exciting topic of image and video compression.

Based on the material we covered during the first

two weeks, everything should make perfect sense by now.

When comes to video, we first still need to either

perform a prediction, or the decorrelate the data, or do both.

Then quantization takes place and finally entropy encoding.

The dominant approach in video compression is

to first perform temporal prediction through motion.

The estimated motion, a topic we covered in week

four, is used to carry out this temporal prediction.

This results in the so called hybrid motion compensated video encoding.

We will, we will describe this in detail in this first segment.

This hybrid motion compensation we're doing encoder is

the scheme of choice for all video compression standards.

So after we understand this model, we

have really understood all video compression standards.

However, although there are major similarities among standards,

there are also important differences that allow the next

generation video compression standard to achieve the same quality

with the previous generation one at half the bitrate.

We will see whether the factors

contributed to this improvement in performance.

So, with this is mind, let us proceed with the last week on compression.

We will talk this week about video compression.

We see here the block diagram that applies to any data compression system.

I showed the same diagram last week when I talked about image compression.

I show it again to exactly make the point that no matter what

data we're compressing, image, video, speech and

so on, these same three blocks apply.

So with the first block the redundancy is removed.

Entropy is reduced by the second block and

lossless coding is the last block inside the system.

As we saw from last week for redundancy to move

over to dominant approaches use of a predictor and/or a transfer.

This week we'll see that indeed we use both, we use a temporal

predictor and we're going to use a special conformation of the prediction error.

Requantization is used for entropy reduction.

And finally for lossless coding one can pick from the

number of approaches we have covered in back in week eight.

One could use Huffman LZW or arithmetic encoding.

So, although we covered each and every of these elements here,

we talked about prediction, transformation,

requantization, as well as lossless coding.

We will see next how, all these things are combined when we try to compress video.

The outline of the materials we covered this week here is as follows.

We will briefly describe some applications

which require and benefit from video compression.

Then we'll describe in some detail the operations

of the hybrid motion compensated video encoder and decoder.

It represents the dominant approach towards video encoding, and it's

adopted by all video compression standards, which you will also discuss.

As already mentioned, video compression is the

enabling technology behind the multimedia revolution we're experiencing.

It is necessary for a number of applications

where digital video is acquired, stored, and transmitted.

Some examples are shown here.

It is needed for video content acquisition and editing.

For storing this style video, in DVDs or Blu-ray discs.

Broadcasting TV signals over either

satellite, cable, or terrestrial transmission systems.

For sending video over mobile and internet networks, wired and wireless.

For real time conversation applications such as video chat, video conferences and

telepresence systems, security applications where leaders

acquire start compressed as made it.

And the more recent applications where we want to

capture, display, and transmit stereo and multi-view video, in general.

We show here the block diagram of the hybrid motion compensated video encoder.

As we'll see in detail, it's a prediction system.

Some of you actually might have already recognized it.

Like the one we discussed in week eight, when we covered lossless compression.

And the DPCM system we discussed last week.

We'll see it performs redundancy reduction by both

temporal prediction and the use of a transfer.

This is the actually applied on the unpredictable part

of the prediction the so called displace frame difference.

Actually if the operations of this system become crystal clear to you

then we are done with video compression at least at the conceptual level.

Only the fine details will need to be worked out.

So let us work through this system.

The video frame to be encoded arrives at the input of the system.

Assume for the time being that we have a mechanism to predict

the current frame from the past already encoded frames, and here's the prediction.

We then form the difference and this is

the unpredictable part of the predictor, the prediction error.

This error is also called the spacial indifference which has a

good number of zeroes or small values when the prediction is good.

These are the grey values in the image, and is typically nonzero

when the prediction, which is based on motion as we will see, fails.

As we discussed a couple of times in

the previous weeks, this is the signal the image,

which represents the unpredictable part of the predictor,

and that needs to be sent to the decoder.

Says that there are still correlations in this image in processing it

will follow the steps of transform posing that we discussed last week.

First lead images for instance with jpg.

So the DCT is taken.

And the DCT values are quantized.

And then these DCT values are enter encoded over

here and are sent to the decoder or to the channel.

Now let's look at this other here

we have the predicted frame come in this direction.

And from this direction here we have

the quantized DCT coefficient of the prediction error.

After inverse quantization which means again that

we multiple values by the step size

of the quantizer and inverseticity here we have the error in the spatial domain.

So prediction plus correction results in the constructed frame current frame.

The only piece left is to see how prediction is done.

Well, prediction is done through motion, or

the motion vectors, or the motion field.

The underlying assumption here is that we are co, we are encoding the dynamic scene.

And the objects can be tracked over time or that you can find out

where each and every pixel went when going from one frame to the next.

We cover motion estimation in detail during week 4.

You could probably go back and have if you can review the material.

We saw, then, that we can only estimate the optical flow, which, however, gives

us most, in most cases a good estimate of the motion in the scene.

And actually, what we estimate is the two

dimensional projection of the scene on the camera plane.

We also discussed that the underlying assumption with commonly

using estimation algorithms motion estimated algorithms, be block based or

region based matching or optical flow based is that

the brightness in the scene remains the, remains the same.

That's the constant brightness constraint.

And that the objects are rigid.

So one can use the favorite algorithm to perform motion estimation.

In doing so we need at least two frames.

The current frame come in this direction and

the previously reconstructed frame that comes from this direction.

The estimated motion will be used to carry out the prediction

of the current frame from the previously in time reconstructed frame.

This is referred to, typically, as motion compensation.

Instead of motion prediction, since the motion vectors

face backwards in time, instead of forward in time.

So, in other words, we're not asking the question where

a block in the previous frame ended up in the current

frame, but instead, we're asking the question, where did a

block in the current frame come from in the previous frame.

The prediction question would not result in a practical system since it would

have multiple predictions for some pixels in the current frame and zero for others.

I have asked this question before and I'm asking

it now again why don't to use the original car

in then previous frames to carry out motion estimation

compensation since then called the has the original frames available.

Actually I can not tell how many of you know the answer but hopefully quite a few.

The answer is that the decoder does not have the original frames,

and therefore if we were to use the original frames for motion

compensation at the encoder, encoder and decoder would be using different data

to carry out motion compensation and therefore the results would, would differ.

And this would result in what's referred to as error drift.

Of course motion estimation is only taking place in the

encoder so it would estimate the motion using the original frame.

But then since the prediction is now using

the reconstructed frames that would not make sense either.

In other words we would be minimizing the prediction error between

regional frames but then the actual prediction error could be different.

Larger, most probably, since we are predicting from the reconstructed frames.

So as in all predictive models, the parameters of the model need

to be sent to the decoder, and these parameters are the motion vectors.

So the motion vectors are entropy encoded.

They're not quantized, clearly.

And threaded to the bit stream.

As was mentioned before, the decoder is

part of the encoder in this predictive scheme.

So, here is the decoder, it will accept this input here that quantize DCT values.

Of the display frame difference, as well as

the motion vectors when performing for quantization in DCT.

Over here, we'll do prediction plus correction,

and this would be the reconstructed values.

Of course, the decoder would not perform motion

estimation, just compensation using the transmitted motion vectors.

Now, I've been deferring in this light to frames, in the generic term.

Since typically the video frame is broken down into micro blocks or blocks

or agents in performing the operations

of motion estimation, compensation, and transfer quality.

So what they have described here applies equally to the whole frame

or to any smaller part of the frame, a block or a region.

What I just described, actually, is referred to as inter-frame coding

because the correlation between frames is taken into account.

The question that arises is what happens if we cannot

carry out meaningful predictions, and that's what we will examine next.

So let us examine the case when we cannot carry out meaningful predictions.

This happens for example at the frame level if we have a scene cut.

Suppose you're watching the nightly news and the

program switches from the news to a commercial.

Is absolutely no correlation was probably between the last frame

of the news to the first frame of the commercial.

Similarly a new object might enter the scene

and therefore cannot be predicted from the previous frame.

In cases like this we are referring to intra-frame coding or intra-coding.

So here we show exactly the same block diagram as in the previous slide.

But now we have these two color switches.

And this switch is operating synchronization, in sync.

In other words, they're either both open or both closed.

So again, a frame is a damper of encoded, needs to be encoded.

And in case like this that no prediction can be carried out then

both figures are open which means this frame appears at that location as well.

So these frames have traded their still frames so

a jpeg like procedure would be applied to them.

So the DCT is applied, and then quantization, and then entropy

encoding over here, and the zeros and ones will be sent to the decoder.

This inter-encoded frames or blocks will also come down here,

come this directions, so in this quantization and with DCT.

They go around there they so they are stored here in memory so that they

are utilised for future estimation, motion estimation and compensation.

So no motion estimation compensation takes place and therefore no motion

vectors are sent to the decoder for this macroblock or cold frame.

Although in general the decoder needs to

be informed that inter-encoding has taken place.

So all these lines are dotted here since

no operation or signal is travelling through these lines.

I should mention here that inter-encoding at both frame and micro-block level.

It's also performed for other reasons such as top

being error propagation when video is transmitted over a channel.

So these intra macroblocks intra frames they can be de, decoded by themselves.

They don't have temporal dependencies, so therefore

robustness is built into the bit stream.

And they can also be used for easy access of the bitstream.

Finally in every video code there's an important and necessary block, the

rate controller which represents in some sense the brains of the system.

It controls the output rate of the encoded bitstream.

Which can be constant or can be varying.

The function of the rate controller is to make

all the decisions that need to be made and set

all the parameters that need to be set at every

time instance, or for every macroblock or for every frame.

So for example the rate controller would decide whether a

macroblock or a frame should be intra or inter encoded.

We'll set the step size of the uniform quantizer in performing quantization.

Which of course directly affects the bitrate.

And also, it might have some effect on the context that is used

when for example, a context adaptive binary automatic encoder, a CABAC is used.

Clearly, the rate controller is utilizing as input the

actual rate that is generated at every time instance.

And make decisions so that the, this bitrate is shaped.

If the rate for example is too high,

then the quantizer step size should become larger.

We should perform coarser quantization.

And or maybe fewer inter-blocks should be

coded since they require more bits than inter-blocks.

So this block diagram represents a complete high level

block diagram of the so-called hybrid motion compensated video encoder.

There are actually two more blocks, a pre

and post processor that I will mention later.

Again, if the operations of this system are clear to you, then you know how

each and every video compression standard works, and

a lot of other algorithms in the literature.

As already mentioned the decoder which is shown here is part of the encoder.

Let us look first at inter-decoding.

The entropy encoded bit stream is the input to the decoder.

And after entropy decoding, what becomes available are

the quantized DCT coefficient of the prediction error and the motion factors.

The, this is the coefficient, are inversely

quantized, the inverselity is taken and this

brings the prediction error or the motion

compensated motion difference back to the spacial domain.

The motion vectors connect the current frame to be reconstructed with the

previous already reconstructed frames, which are

sitting at the memory of the decoder.

They/re used to predict the current frame from the previously reconstructed frames.

This is the

function here of the motion compensation.

Clearly the expensive motion estimation part is not needed here at decoding.

So the prediction is added here to the

correction, to give rise to the reconstructed frame.

The reconstructed frame is also fed back into

the memory here, to be used for future predictions.

We show here the operations for the intra-decoding of a block or a frame.

After entropy decoding where it becomes available are the quantized

DCT coefficients or the intensities of that block or frame.

So after inverse quantization and inverse DCT in

the spatial domain we have available now the reconstructed intensity.

They would go through all the way and form the output of the decoder.

These blocks or frames will be fed back into the memory here.

So that they can be used for future predictions.

Clearly no motion vectors are fed this way

and therefore no motion compensation is taking place.

Explorer notre catalogue

Rejoignez-nous gratuitement et obtenez des recommendations, des mises à jour et des offres personnalisées.

Coursera propose un accès universel à la meilleure formation au monde, en partenariat avec des universités et des organisations du plus haut niveau, pour proposer des cours en ligne.

© 2018 Coursera Inc. Tous droits réservés.

Télécharger dans l'App Store

Disponible sur Google Play