0:00

Hello and welcome back. Let us now describe the first image

Inpainting model. That's going to base, be based on partial

differential equations that we learned the last week.

But again, this is going to be self contained, we're going to understand

exactly what kind of partial differential equation are we going to use?

When we talk about image in painting we talk about the following.

We have an image. Here and we have a region.

Where we want to change the content of the image.

In this case it's the cable. Here.

Someone's going to provide us this. Image and painting.

Includes the topic where we basically have these two inputs into the system.

We are not going to detect, this is a very subjective, very often a very

subjective topic about what is that we want to change.

Maybe somebody wants to change the shoes. In this case, we want to change this.

Now, how we detect these. We talk a lot about segmentation.

So we can segment out objects. And those are the regions of missing

information. We are going to call them origins to be

inpainted. With these two images, we want to make,

basically, in this case, the cable disappear.

We want to inpaint in. These regions.

And the first topic we're going to discuss is how do we inpaint with rich,

with information from the surrounding regions.

We're not going to take information from very far away.

We are only going to use information from the surrounding regions.

So we get two images. The original image and an image which is

a binary mask of the, representing the regions that we want to in paint.

When I personally started working on Image Inpainting with my group, we

visited a professional restaurenteur, a professional curator at the Minneapolis

Institute of arts. And we asked this person how do you

inpaint, how do you restore? And its very interesting because they

were doing it the way that children draw. And here is the illustration.

We have a region to be inpainted and we have the image.

So, they first continue the boundaries of the regions.

So, if there is a boundary here going inside the region to being painted.

So then they continue the boundary inside.

The same way that a child would basically draw the outlines of the objects.

Then, they complete the color. So first, they continue the boundaries.

Then, they propagate the colors inside the region.

And then, lastly, they normally add kind of noise because paintings, and in this

case it was a professional restorator of paintings don't look flawed so you have

to kind of add this granularity to make it look more natural.

And we are going to talk about these. First we will talk about these two steps.

Then we are going to discuss a bit about this step which there are many techniques

to do that. That's what.

So remember, there's kind of a continuation of the geometry, basically

the boundary and then filling with the colors.

It's like if this is water, we are going to let the water flow into the

region, but because we have the boundary, it's going to actually, the right water

is going to flow into the right regions. Now, the job is to design a partial

differential equation that emulates this process.

And here it is. So we have basically the image outside

and we have a region of missing information inside.

And we want to propagate the information from outside in.

The way we're going to do that is with this equation.

This is the first component of imaging painting with partial differential

equations. So what's happening here?

Is the information that we want to propagate.

I am not telling you yet what is that information.So is the information.

That we want to propagate. And n is the direction that we're

propagating that information. So n is the direction that we're doing

the propagation. Now look what happens here.

We have gradient l meaning the change of L.

Inner product with N = 0. This is a change of A and it's projected,

because of the inner product, into N, the direction of propagation and we want it

to be equal to zero. So this is of course a vector.

It's the gradient of information. So we have a vector.

And we want that vector to be perpendicular to the direction that we

are propagate, and we're have, so this is going to be the gradient of L, and this

is going to be the direction. We want them to be basically

perpendicular in the sense that we want to propagate information L.

In such a way that is it does not change, it does not change in the direction of

the propagation.That's exactly what it means propagating information.

I move it in the direction that is constant along that direction.

That's what this equation is basically telling us.

That L, the information is constant in the direction of propagation, constant

along these directions. Now how do we solve this with partial

differential equations? Assume that the image is I.

[SOUND] Now, we have this and we want to deform the image, we want to change the

image, this is a partial differential equation, in such a way that at the end,

we get this equal to zero. We have seen how to do that when we're

talking about Euler-Lagrange equations. When we want something to be equal to

zero, how do I deform the image in order to accomplish that?

Let's think for a second how to do that. Remember last week.

How do we do that? Very simple.

We basically do this. Let me just go back one sec.

We basically do this. We make the image change in time

according to what we want. Now, when this doesn't change anymore,

meaning that steady state, we get this equal to zero.

That's the definition of steady state: no more change.

Actually, the computer, when this very little change, we can stop.

When this is ideally zero, we got what we wanted.

L. Constant in the direction of propagation.

So, this is actually a very simple trick. Every time you want to do something like

A = zero, whatever is A, some information of the image, you do dI dT = A.

And you run that to a steady state. That's basically what we did for the all

day Lagrange. So this is the basic equation for this

type of imaging painting. Now, we haven't finished.

I have to tell you what's L and what's M. What's the information that we want to

propagate? And in which direction do we want to

propagate? And there's a lot of art here.

So I'm just giving you examples. I'm teaching you the concepts.

And these are examples that are already very powerful.

But you can come with different examples. Now, let me start by the information.

We want this propagation to be smooth. We don't want to see a big jump inside

the region that we are inpainting. If we see a big jump, we're going to

notice that. So we want it to be smooth.

So it has to be anything that represent basically smoothness in the image, and

one of the things that represent smoothness is the Laplacian of the image,

the second derivative of the image represent smoothness.

Now. Why not the first derivative?

Wait a second, and you're going to understand why.

Because the first derivative is also a representation of this bothness.

Now, in which direction are we going to propagate?

And now we go back to what we learned from professional restorators.

They say, continue edges. Now, we know that if we have an edge, the

gradient of the image is perpendicular to the edge.

We learned that, that week, last week, so we are going to want to propagate along

the edges, which means that we have to take the perpendicular to the gradient,

so n is the perpendicular to the gradient, as written here, or we can

write that n is basically the gradient. But the perpendicular to it.

So that's important thing here, that we want to continue edges.

And then, for that, we have to go perpendicular to the gradient.

And that basically also explains why we cannot take the first derivative of L.

Because if we were to put L. Equal to the first derivative, let's say

the gradient. These two guys are always zero.

So I'm not solving anything. Because the gradient and its

perpendicular are always zero. And I wanted basically to go kind of.

Constant in that direction. This is not enough.

There is also another reason why, okay this is always perpendicular, it's not

very good. Although remember we're taking the

gradient of L. But the other reason is because we want.

Two things to be smooth inside. We want to continue the grey values

inside to be smooth. We also want to con, continue the

boundaries to be smooth inside, so we need higher order of the realities.

Not only because these two guys are perpendicular and then there's not enough

information, even if we take the gradient of L, which is what actually N pending is

propagating. We want to go a bit higher order so we

get even more smoothness. And as I said, the Laplacian is one of

the simplest measures of smoothness. So, we get smoothness.

In the opposite direction or perpendicular to the gradient or on the

level line's direction. And that basically gives us our equation.

So, we have the change of l. And this is the normal, is the

perpendicular to the gradient. What we're going to get at the end of

this, when we get to steady state, when this becomes zero we basically get that

the Laplacian became constant along the edges.

And we have propagated the Laplacian along the edges.

That's what basically this basic equation makes and once again you just discretize

it in the computer and you run it and you get Image Inpainting.

You get propagation from outside in. Every differential equation has to have

boundary conditions. So, if this is my image and this is my

region to be in painting, what do I assume here?

And these boundary conditions for this type of equations, a bit of a strange

equation, it has three derivatives, two because of the Laplacian, one more

because of the gradient. Remember.

We talked about this in the past. But the Laplacian is the second

derivative of the image. In one direction plus the second

derivative of the image in the other direction.

And then we take the gradient of this. And we get a third derivative of

considering both gray values and edges in the boundary and those are smoothly

propagating inside thanks to have basically the Laplacian here and these

are as we talked the level lines of the image the isopoles the regions of edges,

the directions of edges. These are very simple but very, very

interesting equation. Let us see some examples.

So always when you do image processing, start with very simple examples to see

what's happening. And here is a very simple example.

White is the region. Then it's to be inpainted.

And you can see here that there is a very smooth continuation.

Really, really nice. A smooth continuation.

It continues the boundaries nice. It, of course, continues the colors,

black and grey. But even more, more important, it really

completes the circle. It doesn't know that there is a circle

there. It just goes in the normal direction.

Perpendicular to the normal direction in the edge direction.

Remember the gradient is in this direction.

The perpendicular to the gradient is in this direction.

So it's going and it's filling in. Inside the region.

Propagating information in that direction.

Really, really nice result. Of course, if it works only in artificial

images it's not very good. Hopefully, it works on the, also on real

data. Here is an example.

What is being impainted? The letters.

We wrote on top of this image. This is a nice image of New Orleans.

And we draw with a basic rough and tough and boom, the letters are gone.

So the regions to be painting are all these letters and we go inside and we

make the letters disappear. We're propagating information from

outside in. So hard to see here and now you get the

very nice image where we invented the information that is covered by the

letters. Based on information that is surrounding

the letters. Very nice result, I think.

Here is another example of basically an image that has deteriorations and

degradation. The red are the images, the regions, that

we are going to in paint. Sometimes, you mark regions that are a

bit larger than the actual missing information.

Look here. Is very thing.

You do it a bit larger. It's like giving an impulse.

So this is propagated from outside, in. You start from a, be far away.

It helps you have to move even more smooth as we want.

And here are the results. And it looks pretty good, some place you

might see a tiny error here because it's covering a lot of the eye of this girl,

so it's very difficult to invent information when there is nothing in the

surrounding area that hints about what needs to be basically put in.

But of course you get an extremely good. At least initial result looks almost

excellent and the a professional restorator you see different programs may

basically be able to go and repair only that tiny region if needed to.

Sometimes, there's no need for that. Here is the original one.

We have already seen it. A person is jumping with a cable.

Now the cable is gone, and now you see a zooming region.

Once again, look how nice the socks continue.

Very nice continuation of this thread H because that's what we wanted, we were

continuing HX that's exactly what we wanted, and that's exactly what this type

of algorithm does. So, on our example, just a special

effect, let's just look at what's happening here.

And basically they're all gone. I'm going to just show you that again.

There are people in the boat and they're all gone.

We're basically in-painting what's basically been covered by the people

sitting on the boat and basically they're all gone.

Again, in-painting means modifying an image in a form that is then

non-detectable, at least by the regular consumer.

Here is another example of, we have scratches and now the scratches are gone

basically, we basically removed those scratches.

We inpaint in the information from the surrounding areas and again, looks very

nice. It looks very natural as it was before it

was all scratched. Now there is one example.

And this is the last example I want to show in this video where it's actually

not very hard to detect the regions that need to be inpainted.

In other previous examples we, basically the user, defined somehow the regions.

Now, this is an example, for example, of wireless transmission of images.

Remember, images are transmitted, are restored, using JPEG.

JPEG works in blocks. What happens is, if when you're

transmitting an image, there's a signal drop.

For example, on the cell phone. So half a block gets dropped.

Now, signal drops are not very hard to detect, and basically.

You, you have no signal and you detected it.

Okay? So when there's a drop you can say, hey,

I'm basic going to recover that block by mention painting.

And that's what we are simulating here. And this is the original and all these

blocks that you see here basically are dropped signals, simulation of dropped

signal. You see that there's no signal, you plug

that into a painting and then boom, you reconstruct your image.

Let me show you that again, you basically see signal drop.

And you reconstruct. So basically you may have a better

cellular phone, a cellular phone that detect signal drop in images and boom

does imaging painting, this is still JPEG, still the person that is sending it

to you doesn't know. That the signal might drop.

At the receiver end, you see a signal drop.

This actually can help even compression. You can drop on purpose some of the

blocks, because we know how to recover them with image in painting, and in that

way, you basically improve the compression ratio.

Some blocks are very easy to recover, and then you can actually not even send them

or not even store them. Makes the reconstruction a bit more

computationally expensive, but makes the compression possibly much higher.

So this is the last example from this video.

But there is one thing I haven't told you and that was on purpose just to keep you

wondering a bit. I gave you the equation of.

Image in painting, but here we have three colors.

How do we do that? How do we go from that equation that I

gave you before to colors? Remember the equation was something like

IT equal gradient of ii and the I. In a product.

Gradient I. Perpendicular, so.

You can basically do this for each one of the channels.

Lets say in red, green, blue or in any other color space.

Or you can think about extending this to color images, to vectors.

We know the concept of gradient for color vectors and we can basically define the

Laplacian for color vectors as well. So, both possibilities exist.

You can treat every color independently or you can try to use definition of

vectorial fields for these, every component here Laplacian and gradient in

vectorial fashion. All the examples I show you, every color

is treated independently. So, this is an example of imaging pending

with partial differential equations. Next video, I'm going to show you an

example of energy painting with variational formulations that we learned

last week. Very simple variation of formulation that

will lead us to also a very nice energy painting technique based on the same

colors of propagation and boundaries. I'm looking forward to seeing you in the

next video. Thank you.