WEBVTT

00:00:08.491 --> 00:00:09.490
PROFESSOR: Hi, everyone.

00:00:09.490 --> 00:00:10.780
Welcome back.

00:00:10.780 --> 00:00:14.050
So today, I'd like to talk about
positive definite matrices.

00:00:14.050 --> 00:00:17.650
And specifically, we're going
to analyze several properties

00:00:17.650 --> 00:00:19.440
of positive definite matrices.

00:00:19.440 --> 00:00:21.860
And specifically,
we're going to look

00:00:21.860 --> 00:00:24.710
at why each one of these
following statements is true.

00:00:24.710 --> 00:00:29.970
So first off, why every positive
definite matrix is invertible.

00:00:29.970 --> 00:00:32.270
Why the only positive
definite projection matrix

00:00:32.270 --> 00:00:34.860
is the identity matrix.

00:00:34.860 --> 00:00:38.320
If D is a diagonal matrix
with positive entries,

00:00:38.320 --> 00:00:41.010
show that it must also
be positive definite.

00:00:41.010 --> 00:00:44.260
And then lastly, if S is
a symmetric matrix where

00:00:44.260 --> 00:00:46.980
the determinant S
is bigger than 0,

00:00:46.980 --> 00:00:48.970
show why this might
not necessarily

00:00:48.970 --> 00:00:50.870
imply that it's
positive definite.

00:00:50.870 --> 00:00:53.239
So I'll let you think
about these for a moment.

00:00:53.239 --> 00:00:54.530
And I'll come back in a second.

00:01:08.730 --> 00:01:09.410
Hi, everyone.

00:01:09.410 --> 00:01:10.640
Welcome back.

00:01:10.640 --> 00:01:11.170
OK.

00:01:11.170 --> 00:01:15.820
So let's take a look
at part A. So part A

00:01:15.820 --> 00:01:21.140
is asking why every positive
definite matrix is invertible.

00:01:21.140 --> 00:01:24.680
Well, let's just recall
that if A is a matrix

00:01:24.680 --> 00:01:33.010
and if A is invertible,
then this necessarily

00:01:33.010 --> 00:01:37.795
implies that the determinant
of A is non-zero.

00:01:40.890 --> 00:01:42.500
And I'm going to
just write out det

00:01:42.500 --> 00:01:53.710
A as the product of the
eigenvalues of A. So lambda_1

00:01:53.710 --> 00:02:06.180
to lambda_n are the
eigenvalues of A.

00:02:06.180 --> 00:02:07.070
OK.

00:02:07.070 --> 00:02:18.480
In addition, if A is
positive definite,

00:02:18.480 --> 00:02:21.000
what does this say about
the eigenvalues of A?

00:02:21.000 --> 00:02:24.900
Well, it says that each
eigenvalue of A, lambda_1,

00:02:24.900 --> 00:02:30.700
lambda_2, dot, dot, dot, to
lambda_n, each one of them

00:02:30.700 --> 00:02:33.440
must be bigger than 0.

00:02:33.440 --> 00:02:36.700
So this statement that
each eigenvalue of A

00:02:36.700 --> 00:02:38.790
is bigger than 0 is
completely equivalent

00:02:38.790 --> 00:02:46.860
to A being positive definite
for symmetric matrices A.

00:02:46.860 --> 00:02:50.260
So if I have a whole bunch of
eigenvalues and each of them

00:02:50.260 --> 00:02:52.880
are bigger than 0, what
does this say about det A?

00:02:56.530 --> 00:02:58.780
Well, I can take the product
of all these eigenvalues.

00:03:03.460 --> 00:03:06.740
And of course, the product of a
whole bunch of positive numbers

00:03:06.740 --> 00:03:07.660
must also be positive.

00:03:10.540 --> 00:03:14.640
So if the quantity is positive,
then it certainly cannot equal

00:03:14.640 --> 00:03:15.800
0.

00:03:15.800 --> 00:03:18.770
So this proves that det
A is not equal to 0.

00:03:18.770 --> 00:03:22.590
Hence, A must be invertible.

00:03:22.590 --> 00:03:23.090
OK.

00:03:23.090 --> 00:03:29.850
So for part B,
we're asked to show

00:03:29.850 --> 00:03:31.810
that the only positive
definite projection

00:03:31.810 --> 00:03:35.070
matrix is the identity matrix.

00:03:35.070 --> 00:03:37.220
So again, how do we
tackle this problem?

00:03:37.220 --> 00:03:39.930
We're going to look
at the eigenvalues.

00:03:39.930 --> 00:03:49.690
So remember, if P
is a projection,

00:03:49.690 --> 00:03:52.080
what does it say about
the eigenvalues of P?

00:03:52.080 --> 00:04:03.520
Well, it says that
the eigenvalues of P

00:04:03.520 --> 00:04:05.370
are either 0 or 1.

00:04:09.390 --> 00:04:11.520
So this is point one.

00:04:11.520 --> 00:04:25.150
Point two, if P is
a positive definite,

00:04:25.150 --> 00:04:27.706
what does that say about
the eigenvalues of P?

00:04:27.706 --> 00:04:43.670
Well, as I've noted before,
it means that the eigenvalues

00:04:43.670 --> 00:04:44.580
are bigger than 0.

00:04:47.210 --> 00:04:51.700
So if P is a projection
and it's positive definite,

00:04:51.700 --> 00:04:54.540
the only possible
eigenvalues that

00:04:54.540 --> 00:04:59.310
are both 0 and 1 and
bigger than 0 are 1.

00:04:59.310 --> 00:05:09.540
So the conclusion is
that the eigenvalues of P

00:05:09.540 --> 00:05:12.950
must all equal 1.

00:05:12.950 --> 00:05:17.870
So which matrix has eigenvalues
1 and is also symmetric?

00:05:17.870 --> 00:05:26.110
Well, the only matrix that
satisfies this property

00:05:26.110 --> 00:05:28.660
is the identity matrix.

00:05:28.660 --> 00:05:31.270
Now you might ask, how do
you actually show that?

00:05:31.270 --> 00:05:35.170
Well, you could
argue as follows.

00:05:35.170 --> 00:05:47.350
If P is diagonalizable--
and every symmetric matrix

00:05:47.350 --> 00:05:50.980
is diagonalizable, so
I'm not making this up.

00:05:50.980 --> 00:05:52.770
So if P is
diagonalizable, then you

00:05:52.770 --> 00:05:56.870
can always write P
as some matrix, U,

00:05:56.870 --> 00:06:00.530
times a diagonal matrix--
and we know in this case,

00:06:00.530 --> 00:06:03.900
the diagonal matrix
has eigenvalues 1,

00:06:03.900 --> 00:06:08.907
so it's actually the identity
matrix-- times the inverse

00:06:08.907 --> 00:06:09.990
of the eigenvector matrix.

00:06:12.530 --> 00:06:17.400
But of course, this is
just U times U inverse,

00:06:17.400 --> 00:06:21.660
which then gives me the
identity at the end.

00:06:21.660 --> 00:06:23.970
So U times the identity
times U inverse.

00:06:23.970 --> 00:06:25.859
This is just U times U inverse.

00:06:25.859 --> 00:06:27.900
And of course, U and U
inverse collapse back down

00:06:27.900 --> 00:06:29.050
to the identity.

00:06:29.050 --> 00:06:32.850
So this shows you that the only
matrix that has eigenvalues

00:06:32.850 --> 00:06:35.420
of 1 is the identity matrix.

00:06:35.420 --> 00:06:39.550
So that's just to cross all
the T's and dot all the I's.

00:06:39.550 --> 00:06:40.050
OK.

00:06:40.050 --> 00:06:45.120
For part C, we're given
D as a diagonal matrix

00:06:45.120 --> 00:06:47.140
with positive entries
on the diagonal.

00:06:47.140 --> 00:06:50.140
Now we have to show that
it's positive definite.

00:06:50.140 --> 00:06:50.640
OK.

00:06:50.640 --> 00:06:53.102
So let me write D as follows.

00:06:53.102 --> 00:06:54.310
I'll just write it like this.

00:06:54.310 --> 00:06:57.580
I'm going to use a compact
notation, which is sometimes

00:06:57.580 --> 00:07:05.600
seen: diagonal d_1, d_2, d_n.

00:07:05.600 --> 00:07:09.010
So D is a diagonal matrix whose
elements along the diagonal

00:07:09.010 --> 00:07:13.130
are d_1, d_2, dot,
dot, dot, to d_n.

00:07:13.130 --> 00:07:16.220
Now what does it mean for a
matrix to be positive definite?

00:07:16.220 --> 00:07:28.150
Well, it means that for
any x, for any vector x,

00:07:28.150 --> 00:07:34.950
I have to look at the
product x transpose D*x.

00:07:34.950 --> 00:07:38.070
And I have to show that
it's bigger than 0.

00:07:38.070 --> 00:07:41.100
And I should
qualify this and say

00:07:41.100 --> 00:07:43.330
that the vector we're looking
at is x not equal to 0.

00:07:43.330 --> 00:07:46.050
So we're now looking
at the zero vector.

00:07:46.050 --> 00:07:47.710
But for D to be
positive definite,

00:07:47.710 --> 00:07:51.320
we have to show that x
transpose D*x is bigger than 0.

00:07:51.320 --> 00:07:53.894
This is just one way to show
that it's positive definite.

00:07:53.894 --> 00:07:54.810
It's not the only way.

00:08:00.160 --> 00:08:06.872
So if I write x out using
components, x_1, x_2, dot, dot,

00:08:06.872 --> 00:08:15.300
dot, to x_n-- I'll
write it like this--

00:08:15.300 --> 00:08:19.050
then you can work out the
quantity x transpose D*x.

00:08:19.050 --> 00:08:21.160
And we see that we
get a sum of squares.

00:08:21.160 --> 00:08:28.690
We get d_1 times x_1 squared
plus d_2 x_2 squared plus dot,

00:08:28.690 --> 00:08:33.694
dot, dot plus d_n x_n squared.

00:08:36.960 --> 00:08:41.750
Now by definition, each
coefficient is positive.

00:08:41.750 --> 00:08:44.290
A sum of a square is positive.

00:08:44.290 --> 00:08:45.980
so-- sorry.

00:08:45.980 --> 00:08:49.180
A product of a positive number
with a square is positive.

00:08:49.180 --> 00:08:51.220
And then of course, a
sum of positive numbers

00:08:51.220 --> 00:08:54.060
is going to be positive.

00:08:54.060 --> 00:08:57.340
So this means the whole
thing is positive.

00:08:57.340 --> 00:08:59.940
Now there's other
more efficient ways

00:08:59.940 --> 00:09:01.950
of getting at this using
other tricks we know.

00:09:01.950 --> 00:09:04.230
For example, if we're
given a diagonal matrix,

00:09:04.230 --> 00:09:06.700
we know its eigenvalues
are already d_1,

00:09:06.700 --> 00:09:09.130
d_2, dot, dot, dot, to d_n.

00:09:09.130 --> 00:09:13.240
And we know that a matrix
with positive eigenvalues

00:09:13.240 --> 00:09:16.010
is already positive definite.

00:09:16.010 --> 00:09:20.350
But this is kind of
starting from the base

00:09:20.350 --> 00:09:23.930
to show that it's
positive definite.

00:09:23.930 --> 00:09:27.910
And now lastly,
let's look at part D.

00:09:27.910 --> 00:09:32.730
So S is a symmetric matrix
with det S bigger than 0.

00:09:32.730 --> 00:09:34.510
Show that S might
not necessarily

00:09:34.510 --> 00:09:36.110
be positive definite.

00:09:36.110 --> 00:09:38.020
So there's lots of
counterexamples.

00:09:38.020 --> 00:09:39.480
I only need to construct one.

00:09:42.390 --> 00:09:44.837
So I'm looking at S, which
is a symmetric matrix.

00:09:44.837 --> 00:09:46.920
So I'm just going to throw
in some numbers on some

00:09:46.920 --> 00:09:47.930
off diagonals.

00:09:47.930 --> 00:09:50.400
So I'll just pick one.

00:09:50.400 --> 00:09:55.040
And now I need to pick some
numbers along the diagonal.

00:09:55.040 --> 00:09:56.710
And the easiest way
to do it is just

00:09:56.710 --> 00:09:59.900
to pick two negative
numbers on the diagonal.

00:09:59.900 --> 00:10:04.170
Because if there's a negative
number on a diagonal,

00:10:04.170 --> 00:10:06.972
then we know S can't
be positive definite.

00:10:06.972 --> 00:10:08.680
And I'll say more
about that in a second.

00:10:12.390 --> 00:10:14.380
So I can just pick
negative 2 and negative 3.

00:10:14.380 --> 00:10:15.955
So let's quickly
check what det S is.

00:10:21.520 --> 00:10:25.460
Well, it's negative 2
times negative 3 minus 1.

00:10:25.460 --> 00:10:29.820
So this gives me 6
minus 1, which is 5.

00:10:29.820 --> 00:10:33.610
So by construction, det S
is positive, which is good.

00:10:33.610 --> 00:10:37.410
And then, as I mentioned
before, if there's

00:10:37.410 --> 00:10:39.700
negative elements along
the diagonal of the matrix,

00:10:39.700 --> 00:10:41.500
that matrix can't be
positive definite.

00:10:41.500 --> 00:10:43.250
Well, why is that?

00:10:43.250 --> 00:10:45.850
Well, suppose I
wanted to take a look

00:10:45.850 --> 00:10:52.730
at this upper left
component, negative 3,

00:10:52.730 --> 00:10:55.370
and it's negative, how do
I show that that implies

00:10:55.370 --> 00:10:58.710
S is not positive definite?

00:10:58.710 --> 00:11:03.133
Well, what I can do is I can
look at the product x transpose

00:11:03.133 --> 00:11:03.633
S*x.

00:11:06.790 --> 00:11:08.750
And what I do is I look at it.

00:11:08.750 --> 00:11:12.170
I just take one value of x.

00:11:12.170 --> 00:11:15.050
So we know that this has to be
positive for every value of x.

00:11:15.050 --> 00:11:17.280
So I can pick any value I want.

00:11:17.280 --> 00:11:22.660
So I can take x, say,
[1, 0] transpose.

00:11:25.600 --> 00:11:33.930
And when I do this, we end up
getting that x transpose S*x is

00:11:33.930 --> 00:11:37.270
equal to negative 3.

00:11:37.270 --> 00:11:40.370
So notice how by taking 1
in the first entry and 0

00:11:40.370 --> 00:11:43.770
on the second entry, that picks
out the upper left corner,

00:11:43.770 --> 00:11:44.990
negative 3.

00:11:44.990 --> 00:11:48.010
If I were to take
0 here and 1 here,

00:11:48.010 --> 00:11:52.000
it would pick out the
negative 2 entry in S.

00:11:52.000 --> 00:11:58.200
So by picking 1 in
entry i of a vector x,

00:11:58.200 --> 00:12:01.530
and then computing this
product x transpose S*x,

00:12:01.530 --> 00:12:05.640
I pick out the i-th
element along the diagonal.

00:12:05.640 --> 00:12:09.740
And since that
element is negative,

00:12:09.740 --> 00:12:12.490
this shows me that along
the direction [1, 0],

00:12:12.490 --> 00:12:15.440
the product x transpose
S*x is also negative.

00:12:15.440 --> 00:12:19.090
And hence, S can't possibly
be positive definite.

00:12:19.090 --> 00:12:19.990
OK.

00:12:19.990 --> 00:12:21.730
So just to summarize,
we've taken a look

00:12:21.730 --> 00:12:25.120
at a couple matrices and a
couple different properties

00:12:25.120 --> 00:12:27.090
of positive definite matrices.

00:12:27.090 --> 00:12:30.510
And notably, we've
used the eigenvalues

00:12:30.510 --> 00:12:33.330
to get a handle of the
positive definite matrices.

00:12:33.330 --> 00:12:36.120
And I hope these provide
some useful tricks.

00:12:36.120 --> 00:12:38.570
And I'll see you next time.