Lesson 13Orthonormal Bases and Orthogonal Projections
Math 21b
March 9, 2007
Announcements
I Graded midterms will be returned after class. µ = 77, σ = 10
I Solutions to the first midterm are on the course website.
I Homework for March 12: 5.1: 6,16,18,20,28,38*,14*
I Problem Session Wednesdays, 7-8 PM in SC 101b
I Office hours: Monday 2-4, Tuesday 3-5 in SC 323 (resumingMonday)
Here’s Felix!
Our friend the dot product
DefinitionIf ~u and ~v are two vectors in Rn, we define their dot product to be
~u · ~v = u1v1 + · · ·+ unvn.
In Math 21a1 the dot product in R2 and R3 is used for manydifferent geometric operations:
I projecting a vector onto a line
I Find the equation of a plane
I measuring the angle between two vectors
We want to generalize these constructions to several dimensions.
1Read Appendix A if you did not take Math 21a.
Definition
a. Two vectors ~v and ~w in Rn are called perpendicular ororthogonal if ~v · ~w = 0.
b. The length or magnitude or norm of a vector ~v in Rn is thescalar ‖~v‖ =
√~v · ~v .
c. A vector ~v in Rn is called a unit vector if its length is 1, i.e.,‖~v‖ = 1 or ~v · ~v = 1.
Example
Find two unit vectors in R2 which are orthogonal.
SolutionOne pair is (~e1,~e2). Can you find more?
Example
Find two unit vectors in R2 which are orthogonal.
SolutionOne pair is (~e1,~e2). Can you find more?
Definition
I A set (~u1, ~u2, . . . , ~um) of vectors in Rn is called orthogonal ifeach is orthogonal to every other.
I The set is called orthonormal if additionally every vector is aunit vector.
Thus a set (~u1, ~u2, . . . , ~um) of vectors in Rn is orthonormal if andonly if
~ui · ~uj =
{0 if i 6= j
1 if i = j.
The physicists have a slick way to write this: ~ui · ~uj = δij .
Definition
I A set (~u1, ~u2, . . . , ~um) of vectors in Rn is called orthogonal ifeach is orthogonal to every other.
I The set is called orthonormal if additionally every vector is aunit vector.
Thus a set (~u1, ~u2, . . . , ~um) of vectors in Rn is orthonormal if andonly if
~ui · ~uj =
{0 if i 6= j
1 if i = j.
The physicists have a slick way to write this: ~ui · ~uj = δij .
Definition
I A set (~u1, ~u2, . . . , ~um) of vectors in Rn is called orthogonal ifeach is orthogonal to every other.
I The set is called orthonormal if additionally every vector is aunit vector.
Thus a set (~u1, ~u2, . . . , ~um) of vectors in Rn is orthonormal if andonly if
~ui · ~uj =
{0 if i 6= j
1 if i = j.
The physicists have a slick way to write this: ~ui · ~uj = δij .
Example
In R2, the vectors
[cos θsin θ
]and
[− sin θcos θ
]are orthonormal.
Given a general set of vectors (~v1, ~v2, . . . , ~vm) of vectors in Rn, itwas a drag to decide if they were linearly independent. Not soorthonormal sets:
Fact
a. Orthonormal sets are linearly independent.
b. An orthonormal set of n vectors in Rn is a basis for Rn.
Proof of (b).
A set of n orthonormal vectors in Rn spans an n-dimensionalsubspace of Rn, of which there is only one, Rn itself.
Given a general set of vectors (~v1, ~v2, . . . , ~vm) of vectors in Rn, itwas a drag to decide if they were linearly independent. Not soorthonormal sets:
Fact
a. Orthonormal sets are linearly independent.
b. An orthonormal set of n vectors in Rn is a basis for Rn.
Proof of (b).
A set of n orthonormal vectors in Rn spans an n-dimensionalsubspace of Rn, of which there is only one, Rn itself.
Proof of (a).
Suppose (~u1, ~u2, . . . , ~um) is an orthonormal set of vectors in Rn,and suppose there are scalars c1, c2, . . . , cm such that
~0 = c1~u1 + c2~u2 + · · ·+ cm~um.
We will show all ci must be zero.
To do this, pick one of the ~ui
and dot the above equation with it. The above implies
0 = ~0 · ~ui = c1(~u1 · ~ui ) + c2(~u2 · ~ui ) + · · ·+ cm(~um · ~ui )
Of the dot products on the right-hand side above, most of themare zero except for the ith one, which is one. Hence
0 = ci · 1.
But i was arbitrary, so all c1 = c2 = · · · = cm = 0.
Proof of (a).
Suppose (~u1, ~u2, . . . , ~um) is an orthonormal set of vectors in Rn,and suppose there are scalars c1, c2, . . . , cm such that
~0 = c1~u1 + c2~u2 + · · ·+ cm~um.
We will show all ci must be zero. To do this, pick one of the ~ui
and dot the above equation with it. The above implies
0 = ~0 · ~ui = c1(~u1 · ~ui ) + c2(~u2 · ~ui ) + · · ·+ cm(~um · ~ui )
Of the dot products on the right-hand side above, most of themare zero except for the ith one, which is one. Hence
0 = ci · 1.
But i was arbitrary, so all c1 = c2 = · · · = cm = 0.
Proof of (a).
Suppose (~u1, ~u2, . . . , ~um) is an orthonormal set of vectors in Rn,and suppose there are scalars c1, c2, . . . , cm such that
~0 = c1~u1 + c2~u2 + · · ·+ cm~um.
We will show all ci must be zero. To do this, pick one of the ~ui
and dot the above equation with it. The above implies
0 = ~0 · ~ui = c1(~u1 · ~ui ) + c2(~u2 · ~ui ) + · · ·+ cm(~um · ~ui )
Of the dot products on the right-hand side above, most of themare zero except for the ith one, which is one. Hence
0 = ci · 1.
But i was arbitrary, so all c1 = c2 = · · · = cm = 0.
Proof of (a).
Suppose (~u1, ~u2, . . . , ~um) is an orthonormal set of vectors in Rn,and suppose there are scalars c1, c2, . . . , cm such that
~0 = c1~u1 + c2~u2 + · · ·+ cm~um.
We will show all ci must be zero. To do this, pick one of the ~ui
and dot the above equation with it. The above implies
0 = ~0 · ~ui = c1(~u1 · ~ui ) + c2(~u2 · ~ui ) + · · ·+ cm(~um · ~ui )
Of the dot products on the right-hand side above, most of themare zero except for the ith one, which is one. Hence
0 = ci · 1.
But i was arbitrary, so all c1 = c2 = · · · = cm = 0.
Coordinates with respect to an orthonormal basis
Example
Let V be the subspace of R3 consisting of those vectors[x1 x2 x3
]Tsuch that x1 + x2 + x3 = 0. The set
A = (~u1, ~u2) =
1√2
1−10
,1√6
11−2
is an orthonormal basis for V . Find the coordinates of~v =
[3 4 −7
]Tin this basis.
Solution (Old way)
Form an augmented matrix and row reduce: 1/√
2 1/√
6 3−1/
√2 1/
√6 4
0 −2/√
6 −7
1 0 −1/
√2
0 1 7√
3/√
2
0 0 0
Coordinates with respect to an orthonormal basis
Example
Let V be the subspace of R3 consisting of those vectors[x1 x2 x3
]Tsuch that x1 + x2 + x3 = 0. The set
A = (~u1, ~u2) =
1√2
1−10
,1√6
11−2
is an orthonormal basis for V . Find the coordinates of~v =
[3 4 −7
]Tin this basis.
Solution (Old way)
Form an augmented matrix and row reduce: 1/√
2 1/√
6 3−1/
√2 1/
√6 4
0 −2/√
6 −7
1 0 −1/
√2
0 1 7√
3/√
2
0 0 0
Solution (New way)
Since A is a basis, we know
~v = c1~u1 + c2~u2
So dot with ~u1:
~v · ~u1 = c1(~u1 · ~u1) + c2(~u2 · ~u1)
− 1√2
= c1(1) + c2(0) = c1.
Likewise
c2 = ~v · ~u2 =21√
6=
7√
3√2
Solution (New way)
Since A is a basis, we know
~v = c1~u1 + c2~u2
So dot with ~u1:
~v · ~u1 = c1(~u1 · ~u1) + c2(~u2 · ~u1)
− 1√2
= c1(1) + c2(0) = c1.
Likewise
c2 = ~v · ~u2 =21√
6=
7√
3√2
Solution (New way)
Since A is a basis, we know
~v = c1~u1 + c2~u2
So dot with ~u1:
~v · ~u1 = c1(~u1 · ~u1) + c2(~u2 · ~u1)
− 1√2
= c1(1) + c2(0) = c1.
Likewise
c2 = ~v · ~u2 =21√
6=
7√
3√2
Solution (New way)
Since A is a basis, we know
~v = c1~u1 + c2~u2
So dot with ~u1:
~v · ~u1 = c1(~u1 · ~u1) + c2(~u2 · ~u1)
− 1√2
= c1(1) + c2(0) = c1.
Likewise
c2 = ~v · ~u2 =21√
6=
7√
3√2
Summary
FactSuppose A = (~u1, ~u2, . . . , ~um) is an orthonormal basis for asubspace V of Rn, and ~v is in V . Then
[~v ]A =
~v · ~u1
~v · ~u2...
~v · ~um
The orthogonal complement
DefinitionLet V be a subspace of Rn. The set
V⊥ = { ~w ∈ V | ~w · ~v = 0 for all v ∈ V }
is called the orthogonal complement to V .
FactV⊥ is a subspace of Rn.
FactLet (~u1, . . . , ~um) be a basis for V . A vector ~w is in V⊥ if and onlyif ~w · ~vi = 0 for each i .
The orthogonal complement
DefinitionLet V be a subspace of Rn. The set
V⊥ = { ~w ∈ V | ~w · ~v = 0 for all v ∈ V }
is called the orthogonal complement to V .
FactV⊥ is a subspace of Rn.
FactLet (~u1, . . . , ~um) be a basis for V . A vector ~w is in V⊥ if and onlyif ~w · ~vi = 0 for each i .
The orthogonal complement
DefinitionLet V be a subspace of Rn. The set
V⊥ = { ~w ∈ V | ~w · ~v = 0 for all v ∈ V }
is called the orthogonal complement to V .
FactV⊥ is a subspace of Rn.
FactLet (~u1, . . . , ~um) be a basis for V . A vector ~w is in V⊥ if and onlyif ~w · ~vi = 0 for each i .
Finding a basis for V⊥
Example
The orthonormal vectors ~u1 = 1√30
[1 2 3 4
]Tand
~u2 = 1√12
[−3 1 −1 1
]Tspan a subspace V of R4. Find a
basis for V⊥.
SolutionA vector w is in V⊥ if and only if ~w · ~u1 = ~w · ~u2 = 0. This is trueif and only if ~w is in the kernel of the matrix[
1 2 3 4−3 1 −1 1
]
[1 0 5/7 2/7
0 1 8/7 13/7
]The kernel is spanned by
[−5/7 −8/7 1 0
]Tand[
−2/7 −13/7 0 1]T
.
Finding a basis for V⊥
Example
The orthonormal vectors ~u1 = 1√30
[1 2 3 4
]Tand
~u2 = 1√12
[−3 1 −1 1
]Tspan a subspace V of R4. Find a
basis for V⊥.
SolutionA vector w is in V⊥ if and only if ~w · ~u1 = ~w · ~u2 = 0. This is trueif and only if ~w is in the kernel of the matrix[
1 2 3 4−3 1 −1 1
]
[1 0 5/7 2/7
0 1 8/7 13/7
]The kernel is spanned by
[−5/7 −8/7 1 0
]Tand[
−2/7 −13/7 0 1]T
.
Warning!
Although this process produces a basis for V⊥ no matter whatbasis for V you use (orthonormal or not), it will not necessarilyproduce an orthogonal basis for V⊥.
FactLet S be an n ×m matrix. Then
(image S)⊥ = ker(ST ).
Note that both sides of the equation are subspaces of Rn.
Proof.Let S =
[~v1 · · · ~vm
]. Let ~x be in Rn. Then
ST~x =
~v1 · ~x...
~vm · ~x
So
ST~x = ~0 ⇐⇒ ~vi · ~x = 0 for all i ⇐⇒ ~x ∈ (image S)⊥
FactLet S be an n ×m matrix. Then
(image S)⊥ = ker(ST ).
Note that both sides of the equation are subspaces of Rn.
Proof.Let S =
[~v1 · · · ~vm
]. Let ~x be in Rn. Then
ST~x =
~v1 · ~x...
~vm · ~x
So
ST~x = ~0 ⇐⇒ ~vi · ~x = 0 for all i ⇐⇒ ~x ∈ (image S)⊥
FactLet S be an n ×m matrix. Then
(image S)⊥ = ker(ST ).
Note that both sides of the equation are subspaces of Rn.
Proof.Let S =
[~v1 · · · ~vm
]. Let ~x be in Rn. Then
ST~x =
~v1 · ~x...
~vm · ~x
So
ST~x = ~0 ⇐⇒ ~vi · ~x = 0 for all i ⇐⇒ ~x ∈ (image S)⊥
Orthogonal projections
FactLet L be any line and ~x any vector. Then ~x can be decomposed as
~x = ~x‖ + ~x⊥
where ~x‖ is parallel to L and ~x⊥ is perpendicular to L.
L
~x
~x‖
~x⊥
Orthogonal projections
FactLet L be any line and ~x any vector. Then ~x can be decomposed as
~x = ~x‖ + ~x⊥
where ~x‖ is parallel to L and ~x⊥ is perpendicular to L.
L
~x
~x‖
~x⊥
Orthogonal projections
FactLet L be any line and ~x any vector. Then ~x can be decomposed as
~x = ~x‖ + ~x⊥
where ~x‖ is parallel to L and ~x⊥ is perpendicular to L.
L
~x
~x‖
~x⊥
Proof.Let ~u be any unit vector spanning L. Then ~x‖ = k~u for some k,and ~x⊥ · ~u = 0. So
~x · ~u = (~x‖ + ~x⊥) · ~u = ~x‖ · ~u + ~x⊥ · ~u = k + 0
so~x‖ = (~x · ~u)~u, ~x⊥ = ~x − ~x‖.
How do we generalize this?
Proof.Let ~u be any unit vector spanning L. Then ~x‖ = k~u for some k,and ~x⊥ · ~u = 0. So
~x · ~u = (~x‖ + ~x⊥) · ~u = ~x‖ · ~u + ~x⊥ · ~u = k + 0
so~x‖ = (~x · ~u)~u, ~x⊥ = ~x − ~x‖.
How do we generalize this?
FactConsider a subspace V of Rn and a vector ~x in Rn. Then we canwrite
~x = ~x‖ + ~x⊥,
where ~x‖ is in V and ~x⊥ is in V⊥, and this representation isunique.
Proof.Choose an orthonormal basis ~u1, . . . , ~um of V . Supposing thedecomposition ~x = ~x‖ + ~x⊥ does exist, we know
~x‖ = c1~u1 + · · ·+ cm~um,
and ~x − ~x‖ is in V⊥. In particular, ~x − ~x‖ is orthogonal to ~ui , so(~x − ~x‖) · ~ui = 0. But
(~x − ~x‖) · ~ui = ~x · ~ui − ~x‖ · ~ui = ~x · ~ui − ci
so ci = ~x · ~ui .
FactConsider a subspace V of Rn and a vector ~x in Rn. Then we canwrite
~x = ~x‖ + ~x⊥,
where ~x‖ is in V and ~x⊥ is in V⊥, and this representation isunique.
Proof.Choose an orthonormal basis ~u1, . . . , ~um of V . Supposing thedecomposition ~x = ~x‖ + ~x⊥ does exist, we know
~x‖ = c1~u1 + · · ·+ cm~um,
and ~x − ~x‖ is in V⊥. In particular, ~x − ~x‖ is orthogonal to ~ui , so(~x − ~x‖) · ~ui = 0.
But
(~x − ~x‖) · ~ui = ~x · ~ui − ~x‖ · ~ui = ~x · ~ui − ci
so ci = ~x · ~ui .
FactConsider a subspace V of Rn and a vector ~x in Rn. Then we canwrite
~x = ~x‖ + ~x⊥,
where ~x‖ is in V and ~x⊥ is in V⊥, and this representation isunique.
Proof.Choose an orthonormal basis ~u1, . . . , ~um of V . Supposing thedecomposition ~x = ~x‖ + ~x⊥ does exist, we know
~x‖ = c1~u1 + · · ·+ cm~um,
and ~x − ~x‖ is in V⊥. In particular, ~x − ~x‖ is orthogonal to ~ui , so(~x − ~x‖) · ~ui = 0. But
(~x − ~x‖) · ~ui = ~x · ~ui − ~x‖ · ~ui = ~x · ~ui − ci
so ci = ~x · ~ui .
FactIf V is a subspace of Rn with orthonormal basis ~u1, . . . , ~um, then
projV (~x) = ~x‖ = (~u1 · ~x)~u1 + (~u2 · ~x)~u2 + · · ·+ (~um · ~x)~um
Example
The orthonormal vectors ~u1 = 1√30
[1 2 3 4
]Tand
~u2 = 1√12
[−3 1 −1 1
]Tspan a subspace V of R4. Find the
projection of[1 −1 1 −1
]Tonto V .
Solution
projV (~x) = (~u1 · ~x)~u1 + (~u2 · ~x)~u2
= − 2√30
1√30
1234
− 6√12
1√12
−31−11
=
43/30
−19/30
3/10
−23/30
Example
The orthonormal vectors ~u1 = 1√30
[1 2 3 4
]Tand
~u2 = 1√12
[−3 1 −1 1
]Tspan a subspace V of R4. Find the
projection of[1 −1 1 −1
]Tonto V .
Solution
projV (~x) = (~u1 · ~x)~u1 + (~u2 · ~x)~u2
= − 2√30
1√30
1234
− 6√12
1√12
−31−11
=
43/30
−19/30
3/10
−23/30
Geometric consequences
Geometric consequences
FactGiven two vectors ~x and ~y in Rn, we have
‖~x + ~y‖2 = ‖~x‖2 + ‖~y‖2
if and only if ~x · ~y = 0.
Proof.FOIL it out!
‖~x + ~y‖2 = (~x + ~y) · (~x + ~y) = ~x · ~x + ~x · ~y + ~y · ~x + ~y · ~y= ‖~x‖2 + 2(~x · ~y) + ‖~y‖2
Geometric consequences
FactGiven two vectors ~x and ~y in Rn, we have
‖~x + ~y‖2 = ‖~x‖2 + ‖~y‖2
if and only if ~x · ~y = 0.
Proof.FOIL it out!
‖~x + ~y‖2 = (~x + ~y) · (~x + ~y) = ~x · ~x + ~x · ~y + ~y · ~x + ~y · ~y= ‖~x‖2 + 2(~x · ~y) + ‖~y‖2
FactLet V be a subspace of Rn and ~x in Rn. Then
‖projV ~x‖ ≤ ‖~x‖
and equality holds if and only if ~x is in V .
Proof.Use the Pythagorean theorem:
‖~x‖2 =∥∥∥~x‖ + ~x⊥
∥∥∥2=
∥∥∥~x‖∥∥∥2
+∥∥∥~x⊥
∥∥∥2≥
∥∥∥~x‖∥∥∥2
.
FactLet V be a subspace of Rn and ~x in Rn. Then
‖projV ~x‖ ≤ ‖~x‖
and equality holds if and only if ~x is in V .
Proof.Use the Pythagorean theorem:
‖~x‖2 =∥∥∥~x‖ + ~x⊥
∥∥∥2=
∥∥∥~x‖∥∥∥2
+∥∥∥~x⊥
∥∥∥2≥
∥∥∥~x‖∥∥∥2
.
Let ~x and ~y be vectors and project ~x onto the subspace V spannedby ~u = ~y
‖y‖ . Then the previous fact applied to ~x and V gives
‖~x‖ ≥ ‖projV ~x‖ = ‖(~x · ~u)~u‖ =
∥∥∥∥(~x ·
~y
‖~y‖
)~y
‖~y‖
∥∥∥∥ =|~x · ~y |‖~y‖
Fact (The Cauchy-Schwarz inequality)
If ~x and ~y are vectors in Rn, then
|~x · ~y | ≤ ‖~x‖ ‖~y‖
with equality if and only if the vectors are parallel.
Let ~x and ~y be vectors and project ~x onto the subspace V spannedby ~u = ~y
‖y‖ . Then the previous fact applied to ~x and V gives
‖~x‖ ≥ ‖projV ~x‖ = ‖(~x · ~u)~u‖ =
∥∥∥∥(~x ·
~y
‖~y‖
)~y
‖~y‖
∥∥∥∥ =|~x · ~y |‖~y‖
Fact (The Cauchy-Schwarz inequality)
If ~x and ~y are vectors in Rn, then
|~x · ~y | ≤ ‖~x‖ ‖~y‖
with equality if and only if the vectors are parallel.
The angle between two vectors
In Math 21a we learned that in R2
~x · ~y = ‖~x‖ ‖~y‖ cos θ,
where θ is the angle between ~x and ~y .We can now define for a pair of vectors in any Rn the anglebetween them, by the formula
cos θ =~x · ~y‖~x‖ ‖y‖
The Cauchy-Schwarz inequality says that θ exists.
The angle between two vectors
In Math 21a we learned that in R2
~x · ~y = ‖~x‖ ‖~y‖ cos θ,
where θ is the angle between ~x and ~y .We can now define for a pair of vectors in any Rn the anglebetween them, by the formula
cos θ =~x · ~y‖~x‖ ‖y‖
The Cauchy-Schwarz inequality says that θ exists.
Top Related