# Signal , Weight Vector Spaces and Linear Transformations

## Presentation on theme: "Signal , Weight Vector Spaces and Linear Transformations"â Presentation transcript:

Signal , Weight Vector Spaces and Linear Transformations

Notation Vectors in Ãn. Generalized Vectors. đą đą=đnxn+âĻ+đ1x+đ0
2 n = đą đą=đnxn+âĻ+đ1x+đ0 đą= đ1,1 đ1,2 đ2,1 đ2,2

Vector Space 1. An operation called vector addition is defined such that if xÂ Ã X and y Ã X then x + y Ã X. 2. x + y = y + x 3. (x + y) + z = x + (y + z) 4. There is a unique vector 0Â Ã X, called the zero vector, such that x + 0 = x for all xÂ Ã X. 5. For each vector there is a unique vector in X, to be called (-x ), such that x + (-x ) = 0 .

Vector Space (Cont.) 6. An operation, called multiplication, is defined such that for all scalars a Ã F, and all vectors xÂ  Ã X, a x Â Ã X. 7. For any xÂ  Ã X , 1x = x (for scalar 1). 8. For any two scalars a Ã F and b Ã F, and any xÂ Ã X, aÂ (bx)Â =Â (a b) x . 9. (aÂ + b) xÂ =Â a x + b x . 10. a (x + y) Â =Â a x + a y

Examples (Decision Boundaries)
Is the p2, p3 plane a vector space?

Examples (Decision Boundaries)
Is the line p1 + 2p2 - 2 = 0 a vector space?

Other Vector Spaces Polynomials of degree 2 or less. đļ[0,1]:
Continuous functions in the interval [0,1]. đ=sinâĄ(đĄ) đ´=đâ2đĄ

is a set of linearly independent vectors.
Linear Independence If implies that each then is a set of linearly independent vectors.

Therefore the vectors are independent.
Example Let This can only be true if Therefore the vectors are independent.

Problem P5.3(Gramian) i. đą1= 1 1 1 đą2= 1 0 1 đą3= 1 2 1
= â =â2+0+2=0 G= (đą1,đą1) (đą1,đą2) (đą1,đą3) (đą2,đą1) (đą2,đą2) (đą2,đą3) (đą3,đą1) (đą3,đą2) (đą3,đą3) = = â =24â8â16=0

Problem P5.3(Gramian)(cont.)
đĸđĸđĸ. đą1= đą2= đą3= G= (đą1,đą1) (đą1,đą2) (đą1,đą3) (đą2,đą1) (đą2,đą2) (đą2,đą3) (đą3,đą1) (đą3,đą2) (đą3,đą3) = = â =48â18â30=0 2 đą1- đą2 âđą3=0 G= =4â 0

Gramian đ=[đą1,đą2,â¯,đąđ] G đą1,đą2,â¯,đąđ = đTÃđ = (đą1,đą1) (đą1,đą2) (đą2,đą1) (đą2,đą2) â¯ (đą1,đąđ) (đą2,đąđ) âŽ âą âŽ (đąđ,đą1) (đąđ,đą2) â¯ (đąđ,đąđ) G đą1,đą2,â¯,đąđ â 0 iff đą1,đą2,â¯,đąđ are linearly independent G đą1,đą2,â¯,đąđ =0 iff đą1,đą2,â¯,đąđ are linearly dependent

Spanning a Space A subset spans a space if every vector in the space can be written as a linear combination of the vectors in the subspace.

Basis Vectors A set of basis vectors for the space X is a set of vectors which spans X and is linearly independent. The dimension of a vector space, Dim(X), is equal to the number of vectors in the basis set. Let X be a finite dimensional vector space, then every basis set of X has the same number of elements.

Example Polynomials of degree 2 or less. Basis A: Basis B:
(Any three linearly independent vectors in the space will work.) How can you represent the vector x = 1+2t using both basis sets? đ= đŽ1+đŽ2 2 +đŽ2âđŽ1= 3đŽ2âđŽ1 2 đŽ1+đŽ2=2,đŽ2âđŽ1=2t

Inner Product / Norm A scalar function of vectors x and y can be defined as an inner product, (x , y ), provided the following are satisfied (for real inner products): (x , y ) = (y , x ) . (x , ay1+by2) = a( x , y1) + b( x , y2) . (x , x) â§ 0 , where equality holds iff x = 0 . A scalar function of a vector x is called a norm, ||x || provided the following are satisfied: ||x || â§0 . ||x || = 0 iff x = 0 . ||a x || = |a| ||x || for scalar a . ||x + y || â¤ ||x || + ||y || .

Example Angle: Standard Euclidean Inner Product
Standard Euclidean Norm ||x || = (x , x)1/2 ||x|| = (xTx)1/2 = x12 + x xn2 đļ[0,1] Inner Product(see Problem P5.6): đĨ, đĻ = 0 1 đĨ đĄ đĻ đĄ đđĄ cosq = (x , y) ||x || ||y || Angle:

Problem P5.6 An inner product must satisfy the following properties.
1. đĨ,đĻ =(đĻ,đĨ) đĨ,đĻ = 0 1 đĨ đĄ đĻ đĄ đđĄ = 0 1 đĻ đĄ đĨ đĄ đđĄ =(đĻ,đĨ) 2. đĨ,đđĻ1+đđĻ2 =đ đĨ,đĻ1 +đ(đĨ,đĻ2) đĨ,đđĻ1+đđĻ2 = 0 1 đĨ đĄ đđĻ1 đĄ +đđĻ2 đĄ đđĄ = 0 1 đĨ đĄ đđĻ1 đĄ đđĄ đĨ đĄ đđĻ2 đĄ đđĄ= đ đĨ,đĻ1 +đ(đĨ,đĻ2)

Problem P5.6(cont.) 3. đĨ,đĨ âĨ0,where equality holds iff x is the zero vector đĨ,đĨ = 0 1 đĨ đĄ đĨ đĄ đđĄ 0 1 đĨ đĄ 2đđĄ âĨ0 Equality holds here only if đĨ đĄ =0 for 0â¤đĄâ¤1 ,which is the zero vector

Two vectors x , y ÃX are orthogonal if (x , y) = 0 .
Orthogonality Two vectors x , y ÃX are orthogonal if (x , y) = 0 . Example Any vector in the p2,p3 plane is orthogonal to the weight vector.

Gram-Schmidt Orthogonalization
Independent Vectors Orthogonal Vectors đĻ1,đĻ2,âĻ,đĻđ đŖ1,đŖ2,âĻ,đŖđ Step 1: Set first orthogonal vector to first independent vector. đŖ1=đĻ1 Step 2: Subtract the portion of y2 that is in the direction of v1. đŖ2=đĻ2âđđŖ1 Where a is chosen so that v2 is orthogonal to v1: đŖ1,đŖ2 = đŖ1,đĻ2âđđŖ1 = đŖ1,đĻ2 âđ đŖ1,đŖ1 =0 đ= (đŖ1,đĻ2) (đŖ1,đŖ1)

Gram-Schmidt (Cont.) Projection of y2 on v1: (đŖ1,đĻ2) (đŖ1,đŖ1) đŖ1
Step k: Subtract the portion of yk that is in the direction of all previous vi . đŖđ=đĻđâ đ=1 đâ1 (đŖđ,đĻđ) (đŖđ,đŖđ) đŖđ

Example đ˛1= ,đ˛2=

Example Step 1īŧ đ¯1=đ˛1= 2 1

Example (Cont.) Step 2.

Vector Expansion If a vector space X has a basis set {v1, v2, ..., vn}, then any x ÃX has a unique vector expansion: đ= đ=1 đ đĨđđŖđ =đĨ1đŖ1+đĨ2đŖ2+âĻ+đĨđđŖđ If the basis vectors are orthogonal, and we take the inner product of vj and x : đŖj,đ = đŖj, đ=1 đ đĨiđŖi = đ=1 đ đĨi(đŖj,đŖi) =đĨj(đŖj,đŖj) Therefore the coefficients of the expansion can be computed: đĨj= (đŖj,đ) (đŖj,đŖj)

Column of Numbers The vector expansion provides a meaning for
writing a vector as a column of numbers. đ= đ=1 đ đĨđđŖđ =đĨ1đŖ1+đĨ2đŖ2+âĻ+đĨđđŖđ đĨ= đĨ1 đĨ2 âŽ đĨđ To interpret x, we need to know what basis was used for the expansion.

Reciprocal Basis Vectors
Definition of reciprocal basis vectors, ri: đđ,đŖđ = đâ đ = đ=đ where the basis vectors are {v1, v2, ..., vn}, and the reciprocal basis vectors are {r1, r2, ..., rn}. For vectors in Ãn we can use the following inner product: đđ,đŖ đ =đ đ đđŖđ åæ¸åēåē Therefore, the equations for the reciprocal basis vectors become:

Vector Expansion đ=đĨ1đŖ1+đĨ2đŖ2+âĻ+đĨđđŖđ
Take the inner product of the first reciprocal basis vector with the vector to be expanded: đ1,đ =đĨ1 đ1,đŖ1 +đĨ2 đ2,đŖ2 +âĻ+đĨđ đđ,đŖđ By definition of the reciprocal basis vectors: đ1,đŖ2 = đ1,đŖ3 =âĻ= đ1,đŖđ =0 đ1,đŖ1 =1 Therefore, the first coefficient in the expansion is: đĨ1=(đ1,đ) In general, we then have (even for nonorthogonal basis vectors): đĨđ=(đđ ,đ)

Example đŖ1đ = ,đŖ2đ = 1 2 Basis Vectors: đĨđ = Vector to Expand:

Example (Cont.) Reciprocal Basis Vectors: Expansion Coefficients:
đĢ1= â đĢ2= â Expansion Coefficients: Matrix Form: đą1đŖ=đĢ1đĄđąđ = â =â 1 2 đąđŖ=đđđąđ =đâ1đąs = â 1 3 â = â đą2đŖ=đĢ2đĄđąđ = â =1

Example (Cont.) Ī=0đ 1+ 3 2 đ 2=â 1 2 đŖ1+1đŖ2
The interpretation of the column of numbers depends on the basis set used for the expansion.

Linear Transformation
đŖ1 đŖ2 = đ 1 đ  = đ 1 đ 2 đ đ 1 đ 2 =đâ1 v1 đŖ2 3 2 đ 2= đ 1 đ  = đŖ1 đŖ2 đâ = đŖ1 đŖ â1 3 â = đŖ1 đŖ2 â = â 1 2 đŖ1+đŖ2 đąđŖ=đâ1đąđ  . This operation, call a change of basis