Presentation is loading. Please wait.

Presentation is loading. Please wait.

Signal , Weight Vector Spaces and Linear Transformations

Similar presentations

Presentation on theme: "Signal , Weight Vector Spaces and Linear Transformations"โ€” Presentation transcript:

1 Signal , Weight Vector Spaces and Linear Transformations

2 Notation Vectors in ร‚n. Generalized Vectors. ๐ฑ ๐ฑ=๐‘Žnxn+โ€ฆ+๐‘Ž1x+๐‘Ž0
2 n = ๐ฑ ๐ฑ=๐‘Žnxn+โ€ฆ+๐‘Ž1x+๐‘Ž0 ๐ฑ= ๐‘Ž1,1 ๐‘Ž1,2 ๐‘Ž2,1 ๐‘Ž2,2

3 Vector Space 1. An operation called vector addition is defined such that if xย รŽ X and y รŽ X then x + y รŽ X. 2. x + y = y + x 3. (x + y) + z = x + (y + z) 4. There is a unique vector 0ย รŽ X, called the zero vector, such that x + 0 = x for all xย รŽ X. 5. For each vector there is a unique vector in X, to be called (-x ), such that x + (-x ) = 0 .

4 Vector Space (Cont.) 6. An operation, called multiplication, is defined such that for all scalars a รŽ F, and all vectors xย  รŽ X, a x ย รŽ X. 7. For any xย  รŽ X , 1x = x (for scalar 1). 8. For any two scalars a รŽ F and b รŽ F, and any xย รŽ X, aย (bx)ย =ย (a b) x . 9. (aย + b) xย =ย a x + b x . 10. a (x + y) ย =ย a x + a y

5 Examples (Decision Boundaries)
Is the p2, p3 plane a vector space?

6 Examples (Decision Boundaries)
Is the line p1 + 2p2 - 2 = 0 a vector space?

7 Other Vector Spaces Polynomials of degree 2 or less. ๐ถ[0,1]:
Continuous functions in the interval [0,1]. ๐œ’=sinโก(๐‘ก) ๐’ด=๐‘’โˆ’2๐‘ก

8 is a set of linearly independent vectors.
Linear Independence If implies that each then is a set of linearly independent vectors.

9 Therefore the vectors are independent.
Example Let This can only be true if Therefore the vectors are independent.

10 Problem P5.3(Gramian) i. ๐ฑ1= 1 1 1 ๐ฑ2= 1 0 1 ๐ฑ3= 1 2 1
= โˆ’ =โˆ’2+0+2=0 G= (๐ฑ1,๐ฑ1) (๐ฑ1,๐ฑ2) (๐ฑ1,๐ฑ3) (๐ฑ2,๐ฑ1) (๐ฑ2,๐ฑ2) (๐ฑ2,๐ฑ3) (๐ฑ3,๐ฑ1) (๐ฑ3,๐ฑ2) (๐ฑ3,๐ฑ3) = = โˆ’ =24โˆ’8โˆ’16=0

11 Problem P5.3(Gramian)(cont.)
๐ข๐ข๐ข. ๐ฑ1= ๐ฑ2= ๐ฑ3= G= (๐ฑ1,๐ฑ1) (๐ฑ1,๐ฑ2) (๐ฑ1,๐ฑ3) (๐ฑ2,๐ฑ1) (๐ฑ2,๐ฑ2) (๐ฑ2,๐ฑ3) (๐ฑ3,๐ฑ1) (๐ฑ3,๐ฑ2) (๐ฑ3,๐ฑ3) = = โˆ’ =48โˆ’18โˆ’30=0 2 ๐ฑ1- ๐ฑ2 โˆ’๐ฑ3=0 G= =4โ‰ 0

12 Gramian ๐=[๐ฑ1,๐ฑ2,โ‹ฏ,๐ฑ๐‘›] G ๐ฑ1,๐ฑ2,โ‹ฏ,๐ฑ๐‘› = ๐Tร—๐ = (๐ฑ1,๐ฑ1) (๐ฑ1,๐ฑ2) (๐ฑ2,๐ฑ1) (๐ฑ2,๐ฑ2) โ‹ฏ (๐ฑ1,๐ฑ๐‘›) (๐ฑ2,๐ฑ๐‘›) โ‹ฎ โ‹ฑ โ‹ฎ (๐ฑ๐‘›,๐ฑ1) (๐ฑ๐‘›,๐ฑ2) โ‹ฏ (๐ฑ๐‘›,๐ฑ๐‘›) G ๐ฑ1,๐ฑ2,โ‹ฏ,๐ฑ๐‘› โ‰ 0 iff ๐ฑ1,๐ฑ2,โ‹ฏ,๐ฑ๐‘› are linearly independent G ๐ฑ1,๐ฑ2,โ‹ฏ,๐ฑ๐‘› =0 iff ๐ฑ1,๐ฑ2,โ‹ฏ,๐ฑ๐‘› are linearly dependent

13 Spanning a Space A subset spans a space if every vector in the space can be written as a linear combination of the vectors in the subspace.

14 Basis Vectors A set of basis vectors for the space X is a set of vectors which spans X and is linearly independent. The dimension of a vector space, Dim(X), is equal to the number of vectors in the basis set. Let X be a finite dimensional vector space, then every basis set of X has the same number of elements.

15 Example Polynomials of degree 2 or less. Basis A: Basis B:
(Any three linearly independent vectors in the space will work.) How can you represent the vector x = 1+2t using both basis sets? ๐œ’= ๐ฎ1+๐ฎ2 2 +๐ฎ2โˆ’๐ฎ1= 3๐ฎ2โˆ’๐ฎ1 2 ๐ฎ1+๐ฎ2=2,๐ฎ2โˆ’๐ฎ1=2t

16 Inner Product / Norm A scalar function of vectors x and y can be defined as an inner product, (x , y ), provided the following are satisfied (for real inner products): (x , y ) = (y , x ) . (x , ay1+by2) = a( x , y1) + b( x , y2) . (x , x) โ‰ง 0 , where equality holds iff x = 0 . A scalar function of a vector x is called a norm, ||x || provided the following are satisfied: ||x || โ‰ง0 . ||x || = 0 iff x = 0 . ||a x || = |a| ||x || for scalar a . ||x + y || โ‰ค ||x || + ||y || .

17 Example Angle: Standard Euclidean Inner Product
Standard Euclidean Norm ||x || = (x , x)1/2 ||x|| = (xTx)1/2 = x12 + x xn2 ๐ถ[0,1] Inner Product(see Problem P5.6): ๐‘ฅ, ๐‘ฆ = 0 1 ๐‘ฅ ๐‘ก ๐‘ฆ ๐‘ก ๐‘‘๐‘ก cosq = (x , y) ||x || ||y || Angle:

18 Problem P5.6 An inner product must satisfy the following properties.
1. ๐‘ฅ,๐‘ฆ =(๐‘ฆ,๐‘ฅ) ๐‘ฅ,๐‘ฆ = 0 1 ๐‘ฅ ๐‘ก ๐‘ฆ ๐‘ก ๐‘‘๐‘ก = 0 1 ๐‘ฆ ๐‘ก ๐‘ฅ ๐‘ก ๐‘‘๐‘ก =(๐‘ฆ,๐‘ฅ) 2. ๐‘ฅ,๐‘Ž๐‘ฆ1+๐‘๐‘ฆ2 =๐‘Ž ๐‘ฅ,๐‘ฆ1 +๐‘(๐‘ฅ,๐‘ฆ2) ๐‘ฅ,๐‘Ž๐‘ฆ1+๐‘๐‘ฆ2 = 0 1 ๐‘ฅ ๐‘ก ๐‘Ž๐‘ฆ1 ๐‘ก +๐‘๐‘ฆ2 ๐‘ก ๐‘‘๐‘ก = 0 1 ๐‘ฅ ๐‘ก ๐‘Ž๐‘ฆ1 ๐‘ก ๐‘‘๐‘ก ๐‘ฅ ๐‘ก ๐‘๐‘ฆ2 ๐‘ก ๐‘‘๐‘ก= ๐‘Ž ๐‘ฅ,๐‘ฆ1 +๐‘(๐‘ฅ,๐‘ฆ2)

19 Problem P5.6(cont.) 3. ๐‘ฅ,๐‘ฅ โ‰ฅ0,where equality holds iff x is the zero vector ๐‘ฅ,๐‘ฅ = 0 1 ๐‘ฅ ๐‘ก ๐‘ฅ ๐‘ก ๐‘‘๐‘ก 0 1 ๐‘ฅ ๐‘ก 2๐‘‘๐‘ก โ‰ฅ0 Equality holds here only if ๐‘ฅ ๐‘ก =0 for 0โ‰ค๐‘กโ‰ค1 ,which is the zero vector

20 Two vectors x , y รŽX are orthogonal if (x , y) = 0 .
Orthogonality Two vectors x , y รŽX are orthogonal if (x , y) = 0 . Example Any vector in the p2,p3 plane is orthogonal to the weight vector.

21 Gram-Schmidt Orthogonalization
Independent Vectors Orthogonal Vectors ๐‘ฆ1,๐‘ฆ2,โ€ฆ,๐‘ฆ๐‘› ๐‘ฃ1,๐‘ฃ2,โ€ฆ,๐‘ฃ๐‘› Step 1: Set first orthogonal vector to first independent vector. ๐‘ฃ1=๐‘ฆ1 Step 2: Subtract the portion of y2 that is in the direction of v1. ๐‘ฃ2=๐‘ฆ2โˆ’๐‘Ž๐‘ฃ1 Where a is chosen so that v2 is orthogonal to v1: ๐‘ฃ1,๐‘ฃ2 = ๐‘ฃ1,๐‘ฆ2โˆ’๐‘Ž๐‘ฃ1 = ๐‘ฃ1,๐‘ฆ2 โˆ’๐‘Ž ๐‘ฃ1,๐‘ฃ1 =0 ๐‘Ž= (๐‘ฃ1,๐‘ฆ2) (๐‘ฃ1,๐‘ฃ1)

22 Gram-Schmidt (Cont.) Projection of y2 on v1: (๐‘ฃ1,๐‘ฆ2) (๐‘ฃ1,๐‘ฃ1) ๐‘ฃ1
Step k: Subtract the portion of yk that is in the direction of all previous vi . ๐‘ฃ๐‘˜=๐‘ฆ๐‘˜โˆ’ ๐‘–=1 ๐‘˜โˆ’1 (๐‘ฃ๐‘–,๐‘ฆ๐‘˜) (๐‘ฃ๐‘–,๐‘ฃ๐‘–) ๐‘ฃ๐‘–

23 Example ๐ฒ1= ,๐ฒ2=

24 Example Step 1๏ผš ๐ฏ1=๐ฒ1= 2 1

25 Example (Cont.) Step 2.

26 Vector Expansion If a vector space X has a basis set {v1, v2, ..., vn}, then any x รŽX has a unique vector expansion: ๐œ’= ๐‘–=1 ๐‘› ๐‘ฅ๐‘–๐‘ฃ๐‘– =๐‘ฅ1๐‘ฃ1+๐‘ฅ2๐‘ฃ2+โ€ฆ+๐‘ฅ๐‘›๐‘ฃ๐‘› If the basis vectors are orthogonal, and we take the inner product of vj and x : ๐‘ฃj,๐œ’ = ๐‘ฃj, ๐‘–=1 ๐‘› ๐‘ฅi๐‘ฃi = ๐‘–=1 ๐‘› ๐‘ฅi(๐‘ฃj,๐‘ฃi) =๐‘ฅj(๐‘ฃj,๐‘ฃj) Therefore the coefficients of the expansion can be computed: ๐‘ฅj= (๐‘ฃj,๐œ’) (๐‘ฃj,๐‘ฃj)

27 Column of Numbers The vector expansion provides a meaning for
writing a vector as a column of numbers. ๐œ’= ๐‘–=1 ๐‘› ๐‘ฅ๐‘–๐‘ฃ๐‘– =๐‘ฅ1๐‘ฃ1+๐‘ฅ2๐‘ฃ2+โ€ฆ+๐‘ฅ๐‘›๐‘ฃ๐‘› ๐‘ฅ= ๐‘ฅ1 ๐‘ฅ2 โ‹ฎ ๐‘ฅ๐‘› To interpret x, we need to know what basis was used for the expansion.

28 Reciprocal Basis Vectors
Definition of reciprocal basis vectors, ri: ๐‘Ÿ๐‘–,๐‘ฃ๐‘— = ๐‘–โ‰ ๐‘— = ๐‘–=๐‘— where the basis vectors are {v1, v2, ..., vn}, and the reciprocal basis vectors are {r1, r2, ..., rn}. For vectors in ร‚n we can use the following inner product: ๐‘Ÿ๐‘–,๐‘ฃ ๐‘— =๐‘Ÿ ๐‘– ๐‘‡๐‘ฃ๐‘— ๅ€’ๆ•ธๅŸบๅบ• Therefore, the equations for the reciprocal basis vectors become:

29 Vector Expansion ๐œ’=๐‘ฅ1๐‘ฃ1+๐‘ฅ2๐‘ฃ2+โ€ฆ+๐‘ฅ๐‘›๐‘ฃ๐‘›
Take the inner product of the first reciprocal basis vector with the vector to be expanded: ๐‘Ÿ1,๐œ’ =๐‘ฅ1 ๐‘Ÿ1,๐‘ฃ1 +๐‘ฅ2 ๐‘Ÿ2,๐‘ฃ2 +โ€ฆ+๐‘ฅ๐‘› ๐‘Ÿ๐‘›,๐‘ฃ๐‘› By definition of the reciprocal basis vectors: ๐‘Ÿ1,๐‘ฃ2 = ๐‘Ÿ1,๐‘ฃ3 =โ€ฆ= ๐‘Ÿ1,๐‘ฃ๐‘› =0 ๐‘Ÿ1,๐‘ฃ1 =1 Therefore, the first coefficient in the expansion is: ๐‘ฅ1=(๐‘Ÿ1,๐œ’) In general, we then have (even for nonorthogonal basis vectors): ๐‘ฅ๐‘—=(๐‘Ÿ๐‘— ,๐œ’)

30 Example ๐‘ฃ1๐‘ = ,๐‘ฃ2๐‘ = 1 2 Basis Vectors: ๐‘ฅ๐‘ = Vector to Expand:

31 Example (Cont.) Reciprocal Basis Vectors: Expansion Coefficients:
๐ซ1= โˆ’ ๐ซ2= โˆ’ Expansion Coefficients: Matrix Form: ๐ฑ1๐‘ฃ=๐ซ1๐‘ก๐ฑ๐‘ = โˆ’ =โˆ’ 1 2 ๐ฑ๐‘ฃ=๐‘๐‘‡๐ฑ๐‘ =๐โˆ’1๐ฑs = โˆ’ 1 3 โˆ’ = โˆ’ ๐ฑ2๐‘ฃ=๐ซ2๐‘ก๐ฑ๐‘ = โˆ’ =1

32 Example (Cont.) ฯ‡=0๐‘ 1+ 3 2 ๐‘ 2=โˆ’ 1 2 ๐‘ฃ1+1๐‘ฃ2
The interpretation of the column of numbers depends on the basis set used for the expansion.

33 Linear Transformation
๐‘ฃ1 ๐‘ฃ2 = ๐‘ 1 ๐‘  = ๐‘ 1 ๐‘ 2 ๐ ๐‘ 1 ๐‘ 2 =๐โˆ’1 v1 ๐‘ฃ2 3 2 ๐‘ 2= ๐‘ 1 ๐‘  = ๐‘ฃ1 ๐‘ฃ2 ๐โˆ’ = ๐‘ฃ1 ๐‘ฃ โˆ’1 3 โˆ’ = ๐‘ฃ1 ๐‘ฃ2 โˆ’ = โˆ’ 1 2 ๐‘ฃ1+๐‘ฃ2 ๐ฑ๐‘ฃ=๐โˆ’1๐ฑ๐‘  . This operation, call a change of basis

Download ppt "Signal , Weight Vector Spaces and Linear Transformations"

Similar presentations

Ads by Google