1. Lecture 7 Linear time invariant systems
Stochastic processes
Lecture 7
Linear time invariant systems

2. Random process

3. 1st order Distribution & density function
First-order distribution First-order density function

4. 2end order Distribution & density function
2end order distribution 2end order density function

5. EXPECTATIONS
Expected value
The autocorrelation

6. Some random processes Single pulse Multiple pulses
Periodic Random Processes
The Gaussian Process
The Poisson Process
Bernoulli and Binomial Processes
The Random Walk Wiener Processes
The Markov Process

7. Recap: Power spectrum density

8. Power spectrum density
Since the integral of the squared absolute Fourier transform contains the full power of the signal it is a density function.
So the power spectral density of a random process is:
Due to absolute factor the PSD is always real
𝑆𝑥𝑥 𝑓 = 𝑙𝑖𝑚 𝑇→∞ ⁡𝐸 −𝑇 𝑇 𝑠 𝑡 𝑒 −𝑗2𝜋𝑓𝑡 𝑑𝑡 2 2𝑇

9. Power spectrum density
The PSD is a density function.
In the case of the random process the PSD is the density function of the random process and not necessarily the frequency spectrum of a single realization.
Example
A random process is defined as
Where ωr is a unifom distributed random variable wiht a range from 0-π
What is the PSD for the process and
The power sepctrum for a single realization
X 𝑡 =sin⁡( 𝜔 𝑟 𝑡)

10. Properties of the PSD Sxx(f) is real and nonnegative
The average power in X(t) is given by:
𝐸 𝑋 2 (𝑡) =𝑅𝑥𝑥 0 = −∞ ∞ 𝑆𝑥𝑥 𝑓 𝑑𝑓
If X(t) is real Rxx(τ) and Sxx(f) are also even
𝑆𝑥𝑥 −𝑓 =𝑆𝑥𝑥 𝑓
If X(t) has periodic components Sxx(f)has impulses
Independent on phase

11. Wiener-Khinchin 1
If the X(t) is stationary in the wide-sense the PSD is the Fourier transform of the Autocorrelation

12. Wiener-Khinchin Two method for estimation of the PSD
Fourier Transform
|X(f)|2
X(t)
Sxx(f)
Fourier Transform
Autocorrelation

13. The inverse Fourier Transform of the PSD
Since the PSD is the Fourier transformed autocorrelation
The inverse Fourier transform of the PSD is the autocorrelation

14. Cross spectral densities
If X(t) and Y(t) are two jointly wide-sense stationary processes, is the Cross spectral densities Or

15. Properties of Cross spectral densities
Since is
Syx(f) is not necessary real
If X(t) and Y(t) are orthogonal
Sxy(f)=0
If X(t) and Y(t) are independent
Sxy(f)=E[X(t)] E[Y(t)] δ(f)

16. Cross spectral densities example
1 Hz Sinus curves in white noise
Where w(t) is Gaussian noise
𝑋 𝑡 = sin 2𝜋 𝑡 +3 𝑤(𝑡)
𝑌 𝑡 = sin 2𝜋 𝑡+ 𝜋 𝑤(𝑡)

17. The periodogram The estimate of the PSD
The PSD can be estimate from the autocorrelation
Or directly from the signal
𝑆𝑥𝑥 ω = 𝑚=−𝑁+1 𝑁−1 𝑅𝑥𝑥 [𝑚] 𝑒 −𝑗ω𝑚
𝑆𝑥𝑥 ω = 1 𝑁 𝑛=0 𝑁−1 𝑥 [𝑛] 𝑒 −𝑗ω𝑛 2

18. Bias in the estimates of the autocorrelation
𝑅𝑥𝑥 𝑚 = 𝑛=0 𝑁− 𝑚 −1 𝑥 𝑛 𝑥[𝑛+𝑚]

19. Variance in the PSD
The variance of the periodogram is estimated to the power of two of PSD
𝑉𝑎𝑟 𝑆𝑥𝑥 𝜔 = 𝑆𝑥𝑥(𝜔) 2

20. Averaging Divide the signal into K segments of M length
𝑥𝑖=𝑥 𝑖−1 𝑀+1:𝑖 𝑀 ≤𝑖≤𝐾
Calculate the periodogram of each segment
𝑆𝑖𝑥𝑥 ω = 1 𝑀 𝑛=0 𝑀−1 𝑥 𝑖[𝑛] 𝑒 −𝑗ω𝑛 2
Calculate the average periodogram
𝑆 𝑥𝑥[ω]= 1 𝐾 𝑖=0 𝐾 𝑆𝑖𝑥𝑥[ω]

21. Illustrations of Averaging

22. PSD units Typical units: Electrical measurements: V2/Hz or dB V/Hz
Sound: Pa2/Hz or dB/Hz
How to calculate dB I a power spectrum:
PSDdB(f) = 10 log10 { PSD(f)  } .

23. Agenda (Lec. 7) Recap: Linear time invariant systems
Stochastic signals and LTI systems
Mean Value function
Mean square value
Cross correlation function between input and output
Autocorrelation function and spectrum output
Filter examples
Intro to system identification

24. Focus continuous signals and system
Continuous signal: Discrete signal:

25. Systems

26. Recap: Linear time invariant systems (LTI)
What is a Linear system:
The system applies to superposition 1 2 3 4 5 6 8 10 12 14 16 18 20
Linear system
x(t)
y(t) 1 2 3 4 5
-20
-15
-10 -5 10 15 20 25
Nonlinear systems
x(t)
y(t)
x[n] Ö
20 log(x[n])

27. Recap: Linear time invariant systems (LTI)
A time invariant systems is independent on explicit time
(The coefficient are independent on time)
That means
If: y2(t)=f[x1(t)]
Then: y2(t+t0)=f[x1(t+t0)]
The same to Day tomorrow and in 1000 years
70 years
45 years
20 years
A non Time invariant

28. Examples A linear system A nonlinear system A time invariant system
y(t)=3 x(t)
A nonlinear system
y(t)=3 x(t)2
A time invariant system
A time variant system
y(t)=3t x(t)

29. The impulse response The output of a system if Dirac delta is input

30. Convolution
The output of LTI system can be determined by the convoluting the input with the impulse response

31. Fourier transform of the impulse response
The Transfer function (System function) is the Fourier transformed impulse response
The impulse response can be determined from the Transfer function with the invers Fourier transform

32. Fourier transform of LTI systems
Convolution corresponds to multiplication in the frequency domain
Time domain * =
Frequency domain x =

33. Causal systems
Independent on the future signal

34. Stochastic signals and LTI systems
Estimation of the output from a LTI system when the input is a stochastic process
Α is a delay factor like τ

35. Statistical estimates of output
The specific distribution function fX(x,t) is difficult to estimate. Therefor we stick to
Mean
Autocorrelation
PSD
Mean square value.

36. Expected Value of Y(t) (1/2)
How do we estimate the mean of the output?
𝐸 𝑌 𝑡 =𝐸 −∞ ∞ 𝑋 𝑡−𝛼 ℎ 𝛼 𝑑𝛼
𝑌(𝑡)= −∞ ∞ 𝑋 𝑡−𝛼 ℎ 𝛼 𝑑𝛼
𝐸 𝑌 𝑡 = −∞ ∞ 𝐸 𝑋 𝑡−𝛼 ℎ 𝛼 𝑑𝛼
If mean of x(t) is defined as mx(t)
𝐸 𝑌 𝑡 = −∞ ∞ 𝑚𝑥(𝑡−𝛼)ℎ 𝛼 𝑑𝛼

37. Expected Value of Y(t) (2/2)
If x(t) is wide sense stationary
𝑚𝑥 𝑡−𝛼 =𝑚𝑥 𝑡 =𝑚𝑥 (𝑚𝑥 𝑖𝑠 𝑎 𝑐𝑜𝑛𝑠𝑡𝑎𝑛𝑡)
𝑚𝑦=𝐸 𝑌 𝑡 = −∞ ∞ 𝑚𝑥(𝑡−𝛼)ℎ 𝛼 𝑑𝛼 −∞ ∞ 𝑚𝑥ℎ 𝛼 𝑑𝛼
𝑚𝑦=𝐸 𝑌 𝑡 =𝑚𝑥 −∞ ∞ (𝑡−𝛼)ℎ 𝛼 𝑑𝛼
Alternative estimate:
At 0 Hz the transfer function is equal to the DC gain
−∞ ∞ ℎ 𝛼 𝑑𝛼=𝐻(0)
Therefor:
𝑚𝑦=𝐸 𝑌 𝑡 =𝑚𝑥 𝐻(0)

38. Expected Mean square value (1/2)
𝑌(𝑡)= −∞ ∞ 𝑋 𝑡−𝛼 ℎ 𝛼 𝑑𝛼
𝐸 𝑌 𝑡 2 =𝐸 𝑌 𝑡 𝑌 𝑡
𝐸 𝑌 𝑡 2 =𝐸 −∞ ∞ 𝑋 𝑡−𝛼1 ℎ 𝛼1 𝑑𝛼1 −∞ ∞ 𝑋 𝑡−𝛼2 ℎ 𝛼2 𝑑𝛼2
𝐸 𝑌 𝑡 2 =𝐸 −∞ ∞ −∞ ∞ 𝑋 𝑡−𝛼1 𝑋 𝑡−𝛼2 ℎ 𝛼1 ℎ 𝛼2 𝑑𝛼1𝑑𝛼2
𝐸 𝑌 𝑡 2 = −∞ ∞ −∞ ∞ 𝐸 𝑋 𝑡−𝛼1 𝑋 𝑡−𝛼2 ℎ 𝛼1 ℎ 𝛼2 𝑑𝛼1𝑑𝛼2
𝐸 𝑌 𝑡 2 = −∞ ∞ −∞ ∞ 𝑅𝑥𝑥(𝑡−𝛼1,𝑡−𝛼2) ℎ 𝛼1 ℎ 𝛼2 𝑑𝛼1𝑑𝛼2
𝐸 𝑌 𝑡 2 = −∞ ∞ −∞ ∞ 𝑅𝑥𝑥(𝛼1,𝛼2) ℎ 𝑡−𝛼1 ℎ 𝑡−𝛼2 𝑑𝛼1𝑑𝛼2

39. Expected Mean square value (2/2)
𝐸 𝑌 𝑡 2 = −∞ ∞ −∞ ∞ 𝑅𝑥𝑥(𝛼1,𝛼2) ℎ 𝑡−𝛼1 ℎ 𝑡−𝛼2 𝑑𝛼1𝑑𝛼2
𝛼=𝑡−𝛼1
𝛽=𝑡−𝛼2
By substitution:
𝐸 𝑌 𝑡 2 = −∞ ∞ −∞ ∞ 𝑅𝑥𝑥(𝑡−𝛼,𝑡−𝛽)ℎ 𝛼 ℎ 𝛽 𝑑𝛼1𝑑𝛼2
If X(t)is WSS
𝐸 𝑌 𝑡 2 = −∞ ∞ −∞ ∞ 𝑅𝑥𝑥(𝛼−𝛽) ℎ 𝛼 ℎ 𝛽 𝑑𝛼1𝑑𝛼2
Thereby the Expected Mean square value is independent on time

40. Cross correlation function between input and output
Can we estimate the Cross correlation between input and out if X(t) is wide sense stationary
𝑅𝑦𝑥 𝑡+𝜏,𝑡 =𝐸 𝑌 𝑡+𝜏 𝑋∗(𝑡)
𝑅𝑦𝑥 𝑡+𝜏,𝑡 =𝐸 −∞ ∞ 𝑋 𝑡−𝛼+𝜏 ℎ 𝛼 𝑑𝛼 𝑋 ∗ (𝑡)
𝑅𝑦𝑥 𝑡+𝜏,𝑡 =𝐸 −∞ ∞ 𝑋 𝑡−𝛼+𝜏 𝑋 ∗ (𝑡)ℎ 𝛼 𝑑𝛼
𝑅𝑥𝑥 𝜏 =𝐸 𝑋 𝑡+𝜏 𝑋 (𝑡)
𝑅𝑦𝑥 𝜏 = −∞ ∞ 𝑅𝑥𝑥 𝜏−𝛼 ℎ 𝛼 𝑑𝛼=𝑅𝑥𝑥 𝜏 ∗ℎ(𝜏)
Thereby the cross-correlation is the convolution between the auto-correlation of x(t) and the impulse response

41. Autocorrelation of the output (1/2)
𝑅𝑦𝑦 𝜏 =𝑅𝑦𝑦 𝑡+𝜏,𝑡 =𝐸 𝑌 𝑡+𝜏 𝑌(𝑡)
Y(t) and Y(t+τ) is :
𝑌(𝑡+𝜏)= −∞ ∞ 𝑋 𝑡+𝜏−𝛼 ℎ 𝛼 𝑑𝛼
𝑌(𝑡)= −∞ ∞ 𝑋 𝑡−𝛽 ℎ 𝛽 𝑑𝛽
𝑅𝑦𝑦 𝜏 =𝐸 −∞ ∞ 𝑋 𝑡+𝜏−𝛼 ℎ 𝛼 𝑑𝛼 −∞ ∞ 𝑋 𝑡−𝛽 ℎ 𝛽 𝑑𝛽
𝑅𝑦𝑦 𝜏 =𝐸 −∞ ∞ −∞ ∞ 𝑋 𝑡+𝜏−𝛼 𝑋 𝑡−𝛽 ℎ 𝛼 ℎ 𝛽 𝑑𝛼𝑑𝛽
𝑅𝑦𝑦 𝜏 = −∞ ∞ −∞ ∞ 𝐸[𝑋 𝑡+𝜏−𝛼 𝑋 𝑡−𝛽 ]ℎ 𝛼 ℎ 𝛽 𝑑𝛼𝑑𝛽
𝑅𝑦𝑦 𝜏 = −∞ ∞ −∞ ∞ 𝑅𝑥𝑥(𝜏−𝛼+𝛽)ℎ 𝛼 ℎ 𝛽 𝑑𝛼𝑑𝛽

42. Autocorrelation of the output (2/2)
𝑅𝑦𝑦 𝜏 = −∞ ∞ −∞ ∞ 𝐸[𝑋 𝑡+𝜏−𝛼 𝑋 𝑡−𝛽 ]ℎ 𝛼 ℎ 𝛽 𝑑𝛼𝑑𝛽
By substitution: α=-β
𝑅𝑦𝑦 𝜏 = −∞ ∞ −∞ ∞ 𝐸[𝑋 𝑡+𝜏−𝛼 𝑋 𝑡+𝛼 ]ℎ 𝛼 ℎ −𝑎 𝑑𝛼𝑑𝛼
Remember: 𝑅𝑦𝑥 𝜏 =𝑅𝑥𝑥 𝜏 ∗ℎ 𝜏 = −∞ ∞ 𝑅𝑥𝑥 𝜏−𝛼 ℎ 𝛼 𝑑𝛼
𝑅𝑦𝑦 𝜏 =𝑅𝑦𝑥 𝜏 ∗ℎ(−𝜏)
𝑅𝑦𝑦 𝜏 =𝑅𝑥𝑥 𝜏 ∗ℎ(𝜏)∗ℎ(−𝜏)

43. Spectrum of output Given: The power spectrum is
𝑅𝑦𝑦 𝜏 =𝑅𝑥𝑥 𝜏 ∗ℎ 𝜏 ∗ℎ(−𝜏)
|𝐻 𝑓 | 2 =𝐻 𝑓 𝐻 ∗ (𝑓)
𝑆𝑦𝑦 𝑓 =𝑆𝑥𝑥 𝑓 𝐻 𝑓 𝐻 ∗ (𝑓)
𝑆𝑦𝑦 𝑓 =𝑆𝑥𝑥 𝑓 |𝐻 𝑓 | 2 x =

44. Filter examples

45. Typical LIT filters FIR filters (Finite impulse response)
IIR filters (Infinite impulse response)
Butterworth
Chebyshev
Elliptic

46. Ideal filters
Highpass filter
Band stop filter
Bandpassfilter

47. Filter types and rippels

48. Analog lowpass Butterworth filter
Is ”all pole” filter
Squared frequency transfer function
N:filter order
fc: 3dB cut off frequency
Estimate PSD from filter

49. Chebyshev filter type I
Transfer function
Where ε is relateret to ripples in the pass band
Where TN is a N order polynomium

50. Transformation of a low pass filter to other types (the s-domain)
Filter type
Transformation
New Cutoff frequency
Lowpas>Lowpas
Lowpas>Highpas
Lowpas>Stopband
Old Cutoff frequency
Lowest Cutoff frequency
New Cutoff frequency
Highest Cutoff frequency

51. Discrete time implantation of filters
A discrete filter its Transfer function in the z-domain or Fourier domain
Where bk and ak is the filter coefficients
In the time domain:

52. Filtering of a Gaussian process
X(t1),X(t2),X(t3),….X(tn) are jointly Gaussian for all t and n values
Filtering of a Gaussian process
Where w[n] are independent zero mean Gaussian random variables.

53. The Gaussian Process
X(t1),X(t2),X(t3),….X(tn) are jointly Gaussian for all t and n values
Example: randn() in Matlab

54. The Gaussian Process and a linear time invariant systems
Out put = convolution between input and impulse response
Gaussian input Gaussian output

55. Example
x(t):
h(t): Low pass filter
y(t):

56. Filtering of a Gaussian process example 2
Band pass filter

57. Intro to system identification
Modeling of signals using linear Gaussian models:
Example: AR models
The output is modeled by a linear combination of previous samples plus Gaussian noise.

58. Modeling example
Estimated 3th order model

59. Agenda (Lec. 7) Recap: Linear time invariant systems
Stochastic signals and LTI systems
Mean Value function
Mean square value
Cross correlation function between input and output
Autocorrelation function and spectrum output
Filter examples
Intro to system identification