1. Model Validation
Julian McKenzie, Bruno Muratori, Duncan Scott, Matthew Toplis, Peter Williams

2. The Dream
An online model that can robustly and repeatedly establish a desired operating setup from an unknown switch on condition.
We have a machine that can optimise itself with no requirement for human intervention!

3. Motivation Initial work into an online model of VELA.
Must be able to establish a correspondence between the observed beam and simulation, and to accurately account for observed changes under variation of parameters.

4. What We Did
Developed a method to characterise screen images of the beam (Image Analysis/Tim’s BYScIT).
Developed a model of the VELA injector in ASTRA (Bruno’s Online Model).
Compared the machine and simulation data (analysis scripts prototyped in Mathematica).

5. Talk Outline Data Taking (October 2015 VELA shift) Image Analysis
ASTRA Model & Parameter Scans
Simulation Distribution Analysis
Penalty Function
Results
Conclusions & Further Work

6. Data Taking

7. Data Taking
Took a series of images of the VELA beam under various machine setups.
Setups simplified as much as practicable.

8. Simplified Setup
Following components switched off and deguassed where appropriate:
Bucking Coil (still includes iron)
All correctors
All quads
TDC
Remaining variable parameters:
Gradient of RF
Phase of RF
Solenoid strength
Intensity of the laser spot
Position of the laser spot

9. Operational Nominal
Fixed the RF gradient and phase to a defined operational nominal that produces beam momentum of 4.5 MeV/c.
Centred beam by observing movement on YAG-03 while varying gun phase and solenoid strength.
Due to time constraints only took data at one solenoid setting.
Bsol calculated from magnet table, Egun taken from an ASTRA simulation at 4.5 MeV/c, φgun set on shift.
Parameter
φgun
Egun
Bsol
Unit
MV/m T
Nominal Value
-15 70
0.2

10. Parameter Scans Two parameters to scan over: Bunch charge 2 pC 20 pC
Laser spot position
4 positions on the virtual cathode in a square shape (with the centred position as one of the corners)
Total of 16 combinations of charge and VC position (machine setups) with images on 3 screens.

11. Parameter Scans Images taken on YAG-01/YAG-02/YAG-03:
Burst of five images recorded at 10Hz
Laser shutter closed
Burst of five background images recorded at 10Hz
Burst of five images of virtual cathode recorded at 10Hz
240 beam images with background
240 virtual cathode images

12. VC1, 50 pC Setup

13. Virtual Cathode PositionVC 1 2 3 4
Average X/Y Position (mm)
(-0.05 , 0.05)
(-0.79 , -0.06)
(-0.77 , 0.70)
(-0.06 , 0.64)
Standard Deviation (mm)
(0.06 , 0.03)
(0.05 , 0.04)
(0.05 , 0.01)
(0.05 , 0.1)

14. Image Analysis

15. Method Summary Pre-processing: Subtracting background
Cutting an image down to the screen using an elliptical mask
Fitting normal distributions to the image projections
Using the normal fits to cut to 3σ in X and Y either side of the mean position
Removing saturated points
Mean vector, μ = 𝜇 𝑥 , 𝜇 𝑦 , and covariance matrix, 𝜮= 𝜎 𝑥𝑥 𝜎 𝑥𝑦 𝜎 𝑦𝑥 𝜎 𝑦𝑦 , extracted using two methods:
Least squares fitting of a bivariate normal distribution (BVN Method)
Directly calculating the moments of the distribution which are the maximum likelihood estimators for a normal (MLE Method)

16. VC1, 50 pC

17. Errors
Took the mean μ and 𝜮 of the bursts of 5 images with the sample standard deviation as the error.
Reason was we could use the same method to get the errors for both the BVN and MLE.
We could have:
Done a ‘Monte Carlo’.
Computed the inverse of the Fisher Information Matrix.
Taken the covariance matrix from the fitting procedure.

18. BVN vs. MLE
BVN method seems to handle image noise much better (for example when analysing the 2 pC & 20 pC images) and can still analyse if not all of the beam is on the screen. However is quite slow, taking several seconds for a typical image.
MLE is much faster - taking only a few microseconds to calculate for a typical image, compared to several seconds for the BVN – but can be disrupted by image noise.
Decided to use both methods and compare the results from each to the simulations.

19. VC1 - 100 pC - 50 pC - 20 pC - 2 pC 𝝈 𝒙𝒙 𝝈 𝒚𝒚 5.5 5.6 4.6 4.7 BVN MLE
Units mm
BVN
5.5
5.6
MLE
4.6
4.7

20. ASTRA Model

21. ASTRA Model Used Bruno’s model to simulate the VELA injector.
Performed parameter scans to create a library of simulations that could be compared to the machine images.
Model consists of:
Suite of Fortran Scripts
Combined Solenoid & Bucking Coil Field Map
Generator Input File
ASTRA Input File
RF Field Map

22. ASTRA Inputs 17 inputs: – Fixed – Varied – Set For Each Scan
Run Number
No. Macroparticles (1000’s)
Laser Pulse Length (ps)
rms Laser Spot Size (mm)
Bunch Charge (nC)
Gun Gradient (MV/m)
Gun Phase (deg.)
Bucking Coil & Solenoid Strength (T)
Quad 1 Strength (k)
Quad 2 Strength (k)
11. Quad 3 Strength (k) 12. Quad 4 Strength (k) 13. End of Tracking Line (cm) 14. Space Charge (T/F) 15. Offset in X (mm) 16. Offset in Y (mm) 17 . Thermal Emittance (keV)
– Fixed – Varied – Set For Each Scan

23. Simplified Setup Simplified the model to match that of the machine:
1000 macroparticles
Laser pulse length fixed at 76 fs
All quad strengths set to zero
Tracking ended at 350 cm
Space Charge on
Laser spot distribution set to Gaussian

24. Parameter Scans
Scans conducted over the remaining variable parameters.
~3000 simulations per scan.
Parameter
εth
Rlaser
Egun
φgun
Bsol
Unit eV mm
MV/m ͦ T
Nominal
0.62
0.15 70
-15
0.20
Min
0.42
0.11 64
-21
0.14
Max
0.82
0.19 76 -9
0.26
Step
0.1
0.03 3

25. Parameter Scans
In total there were 16 sets of parameter scans performed.
Each scan used a fixed charge and offset corresponding roughly to the ones used during the data taking shift.
Allowed us to directly compare machine images taken at a given charge and VC position to a simulation under similar conditions.
VC Position 1 2 3 4
Average X/Y Position (mm)
(-0.05 , 0.05)
(-0.79 , -0.06)
(-0.77 , 0.70)
(-0.06 , 0.64)
X/Y Offset in ASTRA (mm)
(0 , 0)
(-0.75 , 0)
(-0.75 , 0.6)
(0 , 0.6)

26. Simulation Distribution Analysis

27. Original Intentions
Take electron distributions output from ASTRA at the screen positions.
Bin them into pixels to build an image.
Perform the image analysis on the constructed ‘images’.
Several problems we encountered…….

28. Issues
Time – BVN fitting took too long to analyse the large amounts of simulations conducted.
Computing Power – BVN fitting of large images at the edge of our parameter space required too much computing power.
Image Granularity

29. Unbinned Maximum Likelihood Estimation
Essentially the same as the MLE method from the image analysis procedure (albeit the data is continuous not discrete).
Solves all of the problems on the previous slide.
Also shows consistency with beam sizes calculated by ASTRA.

30. Errors
Took the standard error, using the number of live macroparticles at each screen position.
where 𝛥 is the error, 𝜆 is the measured value – can be an element of μ or 𝜮 - and 𝑁 𝐿𝑖𝑣𝑒 is the number of live particles at that point in the simulation.
We could have:
Done a ‘Monte Carlo’.
Computed the inverse of the Fisher Information Matrix.
𝛥= 𝜆 𝑁 𝐿𝑖𝑣𝑒

31. Alternative Errors - Standard Error - Monte Carlo - Fisher Information
Tried alternative ways of estimating error in the beam sizes:
Monte Carlo
Fisher Information
- Standard Error
- Monte Carlo
- Fisher Information

32. ‘Monte Carlo’ Estimate μ and 𝜮 of the distribution using MLE.
Construct 1000 random distributions using the estimates of μ and 𝜮 from the original distribution.
Estimate values of μ and 𝜮 for the 1000 random distributions.
Take the sample standard deviation of the 1000 values of μ and 𝜮 as the error in each parameter.

33. Fisher Information
Fisher information ,𝐼 𝜃 , can be written as the negative of the expectation of the second derivative with respect to 𝜃 of the natural logarithm of the Likelihood function:
When there are N parameters, such that, 𝜽= 𝜃 1 , 𝜃 1 ,…, 𝜃 𝑁 then the Fisher Information takes the form of an N x N matrix, the Fisher Information Matrix (FIM):
The covariance matrix for the parameter estimations is the inverse of the FIM:
𝐼 𝜃 =−𝐸 𝜕 2 𝜕𝜃 2 ℒ 𝑿;𝜃 𝜃
𝐼 𝜃 𝑖,𝑗 =−𝐸 𝜕 2 𝜕 𝜃 𝑖 𝜕 𝜃 𝑗 ℒ 𝑿;𝜃 𝜃
cov 𝜃 𝑖,𝑗 = 𝐼 𝜃 𝑖,𝑗 −1

34. Penalty Function

35. Penalty Function
Performed a Welch’s t-test on the beam sizes of the machine images and simulations at each screen position.
Plotted the t-test value vs simulation run number to construct a ‘penalty function’.
The minimum of this penalty function should then be the simulation that best matches the state of the machine.

36. Welch’s t-test
Statistical test that measures how close together two points are, the smaller the value of t, the closer together the points.
where 𝑋 1 , 𝑠 1 2 and 𝑁 1 are the first sample mean, variance and sample size respectively.
where subscripts M and S refer to machine images and simulations.
𝑡= 𝑋 1 − 𝑋 𝑠 𝑁 𝑠 𝑁 2
𝑡= 𝑋 𝑀 − 𝑋 𝑆 𝑠 𝑀 𝑆 𝑆 2

37. Summing Welch’s t-test
Each machine setup consists of three beam sizes in X and Y (one from each screen).
The t-test values for each of the three pairs in X and Y were summed in quadrature.
Initial results presented at IPAC constructed separate penalty functions in X and Y.
𝛹 𝑥 = 𝑡 1𝑥 2 + 𝑡 2𝑥 2 + 𝑡 3𝑥 , 𝛹 𝑦 = 𝑡 1𝑦 2 + 𝑡 2𝑦 2 + 𝑡 3𝑦 2

38. Separate Penalty Function
Penalty Function Value
Run Number

39. Separate Penalty Function
𝝈 𝒙𝒙
YAG-01
YAG-02
YAG-03
Machine
2.6
4.1
5.5
Simulation
2.4
5.9
𝝈 𝒚𝒚
YAG-01
YAG-02
YAG-03
Machine
2.4
3.6
5.8
Simulation
2.3
3.8
5.6
- Machine - Simulation

40. Combined Penalty Function
Since IPAC we have combined the t-tests from the X and Y beam sizes to construct just one penalty function.
Penalty Function Value
𝛹 𝐶 = 𝛹 𝑥 2 + 𝛹 𝑦 2
Run Number

41. Combined Penalty Function
𝝈 𝒙𝒙
YAG-01
YAG-02
YAG-03
Machine
2.6
4.1
5.5
Simulation
2.4
5.9
𝝈 𝒚𝒚
YAG-01
YAG-02
YAG-03
Machine
2.4
3.6
5.8
Simulation
4.0
5.9
- Machine - Simulation

42. Separate Analysis Results

43. Parameter Image Analysis Method Axis Rlaser Egun φgun Bsol εth 2 pC
Units mm
MV/m ͦ T eV
VC1
2 pC
BVN X
0.11 76
-18
0.2
0.82 Y
-21
MLE
0.19 67
-15
0.17
0.62
0.13 64
0.42
20 pC 70 -9
-12
50 pC
0.52
100 pC
0.72

44. Separate Analysis Results
Separate penalty functions show some consistency between machine setups for the higher charges (excluding 2 pC).
Bsol = 0.2 T
Rlaser = 0.11 mm (some exceptions), although this is at the lower limit of our parameter space.
Egun varying slightly between 67 & 70 MV/m.

45. Combined Analysis Results

46. Rlaser Egun φgun Bsol εth
Parameter
Image Analysis Method
Rlaser
Egun
φgun
Bsol
εth
Units mm
MV/m ͦ T eV
VC1
2 pC
BVN
0.11 76
-18
0.2
0.42
MLE
0.19 70
-21
0.17
0.62
20 pC
-15 67
-12
0.72
50 pC
100 pC
0.52

47. Combined Penalty Function
More consistency between setups for combined analysis.
Φgun more consistent, oscillates between -15 ͦand -18 ͦat higher charges.
εth still tends to vary quite a lot.

48. Conclusions
Results show we get good consistency with our nominal setup.
Laser spot size seems to be smaller than we anticipated, although this is at the lower limit of our parameter space so could be even smaller – it also does not increase as the charge increases.
Overall we have a model that shows good agreement with data taken from the machine at several charges and offsets.

49. Further Work Understand the errors better.
Perform more simulations, going lower with the laser spot size.
Split the analysis up and perform on one screen at a time, instead of combining, to see the difference.

50. Further Further Work
Improve the speed of the analysis – able to take data, analyse and get results quickly.
Expand the approach to more complex machine configurations.
Fully automate the procedure and incorporate into machine learning algorithms.

51. Appendix

52. VC1
VC2
- 100 pC pC pC - 2 pC
BVN MLE

53. VC3
VC4
- 100 pC pC pC - 2 pC
BVN MLE

54. Fisher Information
Take the second derivative of the log likelihood function w.r.t the five parameters of μ and 𝜮.
Construct a 5 x 5 covariance matrix of the 2nd differentials of μ and 𝜮, invert, take the diagonals and square root.

55. Separate Penalty Function
Penalty Function Value
Penalty Function Value
Run Number
Run Number

56. Combined Penalty Function
Penalty Function Value
Run Number

57. Parameter Image Analysis Method Axis Rlaser Egun φgun Bsol εth 2 pC
Units mm
MV/m ͦ T eV
VC1
2 pC
BVN X
0.11 76
-18
0.2
0.82 Y
-21
MLE
0.19 67
-15
0.17
0.62
0.13 64
0.42
20 pC 70 -9
-12
50 pC
0.52
100 pC
0.72

58. Parameter Image Analysis Method Axis Rlaser Egun φgun Bsol εth 2 pC
Units mm
MV/m ͦ T eV
VC2
2 pC
BVN X
0.11 73 -9
0.2
0.82 Y 76
-18
0.72
MLE
0.19
0.17
0.42 70
20 pC 67
-12
-21 64
0.62
50 pC
0.13
-15
100 pC
0.52

59. Parameter Image Analysis Method Axis Rlaser Egun φgun Bsol εth 2 pC
Units mm
MV/m ͦ T eV
VC3
2 pC
BVN X
0.19 70
-15
0.17
0.42 Y
0.11 76
-12
0.2
MLE -9
0.72
20 pC 67
0.52
-21
50 pC
-18
0.62
100 pC
0.82
0.13

60. Parameter Image Analysis Method Axis Rlaser Egun φgun Bsol εth 2 pC
Units mm
MV/m ͦ T eV
VC4
2 pC
BVN X
0.19 70
-15
0.17
0.42 Y
0.15 64
-21
MLE 73 -9
-12
20 pC
0.11
-18
0.2
0.82 67
50 pC
0.13
0.52
100 pC
0.72

61. Rlaser Egun φgun Bsol εth
Parameter
Image Analysis Method
Rlaser
Egun
φgun
Bsol
εth
Units mm
MV/m ͦ T eV
VC1
2 pC
BVN
0.11 76
-18
0.2
0.42
MLE
0.19 70
-21
0.17
0.62
20 pC
-15 67
-12
0.72
50 pC
100 pC
0.52

62. Rlaser Egun φgun Bsol εth
Parameter
Image Analysis Method
Rlaser
Egun
φgun
Bsol
εth
Units mm
MV/m ͦ T eV
VC2
2 pC
BVN
0.11 76
-18
0.2
0.72
MLE
0.19 73
-15
0.17
0.42
20 pC 67
-21
50 pC -9
100 pC 70
0.82
0.13

63. Rlaser Egun φgun Bsol εth
Parameter
Image Analysis Method
Rlaser
Egun
φgun
Bsol
εth
Units mm
MV/m ͦ T eV
VC3
2 pC
BVN
0.19 67 -9
0.17
0.42
MLE
0.82
20 pC
0.11 70
-18
0.2
0.52
50 pC
-15
0.72
100 pC

64. Rlaser Egun φgun Bsol εth
Parameter
Image Analysis Method
Rlaser
Egun
φgun
Bsol
εth
Units mm
MV/m ͦ T eV
VC4
2 pC
BVN
0.19 70
-21
0.17
0.52
MLE -9
0.82
20 pC
0.11
0.2 67
-18
50 pC
-15
0.42
100 pC