1. Calibration and Validation
Simulation calibration and validation
• Microscopic traffic calibration and
validation

2. Issues in Simulation Issues in Model Development
– Verification
– Validation
• Issues in Simulation use
– Calibration

3. Simulation Development Process
• Problem structuring and formulation
• System and simulation specification
• Data collection and input modeling
• Model formulation and construction
• Verification and validation
• Experimentation and analysis
• Implementation

4. System and Simulation Specification
Typical specification includes:
– Simulation objectives
– System description and modeling approach
– Animation
– Model input and output

5. Simulation Study Steps

6. Verification
• Verification is a procedure to ensure that the model is built according to specifications and to eliminate errors in the structure, algorithm, and computer implementation of the model.
• In other words:
– Coding the model right, or
– Debugging the model

7. Validation A process by which we ensure that the
model is suitable for the purpose for which
it is built
– in other words: building the right model
– all we have to do is check that the model behaves as the real-world does under the same conditions.

8. Verification and Validation
• System  Model  “Code”
• Validation: Is Model = System?
• Verification: Is “Code” = Model?” (debugging)

9. Verification Build and verify small sub-models
• Test with various combinations of input
parameter values
• Use debugging tools: trace, step-by-step
execution, watch points, breakpoints, etc
• Observe the animation (if available)

10. Validation Based on the collection of reliable data
• Using existing data to see if model outputs match system performances
– Use correct statistical analysis of the data: t-test, paired t-test, etc.
• Testing the model’s ability to predict the future
behavior of the real system
– under known historical circumstances
– under new circumstances

11. Validation Techniques
– Inspections
– Turing test
– Cause-effect graphing
– Interface analysis
– Object-flow testing
– Sensitivity analysis
– Special input testing
– Statistical techniques
– Visualization/animation

12. Is it a valid Model? • What it means, if a simulation model is valid?
• Validation process depends on:
– The complexity of the system
– Whether the system exists
• There is no such thing as absolute model validity
– The more time you spend on model development, the more valid it should become
• Model that is valid for one purpose may not be for another
• Validation is not something you do after the simulation model has been developed

13. Issues in Using Simulation
Calibration
Validation
Multiple runs
Results presentation

14. Calibration Procedure
• Builds validity by verifying that simulation results
match the field observation
• The calibration can be achieved by adjusting
– Input data
– Default values, and
– Model embedded data
Until the simulation results replicates the field conditions

15. Multiple Runs • Multiple runs using different random number
seeds to model stochastic behavior
• Fixed random number stream typically used for
model development and verification (for ease of
debugging and testing)
• Multiple runs most critical in near or over saturation conditions
• Use multiple run statistics for final results (instead of those from one run)

16. Microscopic Traffic Simulation Calibration
Several levels
• Calibrate the network
– Code links and intersections/junctures to accurately
represent and reflect existing conditions
• Calibrate vehicle/driver performances
– Adjust vehicle/driver characteristics to make the model
more accurately reflect existing conditions
• Calibrate model parameters
– Car-following
– Lane change
– Gap acceptance
– Etc.

17. Model Calibration Use default parameters first
• Check code errors first!
• Adjust the model embedded data
– The default values embedded typically have been validated against real world results. They are the results of model development and in many times they are accurate and sufficient
– Calibration is most important when the project has unique characteristics
• Area driver population
• Special operation
• Congested network
• Etc.

18. Surface Street Calibration example
Field measurements on surface street
– Vehicle throughput
– Travel times
– Vehicle delay
– Queue lengths

19. Surface Street Calibration example
• Calibration parameters on surface
street/intersection
– Desired free flow speed
– Queue discharge headway
– Start-up lost time

20. Surface Street Calibration example
• Factors affecting free flow speed on a link
– Intersection spacing
– Speed limit
– # lanes
– Lane width
– Parking
– Mid-block pedestrians
– Land use access
– Availability of turning bays
– Etc.

21. Surface Street Calibration example
Factors affecting start-up lost time
– Perception/reaction time
– Driver attentiveness
– Driver caution for red-light running, pedestrian violations, etc.
– Driver familiarity with the area

22. Calibration Example: CORSIM
• Comprehensive parameters (network/driver/driver)
– Free flow speed
– Start-up lost time/queue discharge headway
– OD and routes (u-turns, circling, etc…)
– Vehicle composition
– Driver composition

23. Calibration Example: CORSIM
Geometric conditions
– Warning signs to:
• HOV
• Off-ramp
• Incident Blockage
• Lane Drop/Add
– Intersection geometry (turning bay length etc.)
– Entry link length

24. Calibration Example: CORSIM
• Vehicle/driver related parameters
– Car-following
– Lane change
– Driver familiarity with routes
– % “cooprative” drivers
– Probability joining spillback

25. Calibration Example: CORSIM
Traffic control related parameters
– Jumpers/sneakers
– Acceptable gaps
• permitted left
• right turns (RTOR)
• crossing at stop sign

26. Calibration Example: CORSIM
Calibration for congested network
– Adjust percentages of vehicles that join spill-backs
– Check the proper lengths of turning lanes
– Change acceptable gaps for turning and crossing
– Calibrate lane change parameters
– Increase cooperative driver percentage
– Increase percentage of vehicles that are familiar with the
network
– Adjust driver aggressiveness
– Place warning sign further upstream
– Etc.

27. Calibration Example: CORSIM

28. Calibration Example: CORSIM

29. Calibration Example: CORSIM

30. Calibration Example: CORSIM

31. Model Validation Goal: Model agrees with system
• Systematic validation
• Multiple level:
– From aggregated average to microscopic
• Multiple tools
– Using animation very critical
• High quality data very critical