1. Scheduler overview status & issues
A.Gheata
GeantV CERN meeting
Mar 16, 2015

2. Outlook Data structures and memory New vector balancing policy
New basket management policy
New prioritization policy
Issues

3. GeantTrack_v Contiguous buffer GeantTrack GeantTrack_v 00 40 80 C0 00
fEvent
fEvslot
fParticle
fPDG …
fXpos
fYpos
fZpos
fXdir
fYdir
fZdir
Edep
Pstep
Snext
Safety
*fEventV
*fEvslotV
*fParticleV
*fPDGV …
*fXposV
*fYposV
*fZposV
*fXdirV
*fYdirV
*fZdirV
*fEdepV
*fPstepV
*fSnextV
*fSafetyV
vector 1
fEventV
fEvslotV
fParticle V
fPDGV …
fXPosV
fYPosV
SOA of fNtracks
fZPosV …
192 bytes + 2*sizeof(VolumePath_t)
fNtracks=10 padding=32
fSnextV
fSafetyV
fPathV
fNextpathV
vector 2
*fPath
*fPathV
sizeof(VolumePath_t) = *max_geom_depth
Sizeof(GeantTrack_v) = fNtracks*sizeof(VolumePath_t)
*fNextpath
*fNextpathV

4. Size considerations
Size of a GeantTrack_v depends on the track capacity and maximum geometry depth (15 for CMS)
(256, 15) -> size = 176 kBytes
(16, 15) -> size = 22 kBytes
CMS tracks size for capacity of 256 tracks
4173 basket managers
2 baskets
2 track arrays per basket
4173*4*176 = 2.85 Gbytes (no baskets transported!!!)
For 16 tracks per basket: 358 Mbytes
Pending baskets can be cut off on memory threshold
Smarter basket policy implemented

5. Memory threshold // Maximum user memory limit [MB]
propagator->fMaxRes = 2000;

6. GeantBasket Elementary work unit for GeantV
They currently only hold tracks that are physically inside a given logical volume
Evolution: filter tracks by different criteria providing locality for any processing stage (when it’s worth)
Input GeantTrack_v array, filled by the scheduler
Output GeantTrack_v array, filled during transport
Baskets have thread local access during transport, but concurrent access during scheduling (!)
Recycled after re-basketizing to owner basket managers
Mixed baskets: containing mixed tracks in different volumes
Everything called with scalar interface
To avoid overheads when vectorization is difficult or penalizing
Input
Scheduler multithreaded
re-basketizing
Scheduler re-basketizing
Transport (single thread)
Physics
Output
AddTrack

7. Basket managers One per logical volume (4173 in cms2015)
2 baskets each (4 GeantTrack_v arrays)
Current: normal scheduling operations
Priority: prioritize events
1 queue for concurrent replacement and recycling
Dynamic track content threshold per basket, pushing to work queue
Low basket flow-> small baskets, high flow-> large baskets
At any moment at least 2+Nqueued baskets instantiated and held by one basket manager
Plus a variable number of detached baskets being processed
TGeoVolume
Basket manager
current
priority
Recycled baskets bounded queue

8. Vector size balancing
GeantScheduler
Volume
current BM
Ntotal = Nused + Nrecycle
fThreshold*
Concurrent track addition, garbage collection, collection of tracks from prioritized events
Adjustable threshold, aiming for Nused = Nthreads
Nused (per volume type)
Transport queue

9. Monitoring vectorization
=== Thread 1: exiting ===
=== Thread 3: exiting ===
=== Percent of tracks transported in single track mode: %

10. New basket allocation policy
Monitor distribution of steps in volumes
After initial total number of steps
Sort by activity, sum-up bins to threshold (e.g. 90% of total steps)
Activate basket managers, which will represent a fraction of the total
Redo after every 4x previous number of steps

11. New basket allocation policy
=== Learning phase of steps completed ===
Activated 528 volumes accounting for 90.0% of track steps
* FixedShield102880: steps
* HVQX8780: steps
* ZDC_EMLayer9b00: steps
* BeamTube22b780: steps
* OQUA6780: steps
* QuadInner3300: steps
* ZDC_EMAbsorber9d00: steps
* QuadOuter3700: steps
* QuadCoil3680: steps
* ZDC_EMFiber9e80: steps

12. New priority policy Monitor distribution of tracks per event
Start prioritizing when #tracks is a fraction of the maximum reached (e.g. 1%)
The remaining tracks collected by mixed baskets
One mixed basket per thread
Cutting short event tails
no need for one priority basket per volume
less basket fragmentation
concurrency should get a bit better

13. Queue with new policy …
Imported 340 tracks from events 10 to 10. Dispatched 20 baskets.
### Event 0 prioritized at 1 % threshold
Event 0: tracks transported, max in flight 5187
= digitizing event 0 with 0 tracks
=> Importing event 11

14. Threads WorkloadManager::MainScheduler (1)
Method run as separate thread
Does queue monitoring, triggering actions
WorkloadManager::GarbageCollectorThread (1)
Woke up by MainScheduler every 50 iterations
Garbage collects pending baskets for every basket manager
WorkloadManager::MonitoringThread (1)
Runs as a background thread, activating histograms on demand
Work queue, memory, number of baskets per volume, concurrency, number of tracks in flight per event
WorkloadManager::TransportTracks (Nworkers)
Main basket transport method
Calls also the re-basketizer (GeantScheduler::AddTracks)

15. Issues Basket management too memory hungry
Pre-allocate everything policy not good for CMS
We know that ~10% of volumes take >60% of transport
Implement policy to create baskets only for these volumes – DONE
Priority algorithm producing too fragmented baskets and taking too long to flush events
Preempt starting of priority regime, independent of queue status - DONE
E.g. when number of tracks in flight for one event is less than 5% of the maximum ever reached (cutting tails)
Keep only prioritized tracks in the same mixed basket and reuse baskets without re-basketizing when population low.

16. Issues (2)
Contention on re-basketizing high (specially with fat baskets)
Read from many, write to one policy
Highly optimized using atomics, but still… Amdahl watches
Overheads in concurrent queues non negligible
Currently scaling to ~6 threads
Can be changed to reading concurrently the same basket, but writing to thread local one
Packaging events per group of threads always possible
Concurrency in the new scheduling approach to be closely monitored (VTune)