1. Digital UNIX Internals ISupporting User Threads12 - 1 Supporting User-Threads: NXM 2-Level Scheduling Chapter Thirteen

2. Digital UNIX Internals ISupporting User Threads12 - 2 While Compaq believes the information included in this presentation is correct as of the date produced, it is subject to change without notice. All rights reserved. Compaq, the Compaq logo, DIGITAL and the DIGITAL logo are trademarks of Compaq Computer Corporation. IBM is a registered trademark of International Business Machines. Microsoft is a registered trademark of Microsoft Corporation.

3. Digital UNIX Internals ISupporting User Threads12 - 3 Goals of Many to N (NXM) Scheduling q Schedule multiple user threads managed by the library on the contexts of a smaller set of kernel threads maintained by the kernel F Improved performance F Improved scalability q Library to NXM Communication F System calls q NXM to Library Communication F Upcalls

4. Digital UNIX Internals ISupporting User Threads12 - 4 NXM Terminology Physical Processors Kernel’s Structures for the Process (Mach Threads) Scheduler Threads (Virtual Processors) Library’s Structures for the Process User Threads (Pthreads) Bound Threads LibraryApplication

5. Digital UNIX Internals ISupporting User Threads12 - 5 Thread roles -- I of II q Manager thread F Kernel only thread F Manages the kernel half q Scheduler thread F Kernel thread assigned to user work q Blocked scheduler thread F User thread waiting in the kernel q Performing a system calls q Handling a page fault F “Replaced” by another scheduler thread q so other user work can be done

6. Digital UNIX Internals ISupporting User Threads12 - 6 Thread roles -- II of II q Extra threads F Kernel thread awaiting user work F After blocked threads awaken q the number of kernel threads created for user work may exceed the number of CPU’s plus the number of blocked requests F Terminated within a few seconds q Unless new work appears q Pthreads F Created and managed by the user library F Multiplexed on scheduler threads

7. Digital UNIX Internals ISupporting User Threads12 - 7 Library Syscalls to Kernel q Allow library to request operations specific to 2 level scheduling q Most importantly to bind user-threads to kernel threads.

8. Digital UNIX Internals ISupporting User Threads12 - 8 NXM Syscalls nxm_task_init() initialize process for two level scheduling nxm_thread_create() create a thread for use of thread library nxm_idle() declare this thread to be idle nxm_wakeup_idle() release idle thread(s) nxm_thread_kill() send signal to a user thread nxm_thread_block() block a bound thread nxm_thread_wakeup() unblock a bound thread nxm_set_cancel() set cancel cancellation state nxm_get_state() returns registers (for debuggers) nxm_signal_check() check for signals (for library) nxm_thread_suspend() suspend a bound thread nxm_thread_resume() suspend a thread nxm_thread_destroy() destroy a thread nxm_resched() make thread perform resched upcall This is not a complete list

9. Digital UNIX Internals ISupporting User Threads12 - 9 Syscalls implementation: Mach System calls  kern/syscall_sw.c mach_trap_table[].… FN(nxm_thread_destroy, 0, 0),/* 29 */ FN(nxm_thread_create, 3, 0),/* 32 */ FN(nxm_task_init, 2, 0),/* 33 */ FN(nxm_idle, 1, 0),/* 35 */ FN(nxm_wakeup_idle, 1, 1),/* 36 */ FN(nxm_set_pthid, 2, 0),/* 37 */ FN(nxm_thread_kill, 2, 0),/* 38 */ FN(nxm_thread_block, 5, 0),/* 39 */ FN(nxm_thread_wakeup, 1, 0),/* 40 */....

10. Digital UNIX Internals ISupporting User Threads12 - 10 Kernel Upcalls to Library q Notify the library of significant execution events F arrival of a signals F a user thread blocking F quantum expiration q Allow library to bind a new user thread to a scheduler thread

11. Digital UNIX Internals ISupporting User Threads12 - 11 NXM Upcalls NXM_QUANTUM_EXPIRE quantum expired NXM_SCHED_THREAD_UT_BLOCKED thread blocked in kernel NXM_THREAD_UNBLOCK_NO_SID thread unblocked no sched id NXM_SIGNAL_BLOCKED current thread has blocked sig NXM_SPECULATIVE_EXECUTION call user spec-exec handler NXM_THREAD_INTERRUPTED sched/bound thread interrupted NXM_VP_RESCHED vp should reenter scheduler NXM_GENTRAP_HANDLER call user gentrap handler NXM_STACK_OVERFLOW yellow zone notification NXM_FB_FIXUP fixup FBs (fat binaries) NXM_VP_UNBOUND vp became unbound

12. Digital UNIX Internals ISupporting User Threads12 - 12 Upcall Implementation q Transitions from kernel to user-mode F Into a thread library routine q Initiated by kernel F nxm_upcall() q Made by an idle scheduler thread from F nxm_idle() q Or from AST (Asynchronous System Trap) context F thread_ast_set(thread, AST_BLOCKED_UPCALL)

13. Digital UNIX Internals ISupporting User Threads12 - 13 Handling Blocking System Calls User thread 1, kernel thread 1 sleep() Kernel thread 1 *activates kernel thread 2 and sleeps thread_block() nxm_block() Kernel thread 2 makes a NXM...BLOCK upcall Library maps kernel thread 2 to user thread 2 thread_ast_set() trap() nxm_upcall() Kernel thread 1 is awoken, makes NXM...UNBLOCKED... upcall thread_ast_set() trap() nxm_upcall() Kernel User-Space Library App.

14. Digital UNIX Internals ISupporting User Threads12 - 14 Handling Idle Threads User thread 1, kernel thread 1 nxm_idle() Kernel thread 1 waits, if not needed in 20 seconds will exit. Kernel thread 2 makes a upcall indicating user-thread 2 is runnable Library maps kernel thread 1 to user thread 2 Kernel thread 1 is awoken, makes NXM...UNBLOCKED... upcall thread_ast_set() trap() nxm_upcall() Kernel User-Space Library App. User thread 1 blocks in library, No other threads runnable nxm_wakeup_idle() Library records user thread 2 now runnable

15. Digital UNIX Internals ISupporting User Threads12 - 15 Other Conditions q Signals F Kernel must notify library of signal arrival q Scheduling Attributes F Library wants to set user thread scheduling attributes q nxm_resched  Kernel must track user thread quantum of a running nxm thread in hardclock() and notify library when it expires q nxm_upcall(), NXM_QUANTUM_EXPIRE q Thread Cancellation  One user thread can cancel another with pthread_cancel() F Kernel must test for cancellation in system calls for nxm threads

16. Digital UNIX Internals ISupporting User Threads12 - 16 To sleep, perchance to run q Blocked threads F Sleep on the appropriate kernel event q But with an AST set to do upcall before returning to user mode F If blocked by nxm_thread_block  Sleeps on thread’s nxm_flags field n Manager threads sleep this way q Extra threads  Sleep on nxm_extra_count field of their slot q If time expires, terminate thyself q Idle threads F Sleep on address of nxm slot

17. Digital UNIX Internals ISupporting User Threads12 - 17 NXM Structures: The Big Picture struct nxm_shared nxm_callback 0xBBB nxm_ss[] struct task nxm_user: 0xAAA nxm_share thread_list Process’s seg0 (VAS) XXX() { } struct nxm_shared 0xAAA 0xBBB nxm_sptr nxm_flags uu_sptr uu_share super_thread nxm_sptr nxm_flags uu_sptr uu_share super_thread Process’s Kernel Structures nxm_sptr nxm_flags uu_sptr uu_share super_thread nxm_sptr nxm_flags uu_sptr uu_share super_thread

18. Digital UNIX Internals ISupporting User Threads12 - 18 struct nxm_shared q Mapped into User Address Space q Holds information required by kernel and library  One per NXM process struct nxm_shared { long nxm_callback; Address of upcall routine unsigned short nxm_version; Version number unsigned short nxm_uniq_offset; Correction factor for TEB int pad1; long space[2]; Future growth struct nxm_sched_state nxm_ss[1]; Array of shared areas }; (number of CPUs)

19. Digital UNIX Internals ISupporting User Threads12 - 19 struct nxm_sched_state q Field of nxm_shared q Represents state of a NXM Virtual Processor (Kernel Thread)  Sized One Per CPU struct nxm_sched_state { struct ushared_state nxm_u; State owned by user thread nxm_sched_bits_t nxm_bits; int nxm_quantum; Quantum count-down value int nxm_set_quantum; Quantum reset value int nxm_sysevent; Syscall state struct nxm_upcall *nxm_uc_ret; Stack ptr of null thread void *nxm_tid; Scheduler's thread id long nxm_va; Page fault address struct nxm_pth_state *nxm_pthid; ID id of null thread.... };

20. Digital UNIX Internals ISupporting User Threads12 - 20 struct ushared_state q Context-switched shared-state structure F Field of nxm_sched_state F Shared directly between the running user thread and the kernel  Context-switched by the user thread library: struct ushared_state { sigset_t sigmask; Thread signal mask sigset_t sig; Thread pending mask struct nxm_pth_state *pth_id; Out-of-line state int flags; Shared flags int cancel_state; Thread's cancellation state Semi-shared, visible to kernel but never context-switched by library: int nxm_ssig; Scheduler's synchronous signals int reserved1; long nxm_active; Scheduler active long reserved2; };

21. Digital UNIX Internals ISupporting User Threads12 - 21 struct nxm_upcall q Upcall frame structure F The kernel builds this frame on the stack and its address on an upcall  This same structure can be used for saving/restoring user thread state on library context switches: struct nxm_upcall { int usaved; /* u_state is valid */ int pad; struct ushared_state u_state; /* shared state */ struct alpha_saved_state reg_state; /* register state */ };

22. Digital UNIX Internals ISupporting User Threads12 - 22 struct nxm_pth_state  Out-of-line user thread state. The kernel can access this when needed, but only the pointer to it (ushared_state.nws) lives in the shared state: struct nxm_pth_state { u_long fpregs[32]; u_long fpcr; Must follow the fpregs! stack_t altstack; struct uuprof prof; struct nxm_ieee_state { long ieee_fp_control; Floating point state long ieee_set_state_at_signal; long ieee_fp_control_at_signal; long ieee_fpcr_at_signal; } nxm_ieee_state; int sigforce; int stack_event; long pad[9]; };

23. Digital UNIX Internals ISupporting User Threads12 - 23 struct task NXM fields  struct task {... struct nxm_task *nxm_state; Pointer to all 2-level state …  Where the 2-level scheduling state is: struct nxm_task { cpuset_t _nxm_sched_mask; Mask of blocked sched threads int _nxm_sched_max; Max scheduler threads int _nxm_sched_cnt; Active scheduler threads unsigned int _nxm_retry_count; nxm_get_thread() retries unsigned int _nxm_retry_cpu; nxm_get_thread() timeout cpu struct thread *_nxm_manager; Library's manager thread sigset_t _nxm_signal_upcall; Signals needing upcalls long _nxm_task_callback ; Library upcall routine unsigned int _nxm_task_version; nxm version unsigned short _nxm_task_uniq_offset; Offset to TEB int _nxm_task_quantum; Quantum from task attributes vm_offset_t _nxm_user_cfg_addr; User VA of config area vm_size_t _nxm_cfg_size; Size of config area vm_size_t _nxm_share_size; Size of shared area nxm_config_info_t *_nxm_cfg_ptr; Shared config area struct nxm_shared **_nxm_share; Pointer to shared areas struct nxm_vp_info _nxm_vp[1]; Per-slot data };

24. Digital UNIX Internals ISupporting User Threads12 - 24 struct thread NXM fields struct thread {… struct nxm_sched_state *nxm_sptr; Scheduler state (if NXM thread) int nxm_flags; … int nxm_slot; …} thread.nxm_flags values: #define NXM_IDLE 0x1 Idle with no scheduler context #define NXM_SCHED0x2 Has scheduler context valid #define NXM_NOSCHED 0x4 No scheduler context valid #define NXM_BLOCKED 0x8 Blocked NXM thread #define NXM_BIND_ME 0x10 Special #define NXM_BOUND0x20 User thread bound to kernel thread #define NXM_EXEC 0x40 Exec in progress #define NXM_RESCHED 0x80 Force resched #define NXM_SIGEV0x200 Signal event pending #define NXM_MANAGER 0x400 Thread is a manager #define NXM_WAKEUP 0x800 Thread wakeup is pending #define NXM_SUSPEND 0x1000 Library suspend in effect #define NXM_RESUME_POSTED 0x2000

25. Digital UNIX Internals ISupporting User Threads12 - 25 NXM struct thread nxm_flags Values NXM_IDLE idle with no scheduler context NXM_SCHED has valid scheduler context NXM_NOSCHED no valid scheduler context NXM_BLOCKED blocked NXM thread NXM_FP_STATE blocked and has FP state NXM_BOUND user thread bound to kernel thread NXM_EXEC exec in progress NXM_RESCHED force resched

26. Digital UNIX Internals ISupporting User Threads12 - 26 Other Thread Related NXM fields struct np_uthread { … NXM Information struct ushared_state *uu_sptr; struct ushared_state uu_share; void *uu_tsd[MAX_TSD_SLOTS]; Thread-specific data, currently }; MAX_TSD_SLOTS is 8 struct uthread { … struct nxm_pth_state *uu_proflast; Last nxm thread to use uu_prof … }

27. Digital UNIX Internals ISupporting User Threads12 - 27 Selected Routines q nxm_task_init() q nxm_thread_create() q nxm_get_thread() q nxm_idle() q thread_block() q nxm_block()

28. Digital UNIX Internals ISupporting User Threads12 - 28 nxm_task_init( int max_sthreads,vm_offset_t sthread_array, nxm_task_attr_p sched_attr ) q System call to set up the caller's task to perform 2-level thread scheduling F Create a shared memory region with the kernel q The caller must provide a page-aligned user address for scheduler-thread array q Wires the memory from the kernel so that the page cannot go away or be be deleted by the caller at some later time q Creates a VP thread if older library  Makes the calling thread a scheduler thread with attributes based on the sched_attr argument

29. Digital UNIX Internals ISupporting User Threads12 - 29 nxm_thread_create (nxm_thread_attr_p attr, long *kid, int thread_index) q System call to create a thread for use by the threads library: F This can be one of: q VP/scheduler thread, q a bound (system scope) thread, q or a manager thread F Verify  Thread index is less than NXM_THREAD_MAX (64)  Thread policy and priority F Create the new thread: q Initialize the policy, priority, the scheduling and execution state (i.e. initialize the PCB for the thread) q Deals with CPU binding F Try to start the thread

30. Digital UNIX Internals ISupporting User Threads12 - 30 nxm_get_thread (boolean_t restart)  Run exclusively by the NXM manager thread in response to a scheduler thread calling nxm_block() F Scan the list of blocked scheduler threads and create new threads to replace them after trying to wake up an extra thread to do the work F On failure, set a timeout to come back here and retry F Responsible for creating a new scheduler thread when an existing one blocks and there are currently no idle threads F Creates a new scheduler thread to carry out a block-user- thread upcall to the threads library

31. Digital UNIX Internals ISupporting User Threads12 - 31 nxm_idle(union extra_arg arg) q System call to declare this thread idle q If thread is a scheduler thread  Call nxm_sched_idle q Sleep until needed q If thread is not a scheduler thread  Call nxm_extra_idle q If scheduler thread in this slot is block n Replace it q Else if not too many extra threads n Become an extra n Go to sleep for a while n If awakened with no work to do –Terminate

32. Digital UNIX Internals ISupporting User Threads12 - 32 thread_block() and NXM (1) q thread_block() F selects new thread  calls nxm_block( ) to setup for upcall F context switches to new thread

33. Digital UNIX Internals ISupporting User Threads12 - 33 thread_block() and NXM (2)  nxm_block() F handle case of thread non-interruptible sleep F handle case of a library thread doing library work in a critical section F schedule an AST when awaken to perform upcall q Prepare this scheduler thread to pass on its identity to a new scheduler thread q This requires copying the current shared area back into the kernel, setting our state to BLOCKED, and getting a replacement thread going.

34. Digital UNIX Internals ISupporting User Threads12 - 34 thread_block() and NXM (3)  nxm_block() continued q.... q save off floating point state q set threads NXM state to blocked NXM_BLOCKED q if the thread is in a scheduling state n increment count of blocked NXM threads q if extra threads laying around, n wake up a replacement q else check if enough threads are around. If not, n wake up manager thread to create a replacement.

35. Digital UNIX Internals ISupporting User Threads12 - 35 Source References: NXM Scheduling (1) q kernel/kern/syscall.c q kernel/kern/syscall_subr.c F nxm mach system call definitions q kernel/kern/thread.h F struct thread q kernel/kern/task.h F struct task q kernel/arch/alpha/nxm.h F struct nxm_shared F struct nxm_sched_state q kernel/kern/sched_prim.c F thread_block() F nxm_block()

36. Digital UNIX Internals ISupporting User Threads12 - 36 Source References  kernel/kern/syscall_subr.c F nxm_task_init() F nxm_get_thread() F nxm_idle() q kernel/arch/alpha/nxm.h F nxm upcall definitions F struct ushared_state q kernel/arch/alpha/trap.c F trap()

37. Compaq Computer Corporation © 1998