1. Multi-Thread Programming Vincent Liu MDE GPE-EA Sun Microsystems Inc.

2. What in this topic Basics about multi-threaded Program Solaris's thread model Synchranization mechanism Lock Contention How to measure a MT program's scalability on Solaris Some tools at our command on Solaris Improve MT program's performance

3. Agenda Introduction to Multi-Thread Basic Thread Programming Thread Synchronization Locking Problems Advanced Multi-Thread programming Solaris Thread libraries

4. Introduction to Multithreading Process, Thread and LWP Which Applications Benefit? When Not to Use Threads Thread Standard

5. What Is a Thread? An independent flow of control in a program A “virtual” central processing unit(CPU) Thread share the address space and Process Structure, with its own stack, TCB, KTCB Traditional Process – MultiThreaded process with only one thread

6. Process,Thread and LWP

7. Thread vs. Process Thread : high degree of parallelism on multiprocessor systems Kernel resource consumed ( create, schedule, etc) Thread: lightweight Process: heavyweight Information sharing Thread : share process address space Process: IPC

8. Single LevelThreads Model The default model in Solaris 9 and 10 All user threads bound to LWPs Kernel level scheduling – No more libthread.so scheduler More expensive thread create/destroy,Synchronization More responsive scheduling, synchronization

9. Which Applications Benefit? Threads can : Simplify program structure Improve throughput Improve responsiveness Minimize system resource usage Simplify realtime applications

10. When Not to Use Threads For compute-bound threads on a uniprocessor. For threads that execute very short tasks When there is nothing to run concurrently For multithreaded applications that are more difficult to design and debug than single-threaded applications.

11. Thread Interfaces Two International standards: POSIX Solaris 2.5+, HP-UX 10.30+, Digital Unix 4.0+ IRIX 6.2+, VMS, AS/400, AIX 4.1+ UI Thread Solaris 2.2+, UnixWare

12. Basic Thread Programming Thread Life Cycle Thread APIs Thread Attribute

13. Thread Life Cycle main() { pthread_create( &tid, &attributes, function, arg ); … } void *funciton(void *arg) { … pthread_exit(status); } pthread_create(… function(), arg) Function(arg)pthread_exit(status)

14. Simple Multi-Thread program #include main( ) { pthread_t t1; void * status ; printf ( “main thread id is : %d \n”, pthread_self( ) ); pthread_create( &t1, NULL, func, NULL) ; pthread_join( t1, &status ); pthread_exit ( status ); } void * func( void * arg ){ sleep (10); pthread_exit ( (void * ) 44 ) ; }

15. POSIX Thread APIs int pthread_create(&tid, &attr, func, arg ); void pthread_exit(status); int pthread_join(tid, &status); pthread_t pthread_self(); int pthread_equal(tid1, tid2); int pthread_cancel(tid); void sched_yield();

16. Waiting for a Thread to Exit “Undetached” threads must be joined and may return a status value “Detached” threads cannot be joined and cannot return a status value t1 t2 pthread_join(t2 ) pthread_exit(status )

17. Thread Attributes Usage pthread_attr_t attr; pthread_attr_init( &attr); pthread_attr_setscope(&attr, PTHREAD_SCOPE_SYSTEM ); pthread_attr_setdetachstate ( &attr, PTHREAD_CREATE_JOINABLE); pthread_create( &tid, &attr, func, arg);

18. Thread Synchronization Why use Synchronization Synchronization Mechanism Mutex Semaphore Condition Variable

19. Why use Synchronization Unsynchronized Shared data is a formula of disaster Thread 1 Thread 2 temp = your.bankbalance; dividend = temp * interestrate; newbanlance = dividend + temp; your.bankbalance += deposit ; your.bankbalance = newbalance;

20. All shared Data Must Be Locked Good programmers protect all usage of shared data with locks Shared data lock(M); ……. unlock(M) ; lock(M); ….. unlock(M) ;

21. All shared Data Must Be Locked Global variables Process resources Shared data structures Static variables

22. Synchronization Variables Used to protect shared resource Mutex exclusin locks Used to determine when a thread should run Counting Semaphores Condition variables

23. Mutex Sample Pthread_mutex_t lock; pthread_mutex_init(&lock, NULL); ….. Thread 1 Thread 2 pthread_mutex_lock( &lock); pthread_mutex_lock(&lock); deposit(acct, x); draw(acct, y); pthread_mutex_unlock(&lock); User Account deposit draw

24. Mutexes The blocked thread might not get the mutex Typical lock/unlock time: one cycle Critical sections should be as short as possible Non-critical sections are usually much longer typically

25. Semaphores sem_wait: decrease the count if non-zero; otherwise wait until others execute sem_post sem_post: increase count by 1, and wakeup sleepers if exists sem_init : semaphore initialization Count 0 Sleepers t1t3

26. Semaphore Excecution Graph sem_wait s=0,waiting sem_post s=1,wake up t1 s=0 sem_post s=1 sem_wait s=0 sem_wait s=0,waiting t1 t2 t4 t5 t3

27. EINTR Semaphore can be interrupted by signals, sem_wait can return without decreasing the value while(sem_wait(&s) = = EINTR ) { } do_thing;

28. Avoiding Deadlocks Deadlocks can always be avoided You can establish a hierachy Use a static analysis program(for example lock_lint) to scan your code for hierarchy violations Use trylock primitive pthread_mutex_lock(&m2); … if (EBUSY==pthread_mutex_trylock(&m1)) { pthread_mutex_unlock(&m2); pthread_mutex_lock(&m1); pthread_muetx_lock(&m2); } do_real_work();

29. Advanced Topic Thread Specific Data Unix Signals Advanced scheduling MT-Safe library

30. Why TSD is needed Sometimes it is useful to have a global variable which is local to a thread Thread 1 Thread 2 err = read(..); err = ioctl(…); if ( err ) if(err) printf(“%d\n”, errno); printf(“%d\n”, errno);

31. TSD Usage Key Creation – pthread_key_create( &key, destructor ) Key Delete – pthread_key_delete( key ) Modify TSD – pthread_setspecific( key, value) Access TSD – pthread_getspecific( key )

32. TSD Sample pthread_key_t key1; main(){ pthread_key_create( &key1, destroyer ); pthread_create( &t1, NULL, foo, 3.0); pthread_create ( &t2, NULL, foo, 4.0); } foo( float x ){ pthread_setspecific( key1, x ); bar(); } bar(){ n = pthread_getspecific( key1); }

33. TSD Destructors When a thread exits, it first sets the value of each TSD element to NULL, then calls the destructor on what the value was If you delete a key, the destructor functions will not run, you must deal with it yourself

34. Advanced Topic Thread Specific Data Unix Signals Advanced scheduling MT-Safe library

35. Three uses of Signals Synchronous signals for Error reporting – SIGFPE, SIGSEGV, SIGBUS, etc Asynchronous signals for Situation reporting Asynchronous signals for Interruption – SIGKILL, SIGALRM, SIGSTOP, etc

36. Traditional Signal Handling main()foo() USR1foo 1 2 3 4 5 6

37. POSIX Signal Model signal library’s signal handler routines ? Signal dispatc h table Thread signal mask

38. Dedicated Signal Handling Thread A multithreaded program can create one or more thread dedicated to perform signal handling for whose process by using sigwait pthread_create( &p, &attr, handler, arg ); …. void * handler ( void * arg ){ while (1){ sigwait( & sigset, &sig); …… }

39. POSIX Signal API pthread_sigmask( ) pthread_kill( ) sigwait sigset or sigaction

40. HOL about Signal 1. Install signal Handler 2. In MT, all thread share one handler. 3. setup different sigmask to direct signal delivery

41. Advanced Topic Thread Specific Data Unix Signals Advanced scheduling MT-Safe library

42. Advanced Scheduling Solaris Kernel scheduling RT, SYSTEM, TS POSIX defines 3 scheduling classes SCHED_OTHER : time sharing * SCHED_FIFO * SCHED_RR

43. API for scheduling pthread_attr_setschedpolicy SCHED_OTHER, SCHED_FIFO, SCHED_PR pthread_attr_setschedparam pthread_attr_setscope PTHREAD_SCOPE_PROCESS, PTHREA_SCOPE_SYSTEM pthread_attr_setinheritsched PTHREAD_INHERIT_SCHED, PTHREAD_EXPLICIT_SCHED priocntl

44. MT-Safe Library Thread Safety Level MT-Unsafe MT-Safe Alternative Call

45. Solaris Libraries libpthread.so (POSIX threads) pthread.h libthread.so (UI threads) sync.h, thread.h libposix4.so (POSIX semaphores) posix4.h

46. Compiling— s10-no flag needed Choose semantic s cc [flags] file - D_POSIX_C_SOURCE=199506L [-lposix4] -lpthread cc [flags] file -D_REENTRANT -D_POSIX_C_SOURCE=199506L [-lposix4] -lthread Mixed usage POSIX cc [flags] file -D_REENTRANT [-lposix4] -lthread UI

47. Some commands to Observe Use prstat(1) and ps(1) to monitor running processes and threads mpstat(1) to monitor context switch rates and thread migrations dispadmin(1M) to examine and change dispatch table parameters User priocntl(1) to change scheduling classes and priorities

48. Examining A Thread Structure # mdb -k R 21344 1 21343 21280 2234 0x42004000 ffffffff95549938 tcpPerfServer ffffffff95549938::print proc_t... p_tlist = 0xffffffff8826bc20 ffffffff8826bc20::print kthread_t

49. Thread Semantics Added to pstack, truss # pstack 909/2 909: dbwr -a dbwr -i 2 -s b0000000 -m /var/tmp/fbencAAAmxaqxb ----------------- lwp# 2 -------------------------------- ceab1809 lwp_park (0, afffde50, 0) ceaabf93 cond_wait_queue (ce9f8378, ce9f83a0, afffde50, 0) + 3b ceaac33f cond_wait_common (ce9f8378, ce9f83a0, afffde50) + 1df ceaac686 _cond_reltimedwait (ce9f8378, ce9f83a0, afffdea0) + 36 ceaac6b4 cond_reltimedwait (ce9f8378, ce9f83a0, afffdea0) + 24 ce9e5902 __aio_waitn (82d1f08, 1000, afffdf2c, afffdf18, 1) + 529 ceaf2a84 aio_waitn64 (82d1f08, 1000, afffdf2c, afffdf18) + 24 08063065 flowoplib_aiowait (b4eb475c, c40f4d54) + 97 08061de1 flowop_start (b4eb475c) + 257 ceab15c0 _thr_setup (ce9a8400) + 50 ceab1780 _lwp_start (ce9a8400, 0, 0, afffdff8, ceab1780, ce9a8400) truss -p 2975/3

50. Using dtrace to do sth # dtrace -n 'thread_create:entry { @[execname]=count()}' dtrace: description 'thread_create:entry ' matched 1 probe ^C sh 1 sched 1 do1.6499 2 do1.6494 2 do1.6497 2 do1.6508 2 in.rshd 12 do1.6498 14 do1.6505 16 do1.6495 16 do1.6504 16 do1.6502 16 automountd 17 inetd 19 filebench 34 find 130 csh 177

51. Resources Multithread programming on Sun.com devnull.eng

52. Want to Scale?— Multithread did that Vincent.Liu@sun.com