1. Everything you always wanted to know about threading....
... But were afraid to ask
Elliot H. Omiya (EHO) PRINCIPAL SDE
Windows Developer Experience

2. Agenda A short history of threading in Windows
Threading in the Windows Runtime
Threading and UI programming
Async

3. Short History of Threading in Windows
Anyone remember _beginthread(ex)?? Going back to VC and the early to mid-90’s (16-bit real mode Windows)
Eventually we sorted out CreateThread versus _beginthread issues and interactions with the CRT.
Inevitably this led to an explosion of threads in Windows programs.
Everyone started hand-rolling their own “thread pools” until we shipped ThreadPool in Windows NT.
Dedicated threads (i.e. CreateThread) are still popular but harder for application to manage.

4. ThreadPools
Simple: queue, “n” threads, and an algorithm to dequeue work and run it.
Not simple: the actual implementation
Job 1: queue (actually, this part is relatively simple)
If you have priorities, one queue per priority level is a fine implementation
Job 2: threads
First decision: how many threads?
One per core?

5. Worker threads for thread pool
One per core is fine unless you have: IO
Threads that can block on event/semaphore
A “busy” thread (waiting on IO or event) can do no more work
Result: you need a pool of threads greater than the # of cores because every worker thread can potentially block
... Not to mention pagefaults. Apps (user-mode) can’t possibly avoid being blocked.

6. Work types
Turns out you need “work types” (in the internal implementation)
TP_WORK
TP_WAIT
TP_IO
TP_TIMER
etc.
Basically these are “execution triggers” – how they start executing. You could also explicitly set up work items that you know do IO or will wait on an event/semaphore.
So the API just speaks of work items, but there is additional complexity in the actual implementation.

7. Thread explosion
There are several classic programming problems that can lead to thread explosion.
Simple example: web server servicing web requests:
“n” worker threads servicing incoming requests
One worker thread to read actual html file and associated resources from disk (into cache)
Simple programming error can lead to disastrous results.

8. Every worker thread is blocked on an event
Work Item
Work Item
Work Item
Work Item
Event
(web page read)
Work Item
Work Item
Work Item
Work Item
...
Work Item Queue
Worker Threads
Every worker thread is blocked on an event

9. Work Item
Work Item
Work Item
Work Item
Event
(web page read)
Work Item
Work Item
Work Item
Work Item
...
Work Item Queue
Worker Threads
But the worker thread that signals the event is in the queue! (Bad code!)

10. So you have to create another thread in the pool...
Work Item
Work Item
Work Item
Work Item
Event
(web page read)
Work Item
Work Item
Work Item
Work Item
...
Work Item Queue
Worker Threads
So you have to create another thread in the pool...

11. But you could have a large number of requests for the same web page...
Work Item
Work Item
Work Item
Work Item
Event
(web page read)
Work Item
Work Item
Work Item
Work Item
...
Work Item Queue
Worker Threads
But you could have a large number of requests for the same web page...

12. Result: thread explosion
Work Item
Work Item
Work Item
Work Item
Event
(web page read)
Work Item
Work Item
Work Item
Work Item
...
Work Item Queue
Worker Threads
Result: thread explosion

13. Thread explosion Turns out there are many variations of this problem.
Another example: freezing threads for a GC operation
We call this programming defect “popular internal dependency”
Solution: create a short (or long) delay – (yes, a heuristic)
Algorithm assumes the resource dependency will resolve (i.e. does not cover the programming error we previously described)
Alleviates the thread explosion during this dependency resolution

14. Fairness
Recap: “n” priority work queues and a system that does IO and has events/semaphores
OS has a highly efficient signaling mechanism: IO completion ports – kernel knows when IO completes. Can also be used for events.
So kernel knows immediately when a work thread that was blocked is ready to run (ReadyThread).
It can release work to the user mode side of the threadpool when it is idle.

15. Fairness Windows 7: TP_WORK always trumped TP_IO and TP_WAIT
So lots of TP_WORK could starve work items that unblocked (i.e. unfair)
A fairer algorithm blends TP_WORK, TP_IO, and TP_WAIT.
Servicing “IRP’s” is a kernel mode concept so the “worker thread factory” must be in kernel mode.

16. Work Item
Work Item
Work Item
Work Item
Work Item
Work Item
Work Item Queue
Work Item Queue
User
Kernel
Work
Work IO IO
Wait
...
Worker Thread Factory

17. Work Item
Work Item
Work Item
Work Item
Work Item
Work Item
Work Item Queue
Work Item Queue
User
Kernel
Work
Work IO IO
Wait
...
Worker Thread Factory
IRP’s complete

18. Transported and running
Work Item
Work Item
Work Item
Work Item
Work Item
Work Item
Transported and running
Work Item Queue
Work Item Queue
User
Kernel
Work
Work IO IO
Wait
...
Worker Thread Factory

19. Fairness
Fairness could only be implemented with kernel mode managing work items, IO, timers, and event waiters.
Bonus: IRP’s complete rapidly, we can batch up and do a single transport for multiple work items (fewer ring transitions)
Very difficult to balance throughput, resource management, and fairness in user mode only.
This is a C++ conference, why do you care?

20. Your unique value add is the diverse types of work you do in ISO C++
On Windows, think of how your library functions can benefit from a highly optimized threadpool.
On desktop, you can access CreateThread and the latest TP API’s
On WinRT, you only have the threadpool.
This is where you throw tomatoes at me.

21. I need dedicated threads I can control...
We hear this a lot. 
Not every workload is amenable to (hopefully short) chunks of work items.
“I have long running work”.
Examples:
Populate off-screen parts of game board (viewport scrolling optimization)
Lazy layout (optimize start-up performance)
(insert your favorite scenario here, you all have one or three)

22. Time-sliced
The Windows Threadpool does have a dedicated thread model: time- sliced.
Trade-off of latency versus throughput (and resource consumption)
Using work items is efficient, many work items processed by relatively few threads. (batched mode).
Using “time-siiced” has less latency but burns an expensive resource: a thread. (quantum mode)
Plug: This is a summary of Pedro Texeira’s excellent talk on the threadpool on MSDN. Many more details there.

23. WinRT threadpool
Windows::System::Threading::ThreadPool has support for:
Run a single work item - RunAsync(WorkItemHandler^)
Run a single work item based on timer - CreateTimer(TimerElapsedHandler^, delay)
Run a periodic work item based on a timer – CreatePeriodicTimer(TimerElapsedHandler^, period)
auto WorkItem = ref new WorkItemHandler( [&](IAsyncAction^ workItem) {
..... // background work
});
IAsyncAction ^ ThreadPoolWorkItem = Windows::System::Threading::ThreadPool::RunAsync(WorkItem);
Eventually, background work is reflected in UI, result(s) need to run on the correct UI thread: use CoreWindow Dispatcher

24. Agile Objects (are your friend)

25. In WinRT objects are objects
And threads are threads.
Forgive me while I digress into COM for a bit (you will forgive me ).
How many people in this audience know what a COM apartment is? (If you do, how do you like dealing with apartments?)
Better question: how many people want to know what a COM apartment is?

26. Traditional view of multi-threading
Objects can be accessed from any thread
The hard part is dealing with multi-thread contention, resource management, locking, deadlock prevention, etc.
The hard part is hard enough.
You already deal with “context” on a thread no matter what platform you run on: at a minimum, you deal with the context that is “UI”.
But, you can set up your code so that most objects can run on any thread.
And you want multi-threading to be no harder than this.

27. Agile Objects WinRT objects are agile by default.
This bit of “magic” (1) allows WinRT objects to just be “multi-threaded objects” (as you have traditionally known them).
They can run anywhere (hence the term “agile”) and you have the job of dealing with multi-threaded resource contention. Period.
This is the technique that allows you to ignore that there is a thing called an apartment.
There are also UI-affine objects in WinRT but you are used to dealing with these on whatever platform(s) you code for.
Actually not magic: uses FTM, lousy name, but it saves you from knowing anything about apartments. 
Deep dive on this from Martyn Lovell’s talk at build And our Channel9 video.

28. UI and ASTA

29. UI Threads
All mainstream systems bind a single thread to UI operations.
Implicitly or explicitly a UI thread has a “context” into which it is bound.
In the 90’s we created the notion of “apartment” which to this date almost no one understands. But just think of it as a context.
Very popular “other OS” says: It is strongly recommended not to update UI controls etc from a background thread (e.g. a timer, comms etc).  This can be the cause of crashes which are sometimes very hard to identify. In other words, don’t perform UI operations out of context.

30. UI Contexts
In this world of many threads, we’re still stuck with a single thread and UI.
It gets worse: turns out rather than multi-threaded races and deadlocks, UI threads can suffer from “re-entrancy”.
Re-entrancy: I’m in the middle of doing thing “A”, why am I all of a sudden doing thing “B”?
Re-entrancy is the cause of a large number of crashes and deadlocks.
UI has two models: re-entrancy == surprises, non-re-entrancy == potential deadlocks.

31. How to safely update UI 101
Option 1: Background threads call directly into UI threads. Only safe if you are REALLY careful. (See slide on re-entrancy).
Option 2: Background threads post notifications that UI threads process “when they are ready”. (Dispatcher model).
Can UI threads call directly into objects running on background threads? Only safe if you are somewhat careful.
Oh, and don’t take too long to update UI.
What is definition of “too long”? In the last few years it was all about frame rate. But now, it means “be immediately responsive to input”.

32. Pretend you have the chance to define a new platform... 
What choices do you make?
One of the first questions: Does your UI model allow re-entrancy?
Next: do you allow arbitrary threading?
How much synchronous UI update do you allow in your platform? (i.e. how much risk of “spinning donuts” do you want?)

33. We ended up calling that new platform: WinRT (“Windows Runtime”)
Choice #1: Non-re-entrancy (“ASTA”)
Choice #2: ThreadPool API’s but not arbitrary threading
Choice #3: No spinning donuts: async across the API surface
Choice #4: No arbitrary window creation (and you don’t have to know what an HWND is)
We still have single-threaded UI frameworks: HTML/CSS and XAML
And there are plenty of things you can do synchronously in these frameworks, e.g. toggle button state or update textbox/listbox/etc. contents.

34. Tying it together
Turns out that you have to tie together the following elements for the UI and programming model to work:
Window creation and behavior
Window event (message) processing
Dispatcher processing (remember those UI notifications we need?)
Call processing (incoming and outgoing)
Input
And you have to prioritize these with respect to each other.

35. The entity that ties these together is ASTA
ASTA == “UI context” (forget that one of the A’s is “apartment”)
One thread per window, windows are created by contract activations (e.g. click on a tile or share something from an app).
The main thread of a WinRT application runs in a multi-threaded context but is not a UI thread. i.e. the main thread is not a UI thread.
As a result, of course, there is a main UI window / thread.
ASTA’s are non re-entrant. When an outgoing call is in progress, an incoming call blocks.
This requires careful planning, but there are no surprises. (Re-entrancy is nearly always a surprise).

36. Async (no spinning donuts)
There has been a concerted effort to make UI responsive, this is key to the “fast and fluid” platform promise.
Every API was (and is) reviewed. Any synchronous call that takes longer than the low 10’s of milliseconds to execute must be async.
Every async API conforms to a uniform pattern.
The language projections (C++, C#/VB.Net, Javascript) all have built- in async support.

37. WinRT Async 101
All WinRT async operations are “hot start”, as soon as they are produced, they are running.
You can “attach” to a running async operation by supplying a completion handler (put_Completed).
When the operation completes, it fires a completion callback. The type of each callback is the result type of the async operation.
Of course, when you supply the completion callback, it may fire completion immediately, i.e. the operation completed already.

38. WinRT Async 101
When an async operation completes, its status is reported:
Complete 
Canceled (user request to cancel was honored)
Error 
WinRT async operations are a one-way state machine:
AsyncMethod => Running => Terminal State {Completed|Error|Canceled}
Processing always occurs on background thread (usually a TP work item)
Pattern is completely documented in code: see WRL’s async.h

39. Async and Agile
Async Operation objects are a good example of agile objects
Mostly designed to be called from UI threads
But do all of their work on background (TP) threads
As a result, async operation objects are agile (and therefore directly callable from UI threads)

40. Async support (C++ / PPL)
Language projections are key to productivity in producing WinRT apps.
“Natural and familiar” is a key design point: make the experience natural and familiar for the language that is being used.
For C++, the natural means of consuming async is based on the Concurrency Runtime (PPL).
The model is continuation-based (.then() + lambda/functor/function) with support for exception handling and cancellation.

41. Async and contexts
Consuming async operations is often done for the benefit of UI
create_task takes note of the originating context, i.e. what kind of thread made the original call. (use_default).
create_task will return the result of an async operation (if any) to that originating context by default.
C++ developers of course have control over this (e.g. you may want to “de-bounce” transitions back to the UI thread until the end of a series of continuations).
We’ll talk about CoreWindow’s Dispatcher method soon.

42. CoreWindow Dispatcher
One of the things that the ProcessEvents loop on a CoreWindow schedules is dispatcher work items.
CoreWindowDispatcher::RunAsync() schedules a work item to run on the UI thread associated with the CoreWindow.
Since the CoreWindow processing loop processes multiple types of items, a rich priority scheme is supported.

43. CoreWindow Dispatcher Priority
Normal priority (default) means: “run dispatcher callbacks in FIFO order, cooperative with input (and window management events)”. (Everyone runs cooperatively).
Low priority means: Dispatcher callbacks run when there is no input pending. (Input beats dispatcher, i.e. app is responsive to input).
Idle priority means: Dispatcher callbacks run when there is nothing else in any queue. (everything beats dispatcher).
High priority means: dispatcher callbacks run ahead of everything else. (dispatcher beats everything). Docs say: “don’t use this” (more on this in a second).

44. Rendering UI
This is not your father’s message loop. Everything used to happen on the UI thread: message processing, paint, computation, animation, alpha blending, etc.
As applications became more graphically intensive, everything became a slave to frame rate (60 fps == ms “window”). Get everything done by deadline. Very difficult model to get right.
WinRT render threads are separate from ASTA UI threads. Composition is separate from render.
Goodness: relieves pressure on the (precious) UI thread

45. Responsive UI
The “paint beat” is tied to the frame rate. The frameworks (WinJS and xaml) take care of this.
Most UI operations are “tweaks” to layout (change text in a text box, scroll, get image ready to render, etc.). These occur directly on UI thread.
Initial layout and large changes to layout (e.g. navigation) are critical and time-sensitive. You have to beat the next vsync.
If you can’t beat the next vsync (common) then you can create a transition animation while the next layout is being prepared. (independent animations do not run on the UI thread)

46. Responsive UI What does this have to do with threading?
CoreWindowDispatcher priority High serves a couple of different “system” purposes:
In WinJS – High priority is used for layout changes. Think: layout beats everything
In CoreApplication, the “app object” in every WinRT application, the suspend notification is delivered on UI threads via the Dispatcher (for apps, responding to Suspend is highly time critical)
If you choose to use High priority, remember that it is an extremely sharp knife: highly effective and dangerous when used improperly.

47. Tying it all back together
UI threads are created by the system and managed by CoreWindow processing loop.
Background threads run via WinRT ThreadPool or language projection components (sitting on WinRT ThreadPool).
CoreWindow Dispatcher schedules work on UI threads, with a rich priority scheme.
Responsive and useful app UI means balancing:
CPU utilization on background threads
UI thread processing that is responsive to input and never blocks (including never running long workloads)

48. Which means...
Responsive programs are sequences of async processing followed by a return to UI thread either by:
Direct context capture (return to point of origination); or
Explicit call to CoreWindow Dispatcher

49. Async Investments
Microsoft continues to invest heavily and innovate in the async space. C# introduced await keyword, dramatically simplifying async programming.
Await allows async consumption code to read in a more logical flow.
It looks like synchronous code, but the block of code following an await statement executes “later”.
Think you don’t need/want this? Go to Herb Sutter’s //build 2013 talk, and fast forward to about 50 minutes in.
And then make sure you attend Deon’s talk later today where we tell you Everything You Ever Wanted To Know about C++ await.

50. In Summary... Probably not everything you always wanted to know 
The valuable assets you have in your C++ code come to light in UI
And the threading and coding rules for UI are different than they were years ago (more input responsiveness, more async, etc.)
Optimize background processing in units of work and think carefully about the relative priorities of that work. (And don’t reinvent the threadpool!)
You do really want await (don’t miss Deon’s talk).