1. CS 3214 Introduction to Computer Systems
Lecture 8
Godmar Back

2. Announcements Read Chapter 6 Project 2 due Fri, Sep 25
Need to pass at least “Firecracker” phase
Exercise 6 due Fri, Sep 25
Stay tuned for exercise 7 (tomorrow)
Midterm: Tuesday Oct 20
CS 3214 Fall 2009
5/26/2018

3. Code OPTIMIZATION Part 3
Some of the following slides are taken with permission from
Complete Powerpoint Lecture Notes for Computer Systems: A Programmer's Perspective (CS:APP)
Randal E. Bryant and David R. O'Hallaron
Part 3
Code OPTIMIZATION
CS 3214 Fall 2009
5/26/2018

4. Machine-INDependent Optimizations
CS 3214 Fall 2009
5/26/2018

5. Dangers of Recursion public class List { private List next;
List(List next) { this.next = next; }
int size() {
return next == null ?
1 : next.size() + 1; }
public static void main(String []av) {
List l = null;
for (int i = 0; i < 20000; i++)
l = new List(l);
System.out.println(l.size());
Exception in thread "main" java.lang.StackOverflowError
at List.size(List.java:7)
...
CS 3214 Fall 2009
5/26/2018

6. Optimizing Compilers & Recursion
“To understand recursion, one must first understand recursion”
Once you do, you know not to use it
Naïve use of recursion turns algorithms with O(1) space complexity into (entirely unnecessary) O(n) complexity
Means you run out of memory needlessly
Optimizing compilers will remove recursion if they can
For tail- and linear recursion
CS 3214 Fall 2009
5/26/2018

7. Recursion Removed list_size: pushl %ebp xorl %eax, %eax
movl %esp, %ebp
movl 8(%ebp), %edx
testl %edx, %edx
je L5
movl $1, %ecx
.p2align 4,,7
.L6:
movl (%edx), %edx
movl %ecx, %eax
leal 1(%ecx), %ecx
jne .L6
.L5:
popl %ebp
ret
struct node {
struct node *next;
void *value; };
int list_size(struct node *node) {
return node == 0 ?
0 : list_size(node->next) + 1; }
int list_size(struct node *node) {
int len = 0;
while (node != 0) {
node = node->next;
len++; }
return len;
CS 3214 Fall 2009
5/26/2018

8. Machine-Dependent Optimizations
CS 3214 Fall 2009
5/26/2018

9. Modern CPU Design ILP – Instruction Level Parallelism
Instruction Control
Address
Fetch
Control
ILP – Instruction Level Parallelism
Instruction
Cache
Retirement
Unit
Instrs.
Register
File
Instruction
Decode
Out-of-order:
executes operations in possibly different order than specified in assembly code
Super-scalar:
can do more than one instruction at once (resources permitted)
Pipelined:
some units start with the next operation before current one is finished
Operations
Register
Updates
Prediction
OK?
Execution
Functional
Units
Integer/
Branch
General
Integer FP
Add FP
Mult/Div
Load
Store
Operation Results
Addr.
Addr.
Data
Data
Data
Cache
CS 3214 Fall 2009
5/26/2018

10. What About Branches? Challenge
Instruction Control Unit must work well ahead of Exec. Unit
To generate enough operations to keep EU busy
When encounters conditional branch, cannot reliably determine where to continue fetching
80489f3: movl $0x1,%ecx
80489f8: xorl %edx,%edx
80489fa: cmpl %esi,%edx
80489fc: jnl a25
80489fe: movl %esi,%esi
8048a00: imull (%eax,%edx,4),%ecx
Executing
Fetching &
Decoding

11. Branch Outcomes
When encounter conditional branch, cannot determine where to continue fetching
Branch Taken: Transfer control to branch target
Branch Not-Taken: Continue with next instruction in sequence
Cannot resolve until outcome determined by branch/integer unit
80489f3: movl $0x1,%ecx
80489f8: xorl %edx,%edx
80489fa: cmpl %esi,%edx
80489fc: jnl a25
80489fe: movl %esi,%esi
8048a00: imull (%eax,%edx,4),%ecx
Branch Not-Taken
Branch Taken
8048a25: cmpl %edi,%edx
8048a27: jl a20
8048a29: movl 0xc(%ebp),%eax
8048a2c: leal 0xffffffe8(%ebp),%esp
8048a2f: movl %ecx,(%eax)

12. Branch Prediction Idea Guess which way branch will go
Begin executing instructions at predicted position
But don’t actually modify register or memory data
80489f3: movl $0x1,%ecx
80489f8: xorl %edx,%edx
80489fa: cmpl %esi,%edx
80489fc: jnl a25
. . .
Predict Taken
8048a25: cmpl %edi,%edx
8048a27: jl a20
8048a29: movl 0xc(%ebp),%eax
8048a2c: leal 0xffffffe8(%ebp),%esp
8048a2f: movl %ecx,(%eax)
Execute

13. Branch Prediction Through Loop
80488b1: movl (%ecx,%edx,4),%eax
80488b4: addl %eax,(%edi)
80488b6: incl %edx
80488b7: cmpl %esi,%edx
80488b9: jl b1
Assume vector length = 100
i = 98
Predict Taken (OK)
80488b1: movl (%ecx,%edx,4),%eax
80488b4: addl %eax,(%edi)
80488b6: incl %edx
80488b7: cmpl %esi,%edx
80488b9: jl b1
i = 99
Predict Taken
(Oops)
Executed
80488b1: movl (%ecx,%edx,4),%eax
80488b4: addl %eax,(%edi)
80488b6: incl %edx
80488b7: cmpl %esi,%edx
80488b9: jl b1
Read invalid location
i = 100
Fetched
80488b1: movl (%ecx,%edx,4),%eax
80488b4: addl %eax,(%edi)
80488b6: incl %edx
80488b7: cmpl %esi,%edx
80488b9: jl b1
i = 101

14. Branch Misprediction Invalidation
80488b1: movl (%ecx,%edx,4),%eax
80488b4: addl %eax,(%edi)
80488b6: incl %edx
80488b7: cmpl %esi,%edx
80488b9: jl b1
Assume vector length = 100
i = 98
Predict Taken (OK)
80488b1: movl (%ecx,%edx,4),%eax
80488b4: addl %eax,(%edi)
80488b6: incl %edx
80488b7: cmpl %esi,%edx
80488b9: jl b1
i = 99
Predict Taken (Oops)
80488b1: movl (%ecx,%edx,4),%eax
80488b4: addl %eax,(%edi)
80488b6: incl %edx
80488b7: cmpl %esi,%edx
80488b9: jl b1
i = 100
Invalidate
80488b1: movl (%ecx,%edx,4),%eax
80488b4: addl %eax,(%edi)
80488b6: incl %edx
i = 101

15. Branch Misprediction Recovery
80488b1: movl (%ecx,%edx,4),%eax
80488b4: addl %eax,(%edi)
80488b6: incl %edx
80488b7: cmpl %esi,%edx
80488b9: jl b1
Assume vector length = 100
i = 98
Predict Taken (OK)
80488b1: movl (%ecx,%edx,4),%eax
80488b4: addl %eax,(%edi)
80488b6: incl %edx
80488b7: cmpl %esi,%edx
80488b9: jl b1
80488bb: leal 0xffffffe8(%ebp),%esp
80488be: popl %ebx
80488bf: popl %esi
80488c0: popl %edi
i = 99
Learn not taken
Performance Cost
Misprediction on Pentium III wastes ~14 clock cycles
That’s a lot of time on a high performance processor

16. Branch Prediction Strategies
Most Intel processors use “forward branch not taken, backward branch taken” for first time a branch is seen
Even if done always, fits loops well – misprediction penalty paid only during last iteration
May be augmented by branch history buffer
Some architectures have “hinted branch” instructions where likely direction is encoded in instruction word
User-guided branch hinting
GCC: __builtin_expect(expr, value)
CS 3214 Fall 2009
5/26/2018

17. Branch Hinting #define likely(x) __builtin_expect(!!(x), 1)
#define unlikely(x) __builtin_expect(x, 0)
extern int g(int), h(int);
int f_likely_g(int v) {
return likely(v) ? g(v) : h(v); }
int f_likely_h(int v)
return unlikely(v) ? g(v) : h(v);
CS 3214 Fall 2009
5/26/2018

18. Branch Hinting
f_likely_h:
pushl %ebp
movl %esp, %ebp
movl 8(%ebp), %eax
testl %eax, %eax
jne .L7
movl $0, 8(%ebp)
popl %ebp
jmp h
.L7:
jmp g
f_likely_g:
pushl %ebp
movl %esp, %ebp
movl 8(%ebp), %eax
testl %eax, %eax
je L9
popl %ebp
jmp g
.L9:
movl $0, 8(%ebp)
jmp h
Aside: absent user-provided branch hinting, compiler guesses
which direction the branch goes and arranges code accordingly
CS 3214 Fall 2009
5/26/2018

19. Avoiding Branches On Modern Processor, Branches Very Expensive Example
Unless prediction can be reliable
When possible, best to avoid altogether
Example
Compute maximum of two values
14 cycles when prediction correct
29 cycles when incorrect
movl 12(%ebp),%edx # Get y
movl 8(%ebp),%eax # rval=x
cmpl %edx,%eax # rval:y
jge L11 # skip when >=
movl %edx,%eax # rval=y
L11:
int max(int x, int y) {
return (x < y) ? y : x; }

20. Avoiding Branches with Bit Tricks
Use masking rather than conditionals
Compiler still uses conditional
16 cycles when predict correctly
32 cycles when mispredict
int bmax(int x, int y) {
int mask = -(x>y);
return (mask & x) | (~mask & y); }
xorl %edx,%edx # mask = 0
movl 8(%ebp),%eax
movl 12(%ebp),%ecx
cmpl %ecx,%eax
jle L13 # skip if x<=y
movl $-1,%edx # mask = -1
L13:

21. Avoiding Branches with Bit Tricks
int bvmax(int x, int y) {
volatile int t = (x>y);
int mask = -t;
return (mask & x) |
(~mask & y); }
movl 8(%ebp),%ecx # Get x
movl 12(%ebp),%edx # Get y
cmpl %edx,%ecx # x:y
setg %al # (x>y)
movzbl %al,%eax # Zero extend
movl %eax,-4(%ebp) # Save as t
movl -4(%ebp),%eax # Retrieve t
Force compiler to generate desired code
volatile declaration forces value to be written to memory
Compiler must therefore generate code to compute t
Simplest way is setg/movzbl combination
Not very elegant!
A hack to get control over compiler
22 clock cycles on all data
Better than misprediction
Check before doing (x86_64 code might do already)

22. Machine-Dependent Opt. Summary
Pointer Code
Look carefully at generated code to see whether helpful
Loop Unrolling
Some compilers do this automatically
Generally not as clever as what can achieve by hand
Consider Branch Hinting
Exposing Instruction-Level Parallelism
Very machine dependent
Avoid spilling
Best if performed by compiler when tuning for a particular architecture

23. Role of Programmer Don’t: Smash Code into Oblivion Do:
How should I write my programs, given that I have a good, optimizing compiler?
Don’t: Smash Code into Oblivion
Hard to read, maintain & assure correctness
Do:
Select best algorithm
Write code that’s readable & maintainable
Use procedures, recursion, avoid built-in constant limits
Even though these factors can slow down code
Eliminate optimization blockers
Allows compiler to do its job

24. Profiling
“Premature Optimization is the Root of all Evil” – Donald Knuth
Where should you start optimizing?
Where it pays off the most: inner loops, where code spends most of its time
Amdahl’s Law
If you speed up the portion of a program in which it spends  of its time ( in [0, 1]) by a factor of k, the overall speedup of the program will be
See in book
CS 3214 Fall 2009
5/26/2018

25. Profiling Profilers monitor where a program spends its time
Callgraph profiling:
Instrument procedure entry & exit and record how often each function called and by whom
Statistical profiling
Periodically interrupt (sample) program counter, then extrapolate how much time was spent in function
gprof is a classic if crude tool for that purpose; but demonstrates the principle still used in modern tools such as Intel’s VTune or AMD’s CodeAnalyst
Compile with ‘-pg’ switch
CS 3214 Fall 2009
5/26/2018

26. Example: -pg strlen: pushl %ebp movl %esp, %ebp call mcount
movl 8(%ebp), %edx
movl $0, %eax
cmpb $0, (%edx)
je L4
.L5:
addl $1, %eax
cmpb $0, (%eax,%edx)
jne .L5
.L4:
popl %ebp
ret
int strlen(char *s) {
int len = 0;
while (*s++) {
len++; }
return len;
CS 3214 Fall 2009
5/26/2018

27. Example
void lower(char *s) { int i; for (i = 0; i < strlen(s); i++) if (s[i] >= 'A' && s[i] <= 'Z') s[i] -= ('A' - 'a'); }
void turnAtoX(char *s) { int i, l = strlen(s); for (i = 0; i < l; i++) if (s[i] == 'A') s[i] = 'X'; }
CS 3214 Fall 2009
5/26/2018

28. Example (cont’d) int main(int ac, char *av[]) { char buf[1024];
while (fgets(buf, sizeof buf, stdin)) {
turnAtoX(buf);
lower(buf);
fputs(buf, stdout); }
return 0;
gcc -O2 -pg lower.c -o lower perl -e '$i = 0; while ($i++ < ) { print "AASSAXXXXXXXXXXXXXXXXXXXXXFDFDBCDEF\n"; }' | ./lower > /dev/null gprof ./lower > lower.gprof
CS 3214 Fall 2009
5/26/2018

29. Example Output Sorted by time
% cumulative self self total
time seconds seconds calls us/call us/call name
strlen
lower
turnAtoX
main
Sorted by time
Provide both cumulative (including time spent in called functions) and time spent directly in function
Provides accurate call count
CS 3214 Fall 2009
5/26/2018

30. Call Graph gprof provides sections of callgraph for each function
<spontaneous>
[1] main [1]
/ lower [2]
/ turnAtoX [4]
Call Graph
main
main
turnAtoX
turnAtoX
lower
lower
gprof provides sections of callgraph for each function
These are not backtraces!
strlen
CS 3214 Fall 2009
5/26/2018

31. Call Graph gprof provides sections of callgraph for each function
/ main [1]
[2] lower [2]
/ strlen [3]
Call Graph
main
main
turnAtoX
lower
lower
gprof provides sections of callgraph for each function
These are not backtraces!
strlen
strlen
CS 3214 Fall 2009
5/26/2018

32. Call Graph gprof provides sections of callgraph for each function
/ turnAtoX [4]
/ lower [2]
[3] strlen [3]
Call Graph
main
turnAtoX
turnAtoX
lower
lower
gprof provides sections of callgraph for each function
These are not backtraces!
strlen
strlen
CS 3214 Fall 2009
5/26/2018

33. Call Graph gprof provides sections of callgraph for each function
/ main [1]
[4] turnAtoX [4]
/ strlen [3]
Call Graph
main
main
turnAtoX
turnAtoX
lower
gprof provides sections of callgraph for each function
These are not backtraces!
strlen
strlen
CS 3214 Fall 2009
5/26/2018

34. Benefits & Limitations
Can quickly direct attention to hotspots, especially in complex systems
Limitations:
Input dependent
Shows only this path
Program must run a minimum period for sampling to be accurate
CS 3214 Fall 2009
5/26/2018

35. Advanced Profilers Intel VTune, Code Analyst
Use performance counters of processor and relate to code
Examples: number of retired instructions, floating point ops, etc.
Cache behavior
“Whole-system” profilers, e.g. oprofile
Example showed only application code
Bottleneck may lie in libraries or OS
CS 3214 Fall 2009
5/26/2018