1. Introduction to Python memory management

2. Find the slides at: manuelschipper.com/slides.pdf

3. Manuel Schipper Software Engineer @ThyssenKrupp Presta

4. Goals Get a notion how Python handles memory. Gain awareness of techniques and tools that can help diagnose and fix memory problems in Python. Provide a starting point to further investigate the topic.

5. Disclaimers Everything covered is CPython specific. The focus is practical. No detailed explanations.

6. Let’s start with C code

7. int number = 2; VARIABLEMEMORY ADDRESS BINARY VALUE number0x7fff57d176f810

8. A memory address is a number (e.g. 0x7fff57d176f8) assigned to bytes in the computer’s memory (RAM). Programs can read and write to memory addresses.

9. Memory as a container of bytes MEMORY... 0x7fff59d5b6f89 0x7fff59d5b6f8a 0x7fff59d5b6f8e 0x7fff59d5b6f92... int a = 2; int b = 3; VARIABLEMEMORY ADDRESS BINARY VALUE a0x7fff59d5b6f8a10 VARIABLEMEMORY ADDRESS BINARY VALUE b0x7fff59d5b6f8e11

10. number = 3; VARIABLEMEMORY ADDRESS VALUE number0x7fff57d176f80011 int number = 2; VARIABLEMEMORY ADDRESS VALUE number0x7fff57d176f80010

11. #include int main() { printf("Storage size for int : %ld bytes", sizeof(int)); return 0; } Size of a C primitive int Storage size for int : 4 bytes

12. Size of a Python int >>> import sys >>> x = 5 >>> print sys.getsizeof(x), ‘bytes’ 24 bytes

13. Everything in Python is an object

14. >>> x = 5 A Python object of type int (5) is mapped to the name ‘x’ Names Objects Reference Python has names, references and objects

15. >>> x = 5 # 5 reference count is now 1 >>> y = 5 # reference count is now 2 >>> print id(x), id(y) # the object’s address 140685450723768 >>> id(x) == id(y) True MEMORY... 140685450723768... 5 x y Objects keep a reference count

16. When an object’s reference count falls to 0, the memory address is freed. The most basic Python memory management strategy is to properly manage object’s reference count.

17. Using the del statement: >>> del(x) Change the name’s reference : >>> x = None Managing object’s reference count

18. Object’s reference count will decrease when the reference goes out of the namespace’s scope: x = 10 #10 ref count is 1 def func(): # enclosing func creates separate scope y = 10 # 10 ref count is 2 return y # y is out of scope, ref count is now 1 func()

19. Use the Python scoping rules to your advantage Learn the Python scoping rules Avoid defining global names Encapsulate big memory allocations in functions

20. def memory_intensive_computation(): x = [2] * (2 * 10 ** 7) # big memory alloc #do something with x return x if __name__ == ‘__main__’: #do something memory_intensive_computation() #huge list #list is freed from memory Example: The reference count to the huge list will drop to 0 as soon as the function finishes executing.

21. Memory profiler import memory_profiler import time def memory_intensive_computation(): x = [2] * (2 * 10 ** 7) # big memory alloc return x @profile # profile this function def main(): time.sleep(1) # wait one sec memory_intensive_computation(): time.sleep(1) if __name__ == ‘__main__’: main()

23. Aside managing the reference count, there are many strategies to mitigate Python’s memory usage.

24. Further reading Saving memory when creating a large number of class instances. Saving memory when creating a large number of class instances. Managing memory in cyclic data structures. Putting limits on memory and CPU usage. Putting limits on memory and CPU usage Using the sys module to diagnose and control memory usage. Using the sys module to diagnose and control memory usage. Understand how much memory Python objects use. Consider memory view and buffer to keep a string’s attribute’s references. Consider memory view and buffer to keep a string’s attribute’s references. Consider using deques instead of lists. Consider using numpy for huge computations. Consider multiprocessing for special cases. Consider PyPy. Consider PyPy

25. There are many tools to aid in memory management.

26. Memory management tools Memory profiler Line by line memory inspection and plotting tool. Heapy Powerful memory leak debugger but with a high learning curve. Tracemalloc Python 3 library provides statistics on allocated memory blocks. Has the possibility to print allocation tracebacks and take before-after snapshots of memory to trace leaks. Objgraph Module that lets you visually explore Python object graphs.

27. Tracemalloc Makes it possible to connect an object back to where it was allocated. We can get detailed statistics on the objects dominating the memory usage and where in the source code they are allocated.

28. import tracemalloc tracemalloc.start() time1 = tracemalloc.take_snapshot() #first snapshot import waste_memory #program that keeps lots of references x = waste_memory.run() time2 = tracemalloc.take_snapshot() #second snapshot #compare snapshots and give trace stats = time2.compare_to(time1, 'traceback’) top = stats[0] #get the worst offender print('\n'.join(top.traceback.format())) Example from: Effective Python by Brett SaltkinEffective Python

29. File “waste_memory.py", line 29 self.x = os.urandom(100) File “waste_memory.py", line 35 obj = MyObject() File “waste_memory.py", line 42 deep_values.append(get_data()) File "trace.py", line 29 x = waste_memory.run()

30. Python is designed to let you code rapidly while hiding most of the underlying implementation details. While this is awesome, this lack of awareness can lead to problems in a production environment. Knowing the basics of how it works and what tools are there to help us can make a big difference when faced with a problem.

31. Thanks for listening. Find the slides at: http://manuelschipper.com