1. Autumn School Dynamic MSM16-18 November 2015 | L-Esch-sur-Alzette Slide 1 Aggregates, output and other basic tools

2. Autumn School Dynamic MSM16-18 November 2015 | L-Esch-sur-Alzette Slide 2 Topics 1.Aggregates simple groupby 2.Output console CSV charts 3.Misc 1.Simulation - "init phase" 2.(periodic) Globals 3.Macros 4.Functions local variables

3. Autumn School Dynamic MSM16-18 November 2015 | L-Esch-sur-Alzette Slide 3 Output

4. Autumn School Dynamic MSM16-18 November 2015 | L-Esch-sur-Alzette Slide 4  count([condition])  count()  count(age >= 18 and not gender)  sum(expr[, filter=condition][, skip_na=True]) avg, std, min, max, percentile, median, gini  avg(age)  avg(age, filter=gender)  gini(income, filter=age >= 18, skip_na=False)  all(condition1[, filter=condition2]) any  all(income >= 0, filter=age >= 18)  any(age 120) Aggregates - Simple

5. Autumn School Dynamic MSM16-18 November 2015 | L-Esch-sur-Alzette Slide 5  groupby(expr1[, expr2, expr3,...] [, expr=expression] [, filter=filterexpression] [, percent=True], [, pvalues=possible_values]) group individuals by their value for the given expressions, compute an expression for each group  groupby(agegroup, gender)  groupby(gender)  groupby(agegroup, gender, workstate, percent=True)  groupby(agegroup, gender, filter=MARRIED)  groupby(agegroup, gender, expr=avg(income))  groupby(agegroup, gender, expr=avg(income), filter=age >= 18) Aggregates - Groupby (aka “pivot table”) agegroup | gender | | | False | True | total 0 | 165 | 177 | 342 5 | 263 | 264 | 527 10 | 278 | 290 | 568 15 | 293 | 347 | 640 20 | 294 | 303 | 597 … | … | … | … 40 | 375 | 359 | 734 45 | 362 | 408 | 770 50 | 681 | 694 | 1375 60 | 529 | 532 | 1061 70 | 428 | 340 | 768 80 | 310 | 182 | 492 90 | 83 | 30 | 113 100 | 3 | 1 | 4 total | 5049 | 4951 | 10000

6. Autumn School Dynamic MSM16-18 November 2015 | L-Esch-sur-Alzette Slide 6 Output

7. Autumn School Dynamic MSM16-18 November 2015 | L-Esch-sur-Alzette Slide 7  show(expr1[, expr2, … ]) evaluates the expression(s) and shows the result  show(groupby(age, gender, filter=age <= 10))  show(count(age >= 18))  show("not dead", count(not dead), "\n", "average age", avg(age, filter=not dead))  "skip_shows: True" simulation option => hide them all  qshow(expr1[, expr2, … ]) evaluates the expression(s) and shows the result next to the expression(s)  qshow(count(), avg(age), std(age)) Output - console

8. Autumn School Dynamic MSM16-18 November 2015 | L-Esch-sur-Alzette Slide 8  dump(expr1[, expr2, …, filter, missing, header]) creates table with the given expressions evaluated over some individuals of the dataset filter - take a subset of individuals missing - value to transform ‘nan’ values into (default: no transformation) header - True/False, whether or not to add column names  show(dump(age, partner.age, gender, filter=id < 10)) id | age | partner.age | gender 0 | 27 | -1 | False 1 | 86 | 71 | False 2 | 16 | -1 | True 3 | 19 | -1 | False 4 | 27 | 21 | False 5 | 89 | 92 | True 6 | 59 | 61 | True 7 | 65 | 29 | False 8 | 38 | 35 | True 9 | 48 | 52 | True Output - dump

9. Autumn School Dynamic MSM16-18 November 2015 | L-Esch-sur-Alzette Slide 9  csv(expr1[, expr2, …, suffix, fname, mode]) write to CSV-files fname - (path and) name of file to save to suffix - equivalent to fname = '{entity}_{period}_{suffix}.csv' mode - 'w' (default) or 'a' (overwrite or append)  csv(avg(age))  csv(period, avg(age), fname='avg_income.csv', mode=‘a’)  csv(dump(age, gender), suffix='dump')  csv(groupby(age, gender, filter=age <= 10)) Output - CSV

10. Autumn School Dynamic MSM16-18 November 2015 | L-Esch-sur-Alzette Slide 10  Longitudinal not (currently) supported  Slow on large data  usually a good idea to use groupby instead of plotting millions of points  Arguments  fname - (path and) name of file to save to if not specified, shows interactive viewer supports.png &.pdf  suffix - equivalent to fname = '{entity}_{period}_{suffix}.png'  colors: ['r', 'g', 'b'] (see matplotlib colors documentation)  grid: True | False Output - charts - generic

11. Autumn School Dynamic MSM16-18 November 2015 | L-Esch-sur-Alzette Slide 11 bar(groupby(agegroup, eduach)) Output - charts - bar

12. Autumn School Dynamic MSM16-18 November 2015 | L-Esch-sur-Alzette Slide 12 plot(groupby(agegroup, expr=count(not gender)), 'o--', groupby(agegroup, expr=count(gender)), 's-.', groupby(agegroup), '*-', colors=['r', 'g', 'b']) Output - charts - plot

13. Autumn School Dynamic MSM16-18 November 2015 | L-Esch-sur-Alzette Slide 13 stackplot(groupby(age, eduach)) Output - charts - stackplot

14. Autumn School Dynamic MSM16-18 November 2015 | L-Esch-sur-Alzette Slide 14 pie(groupby(eduach), labels=['Lower secondary', 'Upper secondary', 'Tertiary'], explode=[0.1, 0, 0]) Output - charts - pie

15. Autumn School Dynamic MSM16-18 November 2015 | L-Esch-sur-Alzette Slide 15 boxplot(groupby(eduach, expr=age, filter=eduach != -1), notch=True) Output - charts - boxplot

16. Autumn School Dynamic MSM16-18 November 2015 | L-Esch-sur-Alzette Slide 16 - area: 3.1415 * (4 + 1.5 * children.count()) ** 2 - scatter(age, salary, c=eduach, s=area, grid=True) (dataset with 200 persons) Output - charts - scatter

17. Autumn School Dynamic MSM16-18 November 2015 | L-Esch-sur-Alzette Slide 17 Miscellaneous

18. Autumn School Dynamic MSM16-18 November 2015 | L-Esch-sur-Alzette Slide 18 simulation: init: - household: [init_region_id, household_composition] processes: - person: [ageing] […] start_period: 2016 periods: 2  Init  execute the processes in start_period - 1 (here 2015)  can be used to initialize variables, output files, …  can be used to do static (parts of the) simulation  init functions often contain "init" in their name Simulation - "init phase"

19. Autumn School Dynamic MSM16-18 November 2015 | L-Esch-sur-Alzette Slide 19  Building csv files progressively entities: person: processes: create_files: - csv('period', 'average', 'std dev', fname='age_aggregates.csv') output: - csv(period, avg(age), std(age), fname='age_aggregates.csv', mode='a') simulation: init: - person: [create_files] processes: - person: [output] Output with init phase example periodaveragestd dev 200242.037429.97504 200335.7295131.73392 200431.5097132.06869 200528.6107431.88439 200626.4910231.49843

20. Autumn School Dynamic MSM16-18 November 2015 | L-Esch-sur-Alzette Slide 20 entities: person: processes: retirement: - workstate: if(age >= 65, RETIRED, workstate)  Globals  variables that do not relate to any particular entity  periodic globals can have a different value for each period  in CAPITAL letters by convention (periodic) globals globals: periodic: # PERIOD is implicit - RETIREMENTAGE: float entities: person: processes: retirement: - workstate: if(age >= RETIREMENTAGE, RETIRED, workstate)

21. Autumn School Dynamic MSM16-18 November 2015 | L-Esch-sur-Alzette Slide 21 Macros (easier to read & maintain)  Macros  defined on entity level  re-evaluated wherever they appear  can contain any expression/value (but not other macros)  in CAPITAL letters by convention processes: ageing: - age: age + 1 csv_output: - csv(period, count(age = RETIREMENTAGE) / count(age >= 15 and age < RETIREMENTAGE), fname='dependency_ratio.csv', mode='a') processes: ageing: - age: age + 1 csv_output: - csv(period, count(not ACTIVEAGE) / count(ACTIVEAGE), fname='dependency_ratio.csv', mode='a') person: fields: - age: int […] macros: ACTIVEAGE: (age >= 15) and (age < RETIREMENTAGE)

22. Autumn School Dynamic MSM16-18 November 2015 | L-Esch-sur-Alzette Slide 22 processes: ageing: - age: age + 1 - juniors: 5 * trunc(age / 5) - plus50: 10 * trunc(age / 10) - agegroup: if(age < 50, juniors, plus50)  Local variables  e.g. juniors, plus50 in the ageing function  no need to declare them  temporary and not stored: only available in the function they are defined Functions local variables

23. Autumn School Dynamic MSM16-18 November 2015 | L-Esch-sur-Alzette Slide 23 Exercises (demo03.yml)

24. Autumn School Dynamic MSM16-18 November 2015 | L-Esch-sur-Alzette Slide 24 1. Open demo03.yml in the editor 2. Open the input dataset (demo.h5) in ViTables (F9)  check which fields we have for the person entity 3. Take one individual (e.g. id == 42) and check how he evolved during the simulation (simulation.h5)  e.g. his age increased by one each year 4. In the input dataset (demo.h5), find all persons aged 65+ who are still working (workstate equals 1)  use == to test for equality (as in a LIAM2 model)  use parentheses around conditions in ViTables queries  use "&" to combine conditions (instead of "and" in a LIAM2 model) e.g. (condition1) & (condition2)  can usually be done using LIAM2 interactive console not in this case (cannot use fields not declared). Exercises - ViTables

25. Autumn School Dynamic MSM16-18 November 2015 | L-Esch-sur-Alzette Slide 25 5. initialize agegroup in 2015 and check the result with ViTables (F9) 6. simulate 20 years, instead of 2 in period 2020… 7. … display the number of persons by agegroup 8. … display the number of persons old enough to be retired by sex 9. … for persons old enough to be retired, display the percentage of each gender 10. … display the proportion of males by agegroup 11. … display a bar chart of those proportions Exercises - init and groupby

26. Autumn School Dynamic MSM16-18 November 2015 | L-Esch-sur-Alzette Slide 26 12. demo03.yml already creates "dependency_ratio.csv"  add two columns to that file with the child dependency ratio (0-14 / 15-64) and the aged dependency ratio (65+ / 15-64) for each year 13. create a new csv file called "average_agediff.csv" with the difference of average age between men and women for each year Exercises - progressive csv files