Vladimir Livshits University of Toronto EMME/2 Macro Programming With Release 9: Non Trivial Solutions to Trivial Problems Vladimir Livshits University of Toronto Emme/2 programming tool is a macro language. That means it was initially intended to record keystrokes. On the other hand emme/2 based models can have rather complex logic. They also often deal with complex data structures and requires clear user interface. Users often had to encode parts of emme/2 based models in external to emme/2 software. I think with release 9 emme/2 macro language reached the level when the need in external software can be often eliminated.

Exchange of information between modules, banks, and scenarios Some Major Problems Inherent to the Macro Language As a Keystroke Record: Exchange of information between modules, banks, and scenarios Handling complex data structures User interface There are problems that can’t be solved directly through the emme/2 modules, or that can be solved more efficiently using some macro language features rather than directly recording keystrokes in a module. I would divide these problems in three groups: - exchange of information between modules, banks, and scenarios - handling complex data structures - user interface

More than a Keystroke Record File processing Loops and text registers in compound lines Convolutions of complex data structures in text strings Handling of complex repetitive expressions See paper for: Graphical user interface in macro routines Useful macro utilities Interactions with operating system in macro routines During this presentation I will show some solution for the first two groups. Paper deals with many more issues and has in it examples of useful code From technical point of view I’ll talk about the following topics in emme/2 macro writing: -file processing as a solution to information exchange between modules -flow control in compound lines and utilisation of text strings as a solution to managing complex data structures in macros

Some Painful Stuff (1) Information exchange between: Modules - often the most difficult: 2.14  2.11, 2.41 2.11, 2.14 2.41, 2.24 2.41, 5.34 2.41, etc. Banks - a lot easier with 1.31 (can be done naturally via keystroke record!), But might involve complicated logic in macros (checking what exists, ensuring data compatibility, etc.) Scenarios - depends on a task at hand - can be straight forward, can be difficult There are some common trivial tasks that are fairly easy to do with excel or other external software or even manually in text editors. The very same tasks though used to become an annoyance if should have been done frequently (and many were) or if they should’ve been fully automated within emme/2 based models.

File Processing: Example 1 t nodes a* … centroids data … a … regular nodes data ... t links a … links data … file for links input in 2.11: t links d … links data ... file for nodes input in 2.11: d* … centroids data … d … regular nodes data ... Let’s illustrate the elementary file processing with a simple frequently performed task. We want to delete a sub network from emme/2 links and nodes tables. Assuming the sub-net can be selected in 2.14, we can punch it out in 2.14, reformat the batch output file changing all ‘add line’ commands to ‘delete line’ commands. MACRO

~# One compound row below till the '~$' sign ~<<214.Out ~# One compound row below till the '~$' sign ~+;~t1=~@;~?t1=t links;~$>links; ~t2=%%%t1.1%%%;~>>211n.in;~?t2=a;~"d%%%t1.-1%%%; ~?t2=d;~"a%%%t1.-1%%%;~?t2=t;~"%%%t1%%%;~>>;~$ ~# End of compound line ~:links ~+;~>>211l.in;~"%%%t1%%%;~>> ~+;~t1=~@;~?e;~$>exit;~t2=%%%t1.1%%%;~>>211l.in;~?t2=a;~"d%%%t1.-1%%%;~?t2=d;~"a %%%t1.-1%%%;~?t2=t;~"%%%t1%%%;~>>;~$ ~:exit ~+;~<< ~+;~>>211l.in;~" ;~>>;~>>211n.in;~" ;~>> The resulting macro will look like this. It takes only 8 macro rows to do the task using file processing features of release 9. Old file is opened for reading and lines from the old file get reformatted on-fly and inserted into a new input file.

File Processing: Example 2 ~/Select: 1. Delete all old station centroids ~/ 2. Add new station centroids ~/ 3. Remove all subway station centroids ~/ without parking lots and insert parking ~/ data in the retained centroids attributes ~/ 4. Connect station centroids to the road ~/ network ~/ 5. Check if station centroids have been ~/ inserted ~/ 6. Speed run - run options 1,2,3,4 using ~/ defaults ~/ 7. Change macro parameters ~/ 8. Display copyright notice Next is real life example with rather complex network modifications from the Greater Toronto Area model developed at the University of Toronto. It was presented in details at the regional Ontario emme/2 users conference. The station choice part of the model needed insertion / deletion of the station centroids and their proper connection to the integrated road and transit emme/2 network. The traditional approach would have been to identify manually points of connection and centroids' locations and create corresponding network files in order to input/output them in the model when station centroids addition / elimination is required by the model run. If the network was changed this manual tedious task would have been performed again. To make things more complicated in some cases only stations with certain capacities of parking lots should have been considered. The macro which fully automates all these tasks has thousands of rows of code. The slide shows the resulting macro front interface.

Example Schema for Automated Road Network Modifications Mark up network elements 2.41 2.14 2.43 2.12 Produce input files for file processing 2.41 2.14 2.12 Experience shows that most of the automated network modifications follow the same schema. First network elements are marked for subsequent modifications, then files for subsequent reformatting are produced and finally a macro procedure reformats the files and produces input for 2.41 or 2.14. Similar schema applies for the transit networks using modules 2.22, 2.24, 5.34 and 2.41 (5.34 actually has been use to mark up nodes nearest to the rail stations in the station choice macro). Reformat files in a macro MACRO Read files in a bank 2.41 2.11

Main Principles of File Processing in Emme/2 Macro Language Release 9 Utilisation of text registers to exchange information between different files Utilisation of sub-strings (text substitutions) to reformat text lines Careful closing of files opened for writing In a compound line in order to avoid undesirable writing to the files Utilisation of compound lines For efficient handling of file processing procedures Utilisation of the t0 text register and positional parameters %1% … %32% For reformatting and for exchanging text strings between files Text registers serves as a buffer between file opened for reading and output file opened for writing. We manipulate text strings in the buffer with text substitutions (using sub-strings). Entire link or node table can be processed in just one compound macro line.

Some Painful Stuff (2) Processing complex data structures: No ‘modular programming’ or ‘object-oriented programming’ elements in the macro language Many modeling procedures are based on data structure rather then operations (hierarchical logit models estimations, counts stations and screen lines) The second problem inherent to a macro language. Full scale programming languages often have built in features for modular programming or even for object-oriented programming.

Expressions in Compound Lines and Convolution of Complex Data Structures in Text Strings The common problem of the macro languages is difficulty handling complex data structures. For example hierarchical structures. Let see ways around this in emme/2 macro language. Let's assume for the example purposes that we have three screen lines {1,2,3}, two stations {001,002} for the each of screen lines, and one direction {01}. We want to punch out volumes for all the stations for the direction on the first screen line.

Complex Data Structures: Example 1 station code Input file: inode jnode result 60233 60239 0101002 60239 60233 0101001 Output file inode jnode result 60233 60239 285 60239 60233 229 ~<macro_name 0102 001002 As an input file we have a file with three fields: i node, j node and station code. Hierarchical data tree is convoluted into a text string where first two digits represent screen line number, second two digits represent direction and the last three digits are the station code. the example macro takes string of stations and string of screen lines as macro arguments and extracts volumes for the relevant links. volume

~+;~t1=%%%1%%%;~t2=%%%2%%% ~# String of screen lines: %t1% ~# String of stations: %t2% batchout=output m=2.41 ~# Change module parameters ~+;5;n;y;y;n;n;n ~# Input stations codes in tmpl1 ~+;3;tmpl1;*;;~* input file name;n;y ~+;1;n ~# One compound row below till ‘0’: ~+;~?t2=;~$>l;volau*(tmpl1==%%%t1.2%%%01%%%t2.3%%%)+; ~t2=%%%t2.-3%%%;~?!t2=;~$;0 ~+;;!ul1=0;;3 ~:l q batchout=^ In the macro above input file with stations 001 and 002 for the screen line 01 direction 01 (0101001 and 0101002) is read in module 2.41. Station codes are given as macro arguments. this way the procedure itself is not limited to a particular station or a particular arrangement of the registers and can be plugged into other macro easily. Stations' codes are inputted into tmpl1. that eliminates a need for a permanent link attribute to store them. Volumes for stations 001 and 002 are punched out in 'output' file. the loop in the compound line includes expression for the network calculation and is based on a text string with stations' codes 001002. the first station is processed as by taking first three characters of the string. then these characters are dropped from the string and the next three are processed.

Main Principles of Convoluting Complex Data Structures for Macro Processing Utilisation of text strings — offers great flexibility for reflection of different data structures (hierarchical data, lists, etc.) and different data types (characters as well as numeric data) in emme/2 macros. Utilisation of text substitutions — handling of the different data structures with text strings can be efficiently done with text substitutions, especially with text substitutions in text registers. Flow control based on the encoded in a text string data structure — text registers with text substitutions can be used as loop counters and / or criteria.

Conclusions Complex emme/2 based models can be efficiently encoded entirely in emme/2 software File processing in emme/2 macro language offers additional possibilities to automate complex network or matrix manipulations outside of matrix or network editors Processing of complex data structures (e.g. hierarchical matrix organisations in nested logit models) can be efficiently done by convoluting information about the data structures in text strings and including expressions in compound line loops

Data Management Group INRO Consultants Inc. University of Toronto Acknowledgements Data Management Group INRO Consultants Inc. University of Toronto