2016 OTN-TOOLBOX Presented by Marta Mihoff and Alex Nunes Assisted by Brian Jones, Sean Carey, Sara Colborne, Lenore Bajona

Start up Toolbox Open CMD window Navigate to the install folder (Desktop/OTN-toolbox) Execute command “vagrant up”

URLS R-Studio (user change to vagrant pw otn123) New R notebooks Python notebooks

Rstudio Changes Cosmetic only User changed to “vagrant”, password is the same “otn123” Removed the Virtual Machine GUI which none of you will notice File structure: programs are in folder “otn-toolbox” “data” folder accessible from inside “otn-toolbox” or on its own.

File Structure Home folder otn-toolbox folders Some new “notebooks” folders which you should ignore

Code Exists in folders The code is PUBLIC. You can see the code and change it in any way you want Changing files in these folders could break everything. You can recover by installing a new copy Recommend you change “copies”

DATA Folder The “data” folder exists independently from all the code It is accessible from RStudio or from Desktop/OTN-Toolbox NEVER delete or rename the data folder Copy files into the data folder to make them accessible to programs. In RStudio files should be saved into the “data” folder Folders will be lost or overwritten on an update if not in “data” folder.

R and PY Notebooks New wrappers for same code executed from RStudio GUI May find easier to use r-notebooks offer same set of functions available in Rstudio py-notebooks offer same set plus new functions In future all new functions developed will be done for py-notebooks

New Tools Available in PY-Notebooks only data_subsetting.ipynb Creates a subset of an input file based on a date range or a column value Useful when input file and run time are extremely large and long residence_index.ipynb Offers four methods to choose from. Mix and Match. interactive_residence_index.ipynb same as previous, different map interactive_residence_index.ipynb visual_detection_timeline.ipynb Creates an interactive time series from a detection file.

File Preparation OTN detection extracts are ready to go as is. VUE CSV export needs preparation: Latitude and longitude columns must be filled in Rename column receiver  station Rename column transmitter  catalognumber Rename column datetime  datecollected Column unqdetecid can be added with function add_uniquecid Data Subset If your file is very large use the subset tool: py_notebooks/data_subsetting.ipynb

Notebook: Execution Current cell is highlighted with a blue bar on LHS. When a cell is highlighted clicking the run button will execute the code in the cell.

Exercise: Filter suspect detections (half hour) Copy your detection file into your “data” folder Choose one of the three urls In py-notebooks open load_and_filter_detections.ipynbload_and_filter_detections.ipynb In r-notebooks open filter_driver.ipynbfilter_driver.ipynb In RStudio open filter_driver.r Need to do this to get a distance matrix

Filter tool What to fill in These are the parameters you need to fill in Filename detection_radius (use 400)

Exercise: Interval or Cohort data (15 min) For Interval data (one step) In py-notebooks or r-notebooks open interval_data_driver.ipynbinterval_data_driver.ipynb In RStudio open interval_data_driver.r For Cohort Data (two steps) In py-notebooks open detection_compression.ipynb first then cohort_data.ipynbdetection_compression.ipynb cohort_data.ipynb In r-notebooks open compress_driver.ipynb first then cohort_driver.ipynbcompress_driver.ipynbcohort_driver.ipynb In RStudio open compress_driver.r then cohort_driver.r

Interval/ Cohort What to fill in Interval: use outputs from Filter step detection_file <- 'detections.csv' #Detection file input name distance_matrix <-'detections_distance_matrix_v00.csv‘ OR for Cohort Compression: detection_file <- 'detections.csv‘ Cohort (need output from compression step) time_interval <- 6 compressed_file <- 'compressed_detections.csv'

Residence / Vis

Teach yourself to program Free open software Extremely powerful Standardized Python Python(x,y): rival to MATLAB and Rstudio PostgreSQL

How? Coursera and Code Academy Code Academy Python course: Rice University : An Introduction to Interactive Programming in Python Next session Sep University of Michigan : Programming for Everybody Next Session Oct Johns Hopkins : R Programming Part of the "Data Science" Specialization Next session Oct 6 Science" Specialization

Python solutions for common Science questions. Data Science from Scratch Joel Grus O’Reilly Media Inc 2015