1. Web-Based Open Source GIS: Decision Support Tools Explaining the Software Stack Presented by Aaron Racicot – GIS Programmer aaronr@ecotrust.org April 19th, 2006 A Citizen of Salmon Nation

2. Outline Introduction – Personal and Ecotrust Software Stack – Desktop, Web, DST Workflow & Dataflow Raster Tools – Mapserver->GRASS Vector Tools – Mapserver->PostGIS Glue – AJAX, OGR/GDAL,… Installation and tool selection Where is it all going?

3. Who am I… B.S. Computer Science Split Personality M.S. Environmental Science Open Source User/Developer GIS Programmer

4. Ecotrust - Salmon Nation

5. Software Stacks… Desktop Generic StackFOSS4G StackGrouping ApplicationQGis, Grass, OSSIM, JUMP, UDig User Interface Application Dev. Environment Eclipse, QT, OpenGL High Level UtilitiesGeoTools, PostGISData Serving High Level Scripting Languages PHP, PERL, Python Low Level UtilitiesShplib, GEOS, OGR/GDAL, PostGIS, R-Statistics, GMT Data Processing Low Level Languages C, C++, Java, FortranSystem Software Operating System Linux, Darwin, Cygwin Hardware Drivers 32-64 bit processor drivers Desktop Software Stack

6. Software Stacks... Server Server Software Stack Generic StackFOSS4G StackGrouping Client Side BrowserFirefox, Safari, NetscapeUser Interface Client Side Scripting Java Script, Java Applets ~~~~~~~~~~~WEB~~~~~~~~~~~ Server Side Application Chameleon, Cartoweb3, CustomData Serving High Level UtilitiesMapserver, PCO, Grass Low Level UtilitiesShplib, GEOS, OGR/GDAL, PostGIS, R-Statistics, GMT Data Processing High Level Scripting Languages PHP, PERL, Python Low Level Languages C, C++, Java, FortranSystem Software Operating System/Drivers Linux, Darwin, Cygwin Hardware Drivers32-64 bit processor drivers

7. Software Stack... DSTs (Ecotrust) DST Software Stack Generic StackFOSS4G StackGrouping Client Side Browser Firefox, Safari, Netscape User Interface Client Side Scripting Chameleon/Custom ~~~~~~~~~~~WEB~~~~~~~~~~~ Server Side Application Chameleon/Custom Data Serving High Level Utilities Mapserver, GRASS, PostGIS Low Level Utilities Shplib, GEOS, R-Statistics, GMT, OGR/GDAL Data Processing High Level Scripting Languages PHP, PERL Low Level Languages C, C++, Java, Fortran System Software Operating System/Drivers Linux (Fedora) Hardware Drivers 32 bit processor drivers

8. Work Flow – What OSGIS is good at Web User Request Web User Response Static Data Storage Web Services Data Gathering Data Formatting Data Processing Map Formatting Map Production The Problem Is The Arrows! Connecting a web request to server side GIS analysis is tricky Real-Time Web Decision Support Tools

9. Data Flow... DSTs (Ecotrust)

10. General PHP Mapscript dl('php_mapscript.so'); dl('php_proj.so'); $oMap = ms_newMapObj(MAPFILE); // Set the data directory $oMap->set("shapepath", $session_tmp_dir."data"); // Now we need to set the save path and image URL to a session based directory... $oWeb = $oMap->web; $oWeb->set("imagepath",$session_tmp_dir); $oWeb->set("imageurl",$session_tmp_dir); // Explode the x,y extents $box_coords = explode(" ", $http_form_vars["imgbox"]); $point_coords = explode(" ", $http_form_vars["imgxy"]);

11. PHP Mapscript - Zooming // Check if the box is to small and we should just use the x,y if ((abs($box_coords[0] - $box_coords[2]) < 5) && (abs($box_coords[1] - $box_coords[3]) < 5)) { zoomPoint( $oMap, ZOOMOUT_FACTOR, $point_coords[0],$point_coords[1] ); } else { zoomRectangle( $oMap, $box_coords[0], $box_coords[1], $box_coords[2], $box_coords[3] ); }

12. PHP Mapscript - Rendering $img = $oMap->draw(); $url = $img->saveWebImage(); $oMap->scalebar->set("transparent", MS_TRUE); $img_scale = $oMap->drawScaleBar(); $url_scale = $img_scale->saveWebImage(); $oMap->legend->set( "template", APP_PATH."legend_template.html" ); $params = array(); $html_legend = $oMap->processLegendTemplate($params); $img_ref = $oMap->drawReferenceMap(); $url_ref = $img_ref->saveWebImage(); $oMap->save($session_tmp_dir."map_tmp.map");

13. PHP Mapscript - Integration <applet codebase="/java/jBox" archive="jBox.jar" code="jBox.class" width="800" height="600" name="jBox" MAYSCRIPT> "> " ALT="Ref">

14. Raster based DST Siuslaw Watershed Restoration Initiative Data Gathering Data Formatting Data Processing PHP Mapscript GRASS

15. Raster based DST $string = host db_str usr_str; $connection = pg_connect($string); if (!$connection){ pg_close($connection); echo "Error:Cannot connect to database "; } else { echo "Connected to the SCA database \n"; } $res_pg_exec = pg_exec($connection,$exec_str); PostgreSQL Type Interface for GRASS GIS

16. Raster based DST – GRASS Class class Grass_GIS { var $data_dir; // Where the GRASS data dir is var $rc_dir; // Where the GRASS rc dir is var $bin_dir; // Where the GRASS bin dir is var $location; var $mapset; function Grass_GIS($data_dir, $rc_dir, $bin_dir, $location, $mapset) { $this->data_dir = $data_dir; $this->rc_dir = $rc_dir; $this->bin_dir = $bin_dir; $this->location = $location; $this->mapset = $mapset; } function Grass_GIS_copy($grass_gis) { $this->data_dir = $grass_gis->data_dir; $this->rc_dir = $grass_gis->rc_dir; $this->bin_dir = $grass_gis->bin_dir; $this->location = $grass_gis->location; $this->mapset = $grass_gis->mapset; } function set_location($location) {$this->location = $location;} function set_mapset($mapset) {$this->mapset = $mapset;} function set_data_dir($data_dir) {$this->data_dir = $data_dir;} function set_rc_dir($rc_dir) {$this->rc_dir = $rc_dir;} function set_bin_dir($bin_dir) {$this->bin_dir = $bin_dir;} function run_command($cmd){ $std_output = ""; $grass_exec = ""; // Check that all the required info is available.. if ($this->data_dir != NULL && $this->rc_dir != NULL && $this->bin_dir != NULL && $this->location != NULL && $this->mapset != NULL){ // Here we will write out the script to run... $fp_data = fopen($this->data_dir."/grass_run.sh","w"); // write out text header fwrite($fp_data,"export GISBASE=".$this->bin_dir." \n"); fwrite($fp_data,"export GISDBASE=".$this->data_dir." \n"); fwrite($fp_data,"export GISRC=".$this->rc_dir."/.grassrc6 \n"); fwrite($fp_data,"export ETC=".$this->bin_dir."/etc \n"); fwrite($fp_data,"export PATH=".$this->bin_dir."/bin:".$this->bin_dir."/scripts:".$this->bin_dir. "/garden/bin:\$PATH:/usr/bin:/usr/local/bin/ \n"); fwrite($fp_data,"export LOCATION_NAME=".$this->location." \n"); fwrite($fp_data,"export MAPSET=".$this->mapset." \n"); fwrite($fp_data,"export LOCATION=".$this->data_dir."/".$this->location."/".$mapset." \n"); fwrite($fp_data,"\n"); fwrite($fp_data,$cmd."\n"); fflush($fp_data); fclose($fp_data); $fp_data = fopen($this->rc_dir."/.grassrc6","w"); fwrite($fp_data,"GISDBASE: ".$this->data_dir."\n"); fwrite($fp_data,"LOCATION_NAME: ".$this->location."\n"); fwrite($fp_data,"MAPSET: ".$this->mapset."\n"); fwrite($fp_data,"PAINTER: ppm\n"); fwrite($fp_data,"MAPLP: stuff.ppm\n"); fflush($fp_data); fclose($fp_data); $grass_exec = sprintf("chmod u+x ".$this->data_dir."/grass_run.sh"); exec($grass_exec,$std_output,$std_error); $grass_exec = sprintf($this->data_dir."/grass_run.sh"); exec($grass_exec,$std_output,$std_error); }

17. Raster based DST – PHP GRASS define("GRASSDATA_DIR", "/var/www/html/apps/siuslaw_target/data/GRASSDATA/"); $location_name = "OR_siuslaw; $mapset_name = "land_targets; $session_grass_dir = $session_tmp_dir."/GRASSDATA"; // Make a Dir to work in echo " Make a directory to work in inside this session "; $dir_exec = sprintf("mkdir ".$session_grass_dir); exec($dir_exec,$arr,$err1); $dir_exec = sprintf("mkdir ".$session_grass_dir."/rc"); exec($dir_exec,$arr,$err1); $template_path = GRASSDATA_DIR."siuslaw_epa_template"; echo " Copying over new GRASS GIS template "; $dir_exec = sprintf("cp -R ".$template_path." ".$session_grass_dir."/".$location_name); exec($dir_exec,$arr,$err1); // Make a MAPSET echo " Setting up the siuslaw specific area "; $dir_exec = sprintf("mkdir ".$ session_grass_dir."/".$location_name."/".$mapset_name); exec($dir_exec,$arr,$err1); // Copy over the projection info from the PERMANENT area echo " Setting projections "; $dir_exec = sprintf("cp ".$template_path."/PERMANENT/PROJ* ".$ session_grass_dir."/".$location_name."/".$mapset_name."/"); exec($dir_exec,$arr,$err1); $grass_session = new Grass_GIS($session_grass_dir, $session_grass_dir."/rc", "/usr/local/grass-6.0.0", $location_name, $mapset_name); $grass_session->run_command("g.list type=rast");

18. Raster based DST – AML GRASS $arr = $grass_session->run_command("g.region rast=owng_grass@PERMANENT"); echo " Starting to process the Siuslaw data \n"; //AML step //AH2 = (ah_idx + 1) * 10 echo " Creating ah2_idx_grass... anchor habitat priorities \n"; $arr = $grass_session->run_command("r.mapcalc ah2_idx_grass = \(\(ah_idx_grass@PERMANENT+1\)*10\)"); //AML step //own_idx = con(owng eq 31, 40,con(owng eq 83,30,con(owng eq 95,10,20))) echo " Creating own_idx_grass... ownership priorities \n"; $arr = $grass_session->run_command("r.mapcalc own_idx_grass = if\(owng_grass@PERMANENT==31,40,". "if\(owng_grass@PERMANENT==83,30,if\(owng_grass@PERMANENT==95,10,20\)\)\)"); //AML step //reclass tpa (use slice) and setnull trees younger than 15 years //veg_idx = setnull(age_g lt 15, slice(tpa_g,eqarea,40)) echo " Reclasification of tpa_g... to 0-40 \n"; $arr = $grass_session->run_command("r.rescale.eq input=tpa_g_grass@PERMANENT ". "output=tpa_rescale to=0,40"); echo " Creating the veg_idx... trees over ".$tpa_age." years of age \n"; $arr = $grass_session->run_command("r.mapcalc veg_idx_grass = ". "if\(age_g_grass@PERMANENT\>=".$tpa_age.",tpa_rescale,null\(\)\)"); //AML step //FING = setnull(rip_stab eq 0, (ah2 + veg_idx + own_idx +..\infra\rd_d_inv)) echo " Creating FING... the sum of ah2_idx_grass+own_idx_grass+veg_idx_grass+rd_d_inv_grass \n"; $arr = $grass_session->run_command("r.mapcalc fing_grass = ". "if\(rip_stab_grass@PERMANENT==0,null\(\),". "\(ah2_idx_grass+own_idx_grass+veg_idx_grass+rd_d_inv_grass@PERMANENT\)\)")\;

19. Raster Example – TREESystem Mapserver on the front end… GRASS on the back end… Web-based Query

20. Raster based DST - Mapfile LAYER NAME "Fishery Rev" STATUS DEFAULT DATA "./GRASSDATA/CAL_ocean2/FISHERY/cellhd/fishery_rast_rev_zone_8bit" TYPE RASTER PROJECTION "proj=aea" "datum=NAD27" "ellps=clrk66" "lat_1=34" "lat_2=40.5" "lat_0=0" "lon_0=-120" "x_0=0" "y_0=-4000000" "units=m" END

21. Vector based DST Mapserver PHP -> PHPMapscript -> PostGIS PostGIS

22. Vector based DST - Database // Now connect to the db $host_str = "host=localhost"; $db_str = "dbname=shelf_closure_db"; $usr_str = "user=web_user"; $string = $host_str." ".$db_str." ".$usr_str; $connection = pg_connect($string); if (!$connection){ pg_close($connection); echo "Error: Cannot connect to the database \n"; } else { echo "Connected to the SCA database \n"; } // Create a dynamic VIEW in PostGIS echo " * Create postgresql VIEW * \n"; $exec_str = sprintf("CREATE VIEW ". $previous_session_name."_".$previous_session_id." AS ". "SELECT bk.the_geom, sum(tr.total_lbs*tb.percent) as total_lbs ". "FROM tow%02d_3kblk_intersect as tb, trawl%02d". " as tr, cablk3km as bk, port as pt ". "where tb.blk_gid=bk.gid ".$species_sql." ".$port_sql." ".$gear_sql. " and tb.tow%02d_id=tr.tow_id2 ". "GROUP BY bk.the_geom HAVING count(tr.tow_id2)>2", $start_year_short,$start_year_short,$start_year_short); $result_pg_exec = pg_exec($connection, $exec_str); if (!$result_pg_exec) { printf(" %s \n", pg_errormessage());}

23. Vector based DST – Dump results // Execute the pgsql2shp to create the output echo " * Execute pgsql2shp create the shapefile in session directory * "; $dir_exec = sprintf("/usr/local/pgsql/bin/pgsql2shp -u web_user -f ". $session_tmp_dir."data/target_areas/test_output.shp shelf_closure_db ". "\"SELECT * from ".$previous_session_name."_".$previous_session_id."\""); exec($dir_exec,$arr,$err1); for ($ii=0;$ii<count($arr);$ii++) { echo $arr[$ii]." \n"; } // Drop the dynamic table echo " * DROP the postgresql VIEW * \n"; $result_pg_exec = pg_exec($connection, "DROP VIEW ". $previous_session_name."_".$previous_session_id); if (!$result_pg_exec) { printf(" %s \n", pg_errormessage()); }

24. Vector Example - OCEANSystem Rockfish – SFA - 2001Rockfish – SFA - 2003 Shelf Closure mid-2002

25. AJAX - Prototype function gotVariable(varName, varVal, showWait) { // URL to send the AJAX request to var url = 'http://boris.ecotrust.org/gulf_project/core/SCA_Model/contents.php'; // Push the variable onto the array of parms var newVars = new Array(varName,varVal); varStack.push(newVars); // Build the params for the URL request var pars = ""; for (var i=0; i<varStack.length;i++) { if (i == 0) {pars = pars + varStack[i][0] + '=' + varStack[i][1];} else {pars = pars + '&' + varStack[i][0] + '=' + varStack[i][1];} } // Debug text var boldText = document.createElement('b'); var actionText = document.createTextNode(varName + ' :: '); var action2Text = document.createTextNode(varVal); boldText.appendChild(actionText); var breakText = document.createElement('br'); document.getElementById('placeholder_current').appendChild(boldText); document.getElementById('placeholder_current').appendChild(action2Text); document.getElementById('placeholder_current').appendChild(breakText); // Make the AJAX request var myAjax; myAjax = new Ajax.Updater('placeholder_select', url, {method: 'get', parameters: pars, onLoading: showWaitImage, onComplete: removeWaitImage}); currentStep++; }

26. AJAX - Prototype $action_array = array("0" => "init_action", "Revenue by Species" => "process_rev_by_species", "Revenue by Gear Type" => "process_rev_by_gear", "Revenue Total" => "process_rev_by_total", "Num Vessel Landings" => "process_num_vessel_landings", "Distinct Vesseles" => "process_distinct_vessels", "Average Rev Per Vessel" => "process_average_rev_per_vessel", "Distinct Fishermen" => "process_distinct_fishermen", "Distinct Species" => "process_distinct_species"); if (isset($http_form_vars["Action"])) { $function_ret = $action_array[$http_form_vars["Action"]]($connection, $http_form_vars); } else { // deal with the initial request echo " \n"; $function_ret = $action_array[0]($connection, $http_form_vars); echo " \n"; }

27. AJAX - Prototype fwrite($fp_data, "palette(rainbow(".count($y_string)."))\n"); fwrite($fp_data, "bitmap(\"species_revenue_".$rand_number.".png\", type =\"png256\", height = 2, width = 4, res = 300)\n"); fwrite($fp_data,"plot(x,seq(min(".$y_s."),max(".$y_s.")*1.4,length=length(x)),type=\"n\",xl b=\"Year\",ylab=\"Revenue\")\n"); for ($i=1; $i<=count($y_string); $i++) { fwrite($fp_data,"lines(x,y_".$i.",col=".$i.")\n"); if ($i==1) { $fill_col = $i; } else { $fill_col = $fill_col.",".$i; } fwrite($fp_data,"legend(x=min(x),y=max(".$y_s.")*1.4,c(".$species_s."),fill=c(".$fill_col."), bg=\"white\", ncol=4)\n"); fwrite($fp_data,"title(\"Revenue plot for Species Groups at Port Group ".$port."\")\n"); fwrite($fp_data,"dev.off()\n"); fflush($fp_data); fclose($fp_data); $dir_exec = "/usr/local/bin/R CMD BATCH species_revenue_".$rand_number.".R"; exec($dir_exec,$arr,$err1); echo " \n";

28. GDAL/OGR – Shapefile validity function check_valid_shapefile($shape_file_name) { //echo "Checking that shapefile ".$shape_file_name." is valid... \n"; $valid = TRUE; $duplicate = FALSE; $importance_present = FALSE; $name_temp = array(); // Here we need to check for the validity of the shapefile $dir_exec = sprintf("/usr/local/bin/ogrinfo -al ".$shape_file_name); exec($dir_exec,$arr1,$err1); for ($ii=count($arr1)-1; $ii>=0; $ii--) { if ((substr_count($arr1[$ii],": Integer") > 0) || (substr_count($arr1[$ii],": Real") > 0) || (substr_count($arr1[$ii],": String") > 0)) { // Push them all onto an array $string_to_chop = strstr($arr1[$ii],":"); $key_word = str_replace($string_to_chop,"",$arr1[$ii]); array_push($name_temp,$key_word); //echo $key_word." \n"; }

29. GDAL/OGR – Cont. for ($ii=0; $ii < count($name_temp); $ii++) { for ($iii=$ii+1; $iii < count($name_temp); $iii++) { if ($name_temp[$iii] == $name_temp[$ii]) { // we have a duplicate $duplicate = TRUE; $duplicate_name = $name_temp[$ii]; } if ($name_temp[$ii] == "IMPORTANCE") { $importance_present = TRUE; } if ($duplicate == TRUE) { echo " ERROR: We found a shapefile with duplicate columns named ". $duplicate_name." \n"; $valid = FALSE; } if ($importance_present == FALSE) { echo " ERROR: We found a shapefile named ".$shape_file_name. " without an IMPORTANCE column \n"; $valid = FALSE; } return $valid; }

30. 10 Step Tool Install 1) First install Fedora Core4 with updates 2) Proj.4 tar -xzvf proj-4.4.9.tar.gz cd proj-4.4.9 cd nad/ cp../../proj-nad27-1.2.tar.gz. tar -xzvf proj-nad27-1.2.tar.gz cd.../configure make make install 3) GDAL Installed the ECW SDK (do a config, make and install) Installed the MrSid SDK (moved to /usr/local) wget http://gdal.org/dl/gdal-1.3.1.tar.gz tar -xzvf gdal-1.3.1.tar.gz cd gdal-1.3.1./configure --with-ecw --with mrsid=/usr/local/GeoExpressSDK/ make ogr-all make install which gdalinfo 4) GEOS wget http://geos.refractions.net/geos-2.2.1.tar.bz2 tar -xjvf geos-2.2.1.tar.bz2 cd geos-2.2.1./configure make make install

31. 10 Step Tool Install 5) POSTGRESQL tar -xzvf postgresql-8.1.2.tar.gz cd postgresql-8.1.2 NOTE the LDFLAGS here... this is to support GEOS for postGIS LDFLAGS=-lstdc++./configure --with-perl -- prefix=/usr/local/pgsql_8_1_2 gmake gmake install Add the shared library path to /etc/ld.so.conf : /usr/local/pgsql_8_1_2/lib Run /sbin/ldconfig Finally add the following to the profile file to make paths available: PATH=$PATH:/usr/local/pgsql/bin MANPATH=$MANPATH:/usr/local/pgsql/man export MANPATH Add the postgis user useradd postgis passwd [postgis@localhost pgsql]$ postmaster -D./data/ >./data/logfile.txt 2>&1 & 6) POSTGIS Just make sure that the Postgresql is configured with the LDFLAGS variable set and that the GEOS software is downloaded and installed from Refractions../configure --with-proj --with-geos --with-pgsql make make install Now we can create the database: createdb -O aaronr test_db createlang -U aaronr plpgsql test_db psql -f /usr/local/pgsql_8_1_2/share/postgresql/ contrib/lwpostgis.sql -d test_db

32. 10 Step Tool Install 7) PHP./configure --prefix=/usr/local/php4 --program-suffix=4 -- enable-force-cgi-redirect --with-config-file- path=/etc/httpd/ --with-gd=/usr/local/ --with-jpeg -- with-png --with-tiff --with-zlib --with-freetype-dir -- without-ttf --with-mysql --with-regex=system --enable- dbase --enable-dbx --enable-versioning --with- pgsql=/usr/local/pgsql_8_1_2/ make make install strip sapi/cgi/php cp sapi/cgi/php /var/www/cgi-bin/php4 cp php.ini-dist /etc/httpd/php.ini Made the following mod to /etc/httpd/php.ini: ; Directory in which the loadable extensions reside. ;extension_dir = "./" extension_dir = "/etc/httpd/php_mods" mkdir /etc/httpd/php_mods Added the following to /etc/httpd/conf/httpd.conf # # For PHP scripts as CGI-BIN AddType application/x-httpd-php-cgi.php4.phtml Action application/x-httpd-php-cgi /cgi-bin/php4 Now re-start the server: /etc/rc.d/init.d/httpd restart Stopping httpd: [ OK ] Starting httpd: [ OK ] 8) MAPSERVER./configure --with-jpeg --with-gd --with-freetype -- with-zlib --with-png --with-pdf --without-tiff -- with-proj --with-threads --with-ogr --with-gdal --with-postgis --with-wfs --with-wmsclient -- with-wfsclient --enable-debug --with- php=/src/php/php-4.4.2 make cp legend mapserv scalebar /var/www/cgi-bin/ cp mapscript/php3/php_mapscript.so /etc/httpd/php_mods/ 9) GRASS CFLAGS="-g -Wall"./configure --with- gdal=/usr/local/bin/gdal-config --with- postgres-includes=/usr/local/pgsql/include/ -- with-postgres-libs=/usr/local/pgsql/lib/ make make install 10) R-Statistics tar -xzvf R-2.1.1.tar.gz cd R-2.1.1./configure make make install

33. Where to go for more info OSGIS Maptools - http://www.maptools.org FreeGIS - http://freegis.org/ Open Source GIS - http://opensourcegis.org/ Standards OGC - http://www.opengeospatial.org/ Desktop GRASS - http://grass.itc.it/ QGIS - http://qgis.org/ UDIG - http://udig.refractions.net/confluence/display/UDIG/Home JUMP – http://jump-project.org/ OpenEV - http://openev.sourceforge.net/ Server/Web Mapserver - http://mapserver.gis.umn.edu/ GRASS - http://grass.itc.it/ PostGIS - http://postgis.refractions.net/ Tools Remote Sensing - http://remotesensing.org/tiki-index.php GDAL/OGR - http://gdal.maptools.org/index.html PROJ.4 - http://proj.maptools.org/ R-Statistics - http://www.r-project.org/ GMT - http://gmt.soest.hawaii.edu/

34. The End Tool Screen Shots Follow

35. Backup - GRASS

36. Backup – R-Statistics

37. Backup – PostGIS Geometry WKT Geometry

38. Backup - QGIS