Common Gateway Interface Mechanism using Perl & Python NOTE: Some of slides are extracted from the course notes of USC CS571 and Deitel & Associates. These documents are copyrighted according to: either "Copyright © Ellis Horowits or PrenticeHall. All Rights Reserved.

Outline Basic Operation –Invoking a CGI Script –CGI Environment variables –CGI Script Output Using Perl/Python for Server-side scripting Program to print environment variables Program that checks the client’s browser Program that restricts access via IP address

Purpose of CGI Common Gateway Interface (CGI) is a mechanism by which programs, called scripts, can be used to create dynamic Web documents –Initially placed in a server directory often named cgi-bin –Serve information that is not directly readable by clients –Dynamically convert data from a non-Web source into Web-compatible documents Current version of CGI is 1.1 The reason for the term “common gateway” is these programs act as gateways between the WWW and any other type of data or service

Basic Operation An executable program that can be run without being directly invoked by users

Languages to Write Gateway Programs Any language that can produce an executable file Some typical ones are: –Traditional compiled languages such as C/C++ –Or interpreted languages such as: Perl Python C-Shell/Bourne Shell TCL Visual Basic or VBScript Interpreted languages are often preferred as they are –Easy to write and portable, and speed is usually not a factor Java and JavaScript were first designed for building client- side applications, but they can be used on the server side as well

Anchors Are Used to Invoke CGI Scripts A hypertext reference can refer to: –A local file –A remote file –An executable script in the cgi-bin directory –An executable script with arguments All of these anchors use the GET method

CGI Script Input There are three ways to pass input to a CGI script: the URL, standard input, environment variables GET Method - places all info in the URL POST Method –sends data to the server via a message body and a CGI script gets it from the server via stdin –The script returns data using stdout Command-line arguments –Many programs accept command-line arguments e.g., tar xvfz files.tar.gz –To invoke a program with command-line arguments, append them to the HREF in an anchor e.g.,

CGI Script Input Environment variables – DOS/Windows and UNIX use these as a means of passing information about the environment – Are set immediately before the server executes the gateway script – Portions of the URL are assigned to variables QUERY_STRING and PATH_INFO; e.g., bin/scriptname/extra_path/afile?input_data QUERY_STRING is assigned input_data PATH_INFO is assigned /extra_path/afile scriptname is executed

CGI Environment Variables Can be classified into two major categories: –1. Non-request specific –2. Request specific –Non-request-specific environment variables are set for all requests: SERVER_SOFTWARE, the name and version of the information server software answering the request e.g. SERVER_SOFTWARE = Apache/1.2.5 SERVER_NAME, server’s hostname, DNS alias, or IP address e.g. SERVER_NAME = nunki.usc.edu GATEWAY_INTERFACE, the revision of the CGI specification with which this server complies

CGI Environment Variables Request-specific environment variables –These variables are set depending on each request SERVER_PROTOCOL, the name and revision of the information protocol with which this request came in e.g. SERVER_PROTOCOL = HTTP/1.0 SERVER_PORT, the port number to which the request was sent e.g. SERVER_PORT = 8088 REQUEST_METHOD, the method with which the request was made; e.g., (GET, POST)

CGI Environment Variables PATH_INFO, the extra path information as given by the client; e.g., given cgi/extra/path then PATH_INFO = /extra/path PATH_TRANSLATED, the PATH_INFO path translated into an absolute document path on the local system PATH_TRANSLATED = /auto/home-scf- 03/csci351/WebServer/apache_1.2.5/htdocs/extra/path SCRIPT_NAME, the path and name of the script being accessed as referenced in the URL SCRIPT_NAME = /cgi-bin/test-cgi QUERY_STRING, the information that follows the ? in the URL that referenced this script

CGI Environment Variables –REMOTE_HOST, Internet domain name of the host making the request –REMOTE_ADDR, the IP address of the remote host making the request –AUTH_TYPE, the authentication method required to authenticate a user who wants access –REMOTE_USER, user name that server and script have authenticated –REMOTE_IDENT, the remote user name retrieved by the server using inetd identification (RFC 1413) –CONTENT_TYPE, for queries that have attached information, such as POST method, this is the MIME content type of the data –CONTENT_LENGTH, the length of the content as given by the client

CGI Environment Variables Also, every item of information in an HTTP request header is stored in an environment variable –Capitalize the name in the request header field –Convert dashes to underscores –Add the prefix HTTP_ For example: –HTTP_USER_AGENT contains the request header User_Agent field data e.g. HTTP_USER_AGENT = Mozilla/4.5 [en]C-DIAL (WinNT; U) –HTTP_ACCEPT contains the request header Accept field, of the form type/subtype –HTTP_REFERER contains the URL of the document that generated this request

CGI Script Output There are two ways a script can return data to the server –The script sends its output to stdout; the server adds appropriate headers and returns this output to the client –If the name of the CGI script starts with nph- (nonparsed header), the server sends whatever it receives directly on to the client Output from a script to the server could be: –A document generated by a script –Instructions to the server for retrieving the desired output The type of document could be: –HTML, plain text, image, or video or audio clip –References to other documents

Server Directives The output of scripts begins with a small header consisting of text lines containing server directives –This must be followed by a blank line Any headers that are not server directives are sent directly back to the client Server directives are used by CGI scripts to inform the server about the type of output The current CGI specification defines three server directives: –Content-type –Location –Status

Server Directives 1. Content-type: type/subtype –The MIME type of the document being returned –For example, content-type: text/html(HTML document) content-type: text/plain(plain-text document) 2. Location –Alerts the server that the script is returning a reference to a document, not an actual document –If the argument is a URL, the server will issue a redirect to the client; for example, location: gopher://gopher.ncsa.uiuc.edu/ –If the argument is a path, the document specified will be retrieved by the server, starting at the document root; for example, location: /path/doc.txt

Things to Check Before Running CGI Scripts The following need to be readable and executable by the server – CGI scripts – Other programs that the scripts call – The directory in which the scripts reside In UNIX, check the read/write permissions of the files and directories In Windows/NT, check the web server settings of the script directories

Perl Program to Print Environment Variables #!/perl5/bin/perl.exe print "Content-type: text/html", "\n\n"; print " ", "\n"; print " Environment Variables ", "\n"; print " Some Environment Variables ", "\n"; print " ", "\n"; print "SERVER NAME: ", $ENV{'SERVER_NAME'}, " ", "\n"; print "SERVER PORT: ", $ENV{'SERVER_PORT'}, " ", "\n"; print "SERVER PROTOCOL: ", $ENV{'SERVER_PROTOCOL'}, " ", "\n"; print "CGI Revision: ", $ENV{'GATEWAY_INTERFACE'}, " ", "\n"; print "REQUEST_METHOD ", $ENV{'REQUEST_METHOD'}, " ", "\n"; print "HTTP_ACCEPT ", $ENV{'HTTP_ACCEPT'}, " ", "\n"; print " ", "\n";

Sample Output

Perl Program That Checks the Client Browser #!/perl5/bin/perl.exe #set location of Perl and document root #place files graphicsver.html and textver.html in document root $document_root = '/web470/exercises'; $nongraphic_browsers = 'Lynx | CERN-LineMode'; $client_browser = $ENV{'HTTP_USER_AGENT'}; $graphic_doc = "graphicsver.html"; $text_doc = "textver.html"; if ($client_browser =~ /$nongraphic_browsers/) {$html_doc = $text_doc; } else {$html_doc = $graphic_doc; } print "Content-type: text/html", "\n\n"; $html_doc = join('/', $document_root, $html_doc); if (open (HTML, $html_doc)) { while ( ) { print; } close (HTML); } else { print "problem with configuration", " ";} exit(0);

Perl Program to Restrict Access #!/perl5/bin/perl.exe #set location of Perl and document root $document_root = '/web470/exercises'; $host_address = "ltree\.com"; $ip_address = "204\.253"; $remote_address = $ENV{'REMOTE_ADDR'}; $remote_host = $ENV{'REMOTE_HOST'}; $local_users = "intranet.html"; $outside_users = "internet.html"; if (($remote_host =~ /\.$host_address$/) && ($remote_address =~ /^$ip_address/)) {$html_doc = $local_users; } else { $html_doc = $outside_users; } print "Content-type: text/html", "\n\n"; $html_doc = join("/", $document_root, $html_doc); if (open(HTML, $html_doc)) { while ( ) { print; } close(HTML); } else { print "a problem", "\n";} exit(0);

HTML Creating Forms

Forms Used to create a set of pages that contain fields in which the viewer can select and supply information –Introduced into HTML 2.0 –Allows WWW users to perform data entry –Permit direct interaction with customers for inquiries, registration, sales of products, and services –To create a capability requires two steps: Use HTML form elements to create the pages that contain the form Write a server-side script to process form data; this program must be placed so the WWW server can execute it

Summary of User Interface Elements Text Checkbox Radio button Submit Reset Password submit reset **** File Browse Red Green Blue

Tag is an HTML tag that contains other tags for capturing user input –Has two attributes, ACTION and optionally METHOD –ACTION specifies the URL of a server-side script where the input data should be sent –METHOD selects variations in the sending protocol GETis the default; form contents are appended to the URL POSTcauses the fill-out form contents to be sent in a data body as standard input –The amount of information that can be sent via POST is not limited by the size of a URL

Tag Used inside the tag to specify a data-entry object Attributes –TYPE:What kind of input the user will supply (default is TEXT) –NAME:Name of data entry object whose value the user will supply –VALUE:Required for radio and checkboxes –CHECKED:For radio buttons and checkboxes –SIZE:Specific to each type of field –MAXLENGTH:Limit on accepted characters –SRC:Image file used as a graphical submit button when TYPE=IMAGE –ALIGN:TOP  MIDDLE  BOTTOM  LEFT  RIGHT

Tag(continued) TYPE:[CHECKBOX  FILE  HIDDEN  IMAGE  PASSWORD  RADIO  RESET  SUBMIT  TEXT] CHECKBOX: A single value, on/off; each generates name/value pair FILE: Users attach a file to the form contents; a text field holds the file name and a button permits browsing HIDDEN: The field is not rendered, so servers can maintain state information

Tag(continued) IMAGE: Used for graphical submit buttons PASSWORD: Just like TYPE=TEXT, but the input is echoed with * RADIO: Used for attributes that take a single value from a set of alternatives; all buttons have same name and explicit value

Tag(continued) RESET: Defines a button that users click to reset fields to their initial state SUBMIT: Defines a button that users click to submit the form’s contents to the server TEXT: An input field of a single line where users can enter data

Example of With Text Widgets Testing Text Widgets <FORMMETHOD="POST" ACTION="/cgi-bin/post-query"> Name: Date of Birth: Social Security Number: You can submit by clicking the SEND button:

Browser Output of Text Widgets Example

Query Results for Text Widget Example

Example of With Checkboxes Testing Checkboxes Fill in facts about yourself: own a house own a car own a boat have a college degree To reset the checkboxes, click here You can submit by clicking on the SEND button:

Browser Output of Checkbox Example

Query Results of Checkbox Example

Example of With Radio Buttons Testing Radio Buttons How would you like to pay? Choose one of the following: Billme Check Credit Card mastercard Visa American Express

Browser Output of Radio Buttons

Query Results for Radio Buttons Example

Tag specifies a large rectangular text-entry object with multi- line input and scroll bars Attributes: NAME=name specifies a name for the data entry object to be sent to the server-side script COLS=num –Width (in characters) of a text-entry region on the screen –If user types more than COLS characters, field is scrolled ROWS=num –Height (in characters) of a text-entry region on the screen –If user types more than ROWS lines, field is scrolled

Example of Multiline Input Areas Form Example with Multiple Multiline Inputs Here is a 10 x 30 text area. Here is a 2 x 20 text area. Here is a 1 x 40 area To submit your comments, press this button:

Browser Output of Multiline Input Areas

Query Results of Textarea Example

Tag Used inside the element to specify a selection list object (a list of items or a pop-down menu that the user can select from) Attributes: –NAME=name Specifies a name for the data entry object to be passed to the server-side script –SIZE=num Number of lines of the list to display at a time If SIZE is 1 or unspecified, browser will display as a drop-down list box If SIZE is greater than 1, browser will display as a scrollable list with only SIZE options visible at a time

Tag Attributes –MULTIPLE Specifies that multiple list items may be selected (whereas normally only 1 item can be selected) All selected values are sent to server-side script as separate name/value pairs

Tag Used inside the tag to specify the start of a new menu item in the selection list Syntax as follows: Text Attributes: –SELECTED Specifies this menu item as pre-selected in the list –VALUE="text" Text specifies the value to be sent to the script if the option is selected By default, the text following the OPTION element is sent –DISABLED Specifies a “grayed” or non-selectable list item

Example of, Tags Forms Example with Options <FORM METHOD="POST" ACTION="/cgi-bin/post-query"> Which School would you like to apply to? Letters&Science Engineering Business Law Medicine What semester do you wish to start? Fall Spring Summer To submit your choices, press this button:. To reset the form, press this button:.

Browser Output of, Example

Query Results for Example

Forms Example Test1 Example First name Last name Address Phone Number This example shows how one can align the fields of a form to match up with related text

Browser Output

LectureLecture CGI Scripts for Processing Forms

Outline Sample CGI Scripts in Perl –complete version of showcgi.pl –processing an application for credit –more examples using showcgi.pl –Program to extract a birthday –Program using extra path information –Program to echo form input –Program to return a GIF image –Graphic counter –Redirection –Creating a list of files Location of Perl CGI Scripts CGI Libraries –cgi-lib.pl –libwww.pl

A General Perl Program We have already seen a program that prints out the environment variables created by the server We extend this program so it also prints out –any command line arguments –any input sent on standard in –the name=value pairs, when there are any

General Algorithm for Decoding Form Data 1. determine the request method (GET or POST) by checking REQUEST_METHOD environment variable 2. If the protocol is GET, read the QUERY_STRING variable and/or the extra path information from PATH_INFO 3. If the protocol is POST, determine the size of the request using CONTENT_LENGTH, and read that amount of data from standard input 4. Split the query string on the "&" character, which separates key-value pairs, (the format is key=value&key=value) 5. decode the hexadecimal and "+" charactes in each key-value pair 6. create a key-value table with the key as the index.

Code to Check For GET and POST methods #!/usr/usc/bin/perl $request_method = $ENV{‘REQUEST_METHOD’}; if ($request_method eq “GET) {$form_info=$ENV{‘QUERY_STRING’}; } else { $size_of_form_info=$ENV{‘CONTENT_LENGTH’}; read(STDIN, $form_info, $size_of_form_info); } ($field_name, $input) = split (/=/, $form_info); #field_name will contain the name of the user input, $input the value that was entered

Showcgi.pl - Printing Command Line Arguments #!/perl5/bin/perl.exe #!/usr/usc/bin/perl #Perl script to print CGI inputs print "Content-type: text/html\n\n"; print " \n"; print " Show CGI Inputs \n"; print " "; print " Show CGI Inputs: \n"; print " Command Line Arguments: \n"; $j=1; foreach $a { print "arg$j: $a \n"; $j=$j+1; } Show where your Perl interpreter resides output MIME type ARGV is an array whose elements are the arguments on the command line

Showcgi.pl - Printing Environment Variables (contd) print " "; print " Environment Variables: \n"; print "SERVER_SOFTWARE = $ENV{'SERVER_SOFTWARE'} \n"; print "SERVER_NAME = $ENV{'SERVER_NAME'} \n"; print "GATEWAY_INTERFACE = $ENV{'GATEWAY_INTERFACE'} \n"; print "SERVER_PROTOCOL = $ENV{'SERVER_PROTOCOL'} \n"; print "SERVER_PORT = $ENV{'SERVER_PORT'} \n"; print "REQUEST_METHOD = $ENV{'REQUEST_METHOD'} \n"; print "HTTP_ACCEPT = $ENV{'HTTP_ACCEPT'} \n"; print "PATH_INFO = $ENV{'PATH_INFO'} \n"; print "PATH_TRANSLATED = $ENV{'PATH_TRANSLATED'} \n";

Showcgi.pl - Printing Environment Variables (contd) print "SCRIPT_NAME = $ENV{'SCRIPT_NAME'} \n"; print "QUERY_STRING = $ENV{'QUERY_STRING'} \n"; print "REMOTE_HOST = $ENV{'REMOTE_HOST'} \n"; print "REMOTE_ADDR = $ENV{'REMOTE_ADDR'} \n"; print "REMOTE_USER = $ENV{'REMOTE_USER'} \n"; print "CONTENT_TYPE = $ENV{'CONTENT_TYPE'} \n"; print "CONTENT_LENGTH = $ENV{'CONTENT_LENGTH'} \n"; print "HTTP_REFERER = $ENV{'HTTP_REFERER'} \n"; print "HTTP_USER_AGENT = $ENV{'HTTP_USER_AGENT'} \n"; print "HTTP_COOKIE = $ENV{'HTTP_COOKIE'} \n"; print " \n";

Showcgi.pl - Printing Standard Input (contd) print " Standard Input: \n"; #get buffer from QUERY_STRING (GET) or STDIN (POST) if ($ENV{'REQUEST_METHOD'} eq "GET") { $buffer = $ENV{'QUERY_STRING'}; print "There is no input in STDIN"; print " when using GET method. \n"; } else { read(STDIN, $buffer, $ENV{'CONTENT_LENGTH'}); print "$buffer\n"; } print " "; print " Name/Value pairs extracted \n"; #check for equal signs in buffer $e = index($buffer,"="); if ( $e == -1 ) { print "no name/value pairs in input\n"; } else Test here for the method, GET or POST and assign whatever it is to the variable $buffer If arguments exist, they are in the form of name=value

Showcgi.pl - Printing Name-Value Pairs (contd) { #make an array of pairs split at the & = split(/&/, $buffer); #for each pair, extract name and value foreach $pair {($name, $value) = split(/=/, $pair); #split into name and value #print name/value pair print "$name = $value \n"; } } split the strings in nvpairs using the = sign, and then print their name and value Splits the buffer string into several strings, divided by the & char. Each string is placed in the nvpairs array

Showcgi.pl - Printing Name-Value Pairs (contd) print " "; print " Name/Value pairs decoded \n"; if ( $e != -1 ) {foreach $pair {$pair =~ s/\+/ /g; #convert plusses to spaces ($name, $value) = split(/=/, $pair); #split into name and value #decode any %XX from hex numbers to alphanumeric $name =~ s/%(..)/pack("c",hex($1))/ge; $value =~ s/%(..)/pack("c",hex($1))/ge; #print name/value pair and decoded value print "$name = $value \n"; } } print " \n"; =~is “pattern equality” and the s stands for substitution; g causes a global substitution change the name/value pairs so that + is replaced by blank and hex codes are replaced by their equivalent character

Some Perl Points S/PATTERN/REPLACEMENT/[g][i][e][o] –searches a string for a pattern, and if found, replaces that pattern with the replacement text and returns the number of substitutions made, otherwise false –the g option indicates that all occurrences of the pattern are to be replaced –the i option indicates that matching is to be done in a case insensitive manner –the e option indicates that the replacement string is to be evaluated as an expression rather than just as a double-quoted string pack(template,list) –takes an array, or list of values and packs it into a binary structure returning the string containing the structure –template can be, e.g. c a signed char value, I a signed integer value, f a float value hex(expr) –returns the decimal value of expr

Examples of Showcgi.pl Check the class web page,

Form Input to test showcgi.pl – Form data passed via query using GET method Enter string and click here –Form data passed via stdin using POST method Enter string and click here

Application for Credit Form

Browser Input for Form Sample Form Application for a Credit Card Background Information Name Street City State Alabama California New York Wisconsin Amount of Credit $5,000 $10,000 $15,000 Financial Facts: Own a home Own a boat Own a car Please describe here the names and ages of people in your family and the number of cards you are requesting.

Output of showcgi.pl on Credit Form

Output of showcgi.pl on Credit Form (Pt II)

Output of showcgi.pl on Credit Form(Pt III)

Encoded Data When data is sent certain characters must be encoded, e.g. “, /, blank Each character has a hexidecimal equivalent, as shown previously The browser transforms special characters into their hexidecimal equivalents and the cgi script must transform back from hexidecimal to the character. Example: here is a form to capture a birthday. Slash must be encoded BIRTHDAY When is your birthday? Enter Birthday (mm/dd/yy):

Birthday Perl Script #!/usr/usc/bin/perl $size_of_form_info =$ENV{‘CONTENT_LENGTH’}; read(STDIN, $form_info, $size_of_form_info); $form_info =~ s/%([\dA-Fa-f][\dA-Fa-f])/pack (“C”, hex ($1))/eg; #the above turns %2F into a slash #s is substitute, \dA-Fa-f looks for hex number and stores it in $1 #pack and hex convert the value in $1 to ASCII, e evaluates second part #of the substitute command as an expression, g replaces all occurrences ($field_name, $birthday) = split (/=/, $form_info); print “Content-type: text/plain”, “\n\n”; print “Your birthday is on: $birthday, right?”, “\n”; exit(0);