1. © 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 1 1 Writing Apache 2.0 Modules and porting 1.3 modules to 2.0 Dirk-Willem van Gulik VP of Research, Covalent Technologies dirkx@covalent.net O'Reilly Open Source Conference 2002 v1.09

2. © 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 2 2 Overview u What are modules –Apache fundamentals u Module anatomy u 2.0 - what is new u 1.3 -> 2.0 checklist u Writing an authentication module (PAM) u Sample 1.3 -> 2.0 Migration u Acronym Module - filter sample (time permitting) u Q&A http://www.apache.org/~dirkx/oscon2002/

3. © 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 3 3 Fundamentals u Apache webserver –NCSA roots; The 'Shambala' rewrite –Modular architecture u Other Apache projects –Apr, Apr-utils, Proxy, docs –Tomcat, –Xml, java, Tcl, … http://www.apache.org/~dirkx/oscon2002/

4. © 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 4 4 History Apache 0.9Apache 1.2Apache 1.3.29 shambala Apache 2.0 APRAPR Utils Apache 2.0New Proxy patches ncsa …. modules ? PerlPHPJava

5. © 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 5 5 Apache: Basic facts u Webserver is written in ‘C’ u Runs on virtually all platforms –Unix, QNX, BeOS, Win, OS/390 u Very open modular infrastructure –Sequence of handlers act on each stage of the (http) request u Server core handles ‘child care’ and ensures protocol compliance.

6. © 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 6 6 Changes in 2.0 u Protocol engine abstracted out –http, https –POP3, Commercial FTP module u Filters and daisy chaining now possible u Abstracted out ‘child’ management –(pre)forked, threaded –Or a hybrid u More infrastructure for portability

7. © 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 7 7 Modules tasks: u Modules are small and simple: –Auth:verify a username and password –Access: verify if a username is on a list or an IP address matches –Rewrite: change a URL –Logging: log to a file or backend –Headers: Modify or add a header –Content: Simple content substitution

8. © 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 8 8 When - or when NOT u Simple specific task u Task maps to a single handler u Relatively light weight task for each request u Security close to the wire needed u Complex interaction with other sources required u Complex to secure. u Complex rules u Large footprint

9. © 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 9 9 Module Environment u Modules have –One known entry point –Set of registered callbacks –Set of known directives it can handle u Modules live a shielded live –Memory management –Log files, UID, socket access –Private memory for –Configuration information –Per request data

10. © 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 10 10 Anatomy - the short version u At startup –Load the modules –Set up basic infrastructure (logs, uids) –Read the configuration file –Hand each module its registered configuration directives –Ask each module to initialize u Runtime –Wait for requests to process –(Re)Initialize new children u Shutdown –Ask each module to shut down

11. © 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 11 11 Anatomy - Requests u Request comes in –File name translation –Check user id –Check access permissions –Check type –Last change fixups –Log when all is done

12. © 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 12 12 The API u Module block –Master router u Command block –Registry of configuration commands u Hooks –Registry of functions called at each stage u Handlers –Registry of functions called on condition u APR, APR utils –The backside; system abstraction

13. © 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 13 13 Configuration file u Apache configuration file –Httpd.conf, srm.conf, access.conf –.htaccess –Include 'file.conf' u Three types of directives –Processing and core –Includes, directory –UID, LoadModule, errorlog –Hierarchy –Module specific –Everything else

14. © 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 14 14 Anatomy - Configuration u Configuration file is read –Hierarchy is build –<Directory, <Server,.. –Modules are instructed to create/merge matching configurations –Server –Directory u Directive is read –Registered module is called with

15. © 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 15 15 Anatomy - your inputs u Pointer to your private memory segment. u Specific Arguments for –Parsed Directive details –Request you are expected to handle –I/O and Log streams

16. © 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 16 16 Anatomy - your outputs u Very few –DECLINED: 'aint me gov' –OK: 'Yup - I've done my job' –SERVER_ERROR: 'Scream murder' –Protocol/http status codes u But.. –You can modify data in the structures passed on –url, mime type, headers

17. © 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 17 17 Modules: 1.3 versus 2.0 u No fundamental Changes u More APR and APR utils support –Module authors can rely on a portability layer; existing code often reduced or simplified significantly. u Extra features –Filters; layered IO –More hooks, fewer pointer structs u Process management –Threaded, Process or hybrid

18. © 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 18 18 2.0 Checklist u No global variables –Your environment may be Threading –Use thread-safe libraries u Use Request record for passing state –No assumptions as to which child, process or thread will handle successive requests (from the client) or phase (within a request). u Potential longer lifetimes –Do your cleanups.

19. © 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 19 19 Examples u Mod_auth_pam –Popular existing 1.3 module –Authenticates against PAM –Pluggable authentication layer –Solaris, FreeBSD, Linux, others –/etc/passwd, SecurID, ldap, others u Mod_acronym (time permitting) –Contrived example –Uses filters http://www.apache.org/~dirkx/oscon2002/

20. © 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 20 20 Writing the module u Don’t write modules ! –You sure it is not already in the base install ? u Don't write modules !! –You sure there is not a module already on modules.apache.org ? u Don’t' write modules !!! –There must be a module already which comes close and which you can simply adapt to suit your purpose. http://www.apache.org/~dirkx/oscon2002/

21. © 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 21 21 Essentials u ANSI -C u (binary) version of apache u C compiler (gcc), make, linker u Documentation and Samples: –http://dev.apache.org/apidoc/ –http://modules.apache.org/ http://www.apache.org/~dirkx/oscon2002/

22. © 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 22 22 Initial Setup u Fetch a copy of apache –http://httpd.apache.org/dist u Compile and install –./configure --enable-maintainer-mode –make && make install u And build your own module –cd ~/myModule –apxs -c mod_auth_pam2.c http://www.apache.org/~dirkx/oscon2002/

23. © 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 23 23 Initial testing u Install the module –apxs -I mod_auth_pam2.c –Check httpd.conf –Add directives as needed u (Re)start the server –apachectl stop –apachectl start u Always keep an eye on the log ! –tail -f../logs/errorlog http://www.apache.org/~dirkx/oscon2002/

24. © 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 24 24 Common gotcha's u The 'apxs' in your path is -not- the one of the installed apache u '--enable-maintainer-mode' or -O3, -Wall not present. u You should restart your server each time. u gdb./httpd –run -X (1.2.x and above) –run -D ONE_PROCESS (2.0.x) –Use forking MPM if possible http://www.apache.org/~dirkx/oscon2002/

25. © 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 25 25 Mod PAM - Requirements u Requirements: –Use 'Basic Auth' –Authenticate a user against PAM –Access control on username –Access control on group membership u PAM API provides the basics –Pam_start(), Pam_authenticate().. –In: Username, password –Out: valid, has account –Thread safe. http://www.apache.org/~dirkx/oscon2002/

26. © 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 26 26 MaP: Outline u Configuration: –Accept Configuration directives –Store our configuration u Authenticate a username and a password –Respond with a yes/no/duh –Flag errors as appropriate. http://www.apache.org/~dirkx/oscon2002/

27. © 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 27 27 MaP: Headers #include "ap_config.h" #include "httpd.h" #include "http_config.h" #include "http_core.h" #include "http_log.h" #include "http_protocol.h" #include "http_request.h" #include @144

28. © 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 28 28 MaP: Configuration. u Pick your directives with care –Auth: we authenticate –PAM: against the PAM system u AuthPAM_Enabled –General on/off switch u AuthPAM_FailDelay –PAM specific timeout u AuthPAM_FallThrough –Am I authoritative ?

29. © 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 29 29 MaP: configuration (cont) u Your Private Configuration typedef struct { int fail_delay; int fall_through; int enabled; } auth_pam_dir_config; @191

30. © 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 30 30 MaP: Initialize u Few modules actually need this. u Note: static and different pools. static int auth_pam_init( apr_pool_t *p, *plog,*ptemp, server_rec *s ) { ap_log_error(APLOG_MARK, APLOG_INFO, 0, s,"PAM: mod_auth_pam/" VERSION); return OK; } @201

31. © 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 31 31 MaP: Real initialize 1/4 u Happens piece by piece –Within the hierarchy –Passed the correct pool –Return is opaque pointer.

32. © 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 32 32 MaP: Real Initialize 2/4 static void* create_auth_pam_dir_config(apr_pool_t *p, char *dummy) { auth_pam_dir_config *new = (auth_pam_dir_config*) apr_palloc (p, sizeof(auth_pam_dir_config)); new->fail_delay = 0; /* 0 ms */ new->fall_through = 0; /* off */ new->enabled = 1; /* on */ return new; } @214

33. © 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 33 33 MaP: Real initialize 3/4 u Create your own memory segment. u Use the right pool! u Do not even think to use malloc() auth_pam_dir_config *new = (auth_pam_dir_config*) apr_palloc (p, sizeof(auth_pam_dir_config)); @217

34. © 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 34 34 MaP: Real initialize 4/4 u Always initialize –-O5 –predictable stale pools –Security assumptions. new->fail_delay = 0; /* 0 ms */ new->fall_through = 0; /* off */ new->enabled = 1; /* on */ u And return the initialized memory segment: return new; @220

35. © 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 35 35 MaP: Populating 1/4 static command_rec auth_pam_cmds[] = { AP_INIT_TAKE1("AuthPAM_FailDelay", … AP_INIT_FLAG("AuthPAM_FallThrough",.. AP_INIT_FLAG("AuthPAM_Enabled",.. {NULL} } @245

36. © 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 36 36 MaP: Populating 2/4 u Calls 'parse_function(args)' when directive is seen. u Lots of pre defined parse functions u Be sensibly strict u Always tell the default. AP_INIT_TAKE1("MyDirective", parse_function, extra_argument, allowed, "helptext shown with httpd -V" ), @226

37. © 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 37 37 MaP: Populating 3/4 AP_INIT_TAKE1("AuthPAM_FailDelay", ap_set_int_slot, (void *)APR_OFFSETOF( auth_pam_dir_config, fail_delay), OR_AUTHCFG, "number of micro seconds to wait after failed authentication attempt. (default is 0.)" ), @247

38. © 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 38 38 MaP: Populating: 4/4 AP_INIT_FLAG("AuthPAM_Enabled", ap_set_flag_slot, (void *)APR_OFFSETOF(auth_pam_dir_config, enabled), OR_AUTHCFG, "on|off - determines if PAM authentication is enabled; " "(default is on.)" ), @259

39. © 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 39 39 MaP: recap u Configuration: AuthType Basic AuthName "Your Intranet passwd" AuthPAM_Enable yes require valid-user

40. © 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 40 40 MaP: Check User 1/3 u On entry –We get our configuration back. –The protocol (http) and core can supply more information. static int pam_auth_basic_user (request_rec *r) { auth_pam_dir_config *conf = ap_get_module_config( r->per_dir_config, &pam_auth_module); @341

41. © 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 41 41 MaP: Check User 2/3 u We're always called. Make sure we are configured (This is why initialization was so important) : if (!conf->enabled) return DECLINED; u And find out if a username and password was supplied: if ((res = ap_get_basic_auth_pw (r, (const char**)&(userinfo.pw)))) return res; userinfo.name = r->user; @357 @361

42. © 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 42 42 MaP: recap u At this point –We have access to the right configuration at the right place in the configuration hierarchy. –The username and password are available to us. –As is everything else in 'r'. Decision time !

43. © 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 43 43 MaP: Check User 3/3 Log! ap_log_rerror(APLOG_MARK, APLOG_ERR, 0, r, "PAM: user '%s' - not authenticated: %s", r->user,compat_pam_strerror(pamh, res)); On Error return HTTP_INTERNAL_SERVER_ERROR; On success return OK; On unkown if (fall_through) return DECLINED; Otherwise ap_note_basic_auth_failure(r); return HTTP_UNAUTHORIZED; @374 @377 @423 @404 @410

44. © 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 44 44 MaP: Wiring it all up 1/3 u Advertising –Our configuration entry points –Our initialization callbacks –Registration of our hooks –Post_config (init message) –Check_user (verification)A u Apache 1.3/2.0 - biggest difference.

45. © 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 45 45 MaP: Wiring it all up 2/3 u Apache 1.3 /2.0 quite similar module AP_MODULE_DECLARE_DATA pam_auth_module = { STANDARD20_MODULE_STUFF, create_auth_pam_dir_config, /* dir config creater */ NULL, /* dir merger */ NULL, /* server config */ NULL, /* merge server config */ auth_pam_cmds, /* command table */ …… @517

46. © 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 46 46 MaP: Wiring it all up 3a/3 u The Apache 1.3 method …. auth_pam_init, /* initialize */ …. NULL, /* handlers */ NULL, /* filename translation */ pam_auth_basic_user, /* check_user_id */ pam_check_auth, /* check auth */ NULL, /* check access */ NULL, /* type_checker */ NULL, /* fixups */ NULL /* logger */ }; @519 @525

47. © 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 47 47 MaP: Wiring it all up 3b/3 u OR The Apache 2.0 way: … pam_register_hooks, /* register hooks */ }; static void pam_register_hooks(apr_pool_t *p) { ap_hook_post_config(auth_pam_init, NULL, NULL, APR_HOOK_MIDDLE); ap_hook_check_user_id(pam_auth_basic_user, NULL,NULL,APR_HOOK_MIDDLE); } @525 @511

48. © 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 48 48 MaP: 1.3 versus 2.0 1/3 u Easy –Thread safe, no global variables u #include "ap_config.h" (114) u Closed security niggle in _init. (209) u Removed 'legacy' (2.0: good excuse) u Removed 'require' checking. (430). u Changed: ap_palloc() -> apr_palloc(). u All directive functions removed (226) –Stock string/int/boolean parsers used. u Rewrote command_rec with macro's. (245)

49. © 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 49 49 MaP: 1.3 to 2.0 changes 2/3 u server_rec; (336) –r->connection->user changed location to r- >user u Logging functions (374) –'r' variant (security) –PAM: prefic for clarity. –Use of log levels. u AUTH_REQUIRED became (419) –HTTP_UNAUTHORIZED u Module callback list shorter (511) –And callbacks actively registered instead.

50. © 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 50 50 MaP: Conclusion u Very typical module u Change to 2.0 –Made the module simpler. –Made it more portable –Was simple and we could borrow from 'modules/aaa/mod_auth_*.c

51. © 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 51 51 Example deficiencies u PAM library assumed to be thread safe –If not - needs to be 'mutex'-ed. u Use of /etc/group rather than a AuthGroupFile –Use mod_access_etc_group.c

52. © 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 52 52 Note: u APXS is not the only way to build a module. Do not use it if: –You need 'autotconf'. –Complex linking or dynamic loading. u Other methods –./configure pointing and Config.m4 –--with-module=type:name

53. © 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 53 53 Fine ! u Questions

54. © 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 54 54 Acronym Expansion - filters u Filters –The true cool feature of 2.0 –Daisy chain modules –On Input and Output –Some header/body understanding –Not carved in stone –Buckets and Brigades http://www.apache.org/~dirkx/oscon2002/

55. © 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 55 55 Acronyms u Outline and requirements –Add a footnote to each page containing acronyms (e.g. ASF*). –Detects acronyms a˜utomatically –Be reasonable fast u Solution: –An Output filter. –Filter each APR ‡ buckets and replace or insert where needed *) ASF: Apache Software Foundation ‡) APR: Apache Portable Runtime

56. © 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 56 56 Rough Design u Configuration –Enable/Disable per directory/file or similar resource blocks. –Standard unix dictionary format. –E.g. /usr/share/misc/airports.txt –Not use too much memory. –Needs to work on only the HTML files. u Sample –Lots of deficiencies.

57. © 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 57 57 Outline 1/2 u Configuration –Per directory –For the server as a whole u Read in a dictionary –And store in a hash u Plug the filter into the stream –When enabled and for html only http://www.apache.org/~dirkx/oscon2002/

58. © 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 58 58 Outline 2/2 u Output Filter –Detect an acronym –Add a ‘dagger’ and a link. –Add some in situ javascript/ALT –At the bottom of the page; –add the expansion. http://www.apache.org/~dirkx/oscon2002/

59. © 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 59 59 Bucket Brigades Brigade Bucket Brigade Bucket EOS

60. © 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 60 60 “Hello World of Apache” Brigade “Hell”“o Worl”“d ” Brigade “of Ap”“ache” EOS

61. © 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 61 61 Configuration u In memory structures u Create/Store in our private context. u Configure Directives u Read in the specified acronym file(s). @103 @126 @492 @413

62. © 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 62 62 Parsing (1/8) u Let’s ignore the header - focus on body. u File read or generated. –Verified to be HTML, enabled. –And chained into the stream output stream set up to the client. I am flying out of SFO to night. @147 @168

63. © 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 63 63 Parsing (2/8) u Passed in several brigades –Each brigade is a single call I am flying out of SFO to night. @171

64. © 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 64 64 Parsing (3/8) u Passed in several brigades –Consisting in several buckets! I am flying out of SFO to night. @517

65. © 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 65 65 Parsing (4/8) u Prepare a brigade with the output u First brigade can be parsed until the end u and passed down the chain. I am flying out of SFO to night. I am flying @200 @201 @397

66. © 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 66 66 Parsing (5/8) u Second brigade –Buckets parsed until we hit the acronym –Pass ‘S’ state on to the next brigade I am flying out of SFO to night. I am flying out of S @307

67. © 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 67 67 Parsing (6/8) u Using ‘S’ passed –parse until end of Acronym u Add around acronym. I am flying out of SFO to night. I am flying out of SFO ‡ @319

68. © 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 68 68 Parsing (7/8) u And parse out the final part of the string I am flying out of SFO to night. I am flying out of SFO ‡ To night.

69. © 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 69 69 Parsing (8/8) Once done - add the footnote with the acronyms. I am flying out of SFO to night. I am flying out of SFO ‡ To night. SFO: San Francisco International. @352,208

70. © 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 70 70 Demo Travel Plan Travel for next week: Tuesday: From IAD to SFO Thursday: back from IAD to SLC Sunday: out from SLC to OAK.

71. © 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 71 71 In the browser:

72. © 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 72 72 Summary u Filtering module –Brigades with buckets –Be prepared –Keep state u Simple and Fast –One specific function u Imagination –Combine with anything apache does !

73. © 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 73 73 Reminder u No need to write modules! –It is probably there in core. u No need to write modules! –modules.apache.org may have it. u No need to write modules –Something must be close enough to be subverted for your purpose. u But if you do –Let the world know !

74. © 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 74 74 Fine ! u Questions http://www.apache.org/~dirkx/oscon2002/