1. OWASP ModSecurity Core Rule Set (CRS) Project
Ryan Barnett
OWASP CRS Project Leader
Senior Security Researcher

2. Ryan Barnett - Background
Trustwave
SpiderLabs Research Team
Web application firewall research/development
ModSecurity Community Manager
Interface with the community on public mail-list
Steer the internal development of ModSecurity
Author
“Preventing Web Attacks with Apache”

3. Community Projects Open Web Application Security Project (OWASP)
Project Leader, ModSecurity Core Rule Set
Project Contributor, OWASP Top 10
Project Contributor, AppSensor
Web Application Security Consortium (WASC)
Project Leader, Web Hacking Incident Database
Project Leader, Distributed Web Honeypots
Project Contributor, Web Application Firewall Evaluation Criteria
Project Contributor, Threat Classification
The SANS Institute
Courseware Developer/Instructor
Project Contributor, CWE/SANS Top 25 Worst Programming Errors

4. Session Outline ModSecurity Quick Overview
The Core Rule Set (CRS) Overview
Basic Detection Categories
Latest CRS Improvements
CRS Demonstration Page
Future Directions

5. ModSecurity Quick Overview5

6. What is ModSecurity? Separate Rule and Audit Engines
It is an open source web application firewall (WAF) module for Apache web servers
Separate Rule and Audit Engines
Allows full request/response HTTP logging capability
Deep understanding of HTTP and HTML
Robust Parsing (form encoding, multipart, XML)
Event-based Rules Language
Anti-Evasion Features (normalization functions)
Advanced Capabilities
Transactional and Persistent Collections
Content Injection
Lua API 6

7. ModSecurity’s Rules Language Syntax
Tells ModSecurity how to process
SecRule TARGETS OPERATOR [ACTIONS]
Tells ModSecurity where to look (such as ARGS, ARGS_NAMES or COOKIES).
Tells ModSecurity what to do if a rule matches (such as deny, exec or setvar). 7

8. ModSecurity’s Apache Request Cycle Hooks8

9. OWASP ModSecurity Core Rule Set (CRS) Overview9

10. Project Info

11. What is the Core Rule Set (CRS)?
A generic, plug-n-play set of WAF rules
Choose your mode of operation
Standard vs. Anomaly Scoring
Detection Categories:
Protocol Validation
Malicious Client Identification
Generic Attack Signatures
Known Vulnerabilities Signatures
Trojan/Backdoor Access
Outbound Data Leakage
Anti-Virus and DoS utility scripts 11

12. Initial Configuration
Once unpacked, edit the main config file
modsecurity_crs_10_config.conf
Customize the following items
Mode of Detection – Standard vs. Anomaly Scoring
Anomaly Scoring Severity Levels
Enable/Disable Blocking
Blocking Threshold Levels
Paranoid Mode – aggressive inspection
HTTP Policy Settings
Choose where to log events (Apache error_log and/or ModSecurity’s audit log) 12

13. Traditional Detection Mode
Self Contained Rules Concept
IDS/IPS mode with “self-contained” rules
Like HTTP itself – the rules are stateless
No intelligence is shared between rules
If a rule triggers, it will execute a disruptive/logging action
Easier for the new user to understand
Not optimal from a rules management perspective (handling false positives/exceptions)
Not optimal from a security perspective
Not every site has the same risk tolerance
Lower severity alerts are largely ignored 13

14. Anomaly Scoring Detection Mode
Collaborative Rules Concept
Advanced inspection/detection mode
Delayed blocking
Rules set transactional variables (tx) to store temporary meta-data about the rule match
Rules also increase anomaly scores for both the attack category and global score
The anomaly score enforcement rules decide whether or not to deny transactions at the end of the inbound request phase
modsecurity_crs_49_inbound_blocking.conf 14

15. Anomaly Scoring - Debug Log View
Target value: "" OR 1=1#"
Set variable "tx.sql_injection_score" to "5".
Relative change: sql_injection_score=0+5
Resolved macro %{tx.critical_anomaly_score} to: 5
AdResolved macro %{rule.msg} to: SQL Injection AttackSet variable "tx.msg" to "SQL Injection Attack".
Setting variable: tx.sql_injection_score=+%{tx.critical_anomaly_score}
Recorded original collection variable: tx.sql_injection_score = "0"
Setting variable: tx.%{rule.id}-WEB_ATTACK/SQL_INJECTION-%{matched_var_name}=%{tx.0}
Resolved macro %{rule.id} to:
Set variable "tx.anomaly_score" to "8".
Relative change: anomaly_score=3+5
Original collection variable: tx.anomaly_score = "3"
Setting variable: tx.anomaly_score=+%{tx.critical_anomaly_score}
Set variable "tx WEB_ATTACK/SQL_INJECTION-ARGS:prod_id" to "1=1".
Warning. Pattern match "\b(\d+) ?(?:=|<>|<=>|<|>|!=) ?\1\b|[\'"\`\´\’\‘](\d+)[\'"\`\´\’\‘] ?(?:=|<>|<=>|<|>|!=) ?[\'"\`\´\’\‘]\2\b|[\'"\`\´\‘](\w+)[\'"\`\´\’\‘] ?(?:=|<>|<=>|<|>|!=) ?[\'"\`\´\’\‘]\3\b|([\'"\;\`\´\’\‘]*)?\s+(and|or)\s+([\s\'"\` ..." at ARGS:prod_id. [file "/usr/local/apache/conf/modsec_current/base_rules/modsecurity_crs_41_sql_injection_attacks.conf"] [line "425"] [id "950901"] [rev "2.0.9"] [msg "SQL Injection Attack"] [data "1=1"] [severity "CRITICAL"]
Resolved macro %{tx.0} to: 1=1
Resolved macro %{matched_var_name} to: ARGS:prod_id
Operator completed in 19 usec.Ctl: Set auditLogParts to ABIFHZE.Setting variable: tx.msg=%{rule.msg}
ded regex subexpression to TX.0: 1=1Added regex subexpression to TX.1: 1 15

16. Inspecting Anomaly Scores
# Alert and Block based on Anomaly Scores #
SecRule TX:ANOMALY_SCORE 0" \
"chain,phase:2,t:none,nolog,auditlog,deny,msg:'Inbound Anomaly Score Exceeded (Total Score: %{TX.ANOMALY_SCORE}, SQLi=%{TX.SQL_INJECTION_SCORE}, XSS=%{TX.XSS_SCORE}): %{tx.msg}',setvar:tx.inbound_tx_msg=%{tx.msg},setvar:tx.inbound_anomaly_score=%{tx.anomaly_score}"
SecRule TX:ANOMALY_SCORE %{tx.inbound_anomaly_score_level}" chain
SecRule TX:ANOMALY_SCORE_BLOCKING on" 16

17. Conditional Rules (Weak Sigs)
SQL Injection Example
Aggregate indicators to determine an attack
Strong indicators
Keywords such as: xp_cmdshell, varchar,
Sequences such as: union …. select, select … top … 1
Amount: script, cookie and document appear in the same input field
Weak indicators – meta-characters
--, ;, ', …
CRS only applies weak signatures in the event a stronger signature has previously triggered 17

18. Conditional Rule Example
SecMarker BEGIN_SQL_INJECTION_WEAK
SecRule &TX:/SQL_INJECTION/ 0" "phase:2,t:none,nolog,pass,skipAfter:END_SQL_INJECTION_WEAK"
SecRule TX:/SQL_INJECTION/ "\b(?:rel(?:(?:nam|typ)e|kind)|a(?:ttn(?:ame|um)|scii)|c(?:o(?:nver|un)t|ha?r)|s(?:hutdown|elect)|to_(?:numbe|cha)r|u(?:pdate|nion)|d(?:elete|rop)|group\b\W*\bby|having|insert|length|where)\b" \
"phase:2,chain,capture,t:none,ctl:auditLogParts=+E,block,nolog,auditlog,msg:'SQL Injection Attack',id:'950001',tag:'WEB_ATTACK/SQL_INJECTION',logdata:'%{TX.0}',severity:'2'"
SecRule MATCHED_VAR "(?:[\\\(\)\%#]|--)" \
"t:none,setvar:'tx.msg=%{rule.msg}',setvar:tx.sqli_score=+1,setvar:tx.anomaly_score=+20,setvar:tx.%{rule.id}-WEB_ATTACK/SQL_INJECTION-%{matched_var_name}=%{matched_var}"
SecMarker END_SQL_INJECTION_WEAK 18

19. Standard vs. Correlated Events
Event Logging
Standard vs. Correlated Events
Standard mode
Rules log event data to both the Apache error_log and the ModSecurity Audit log
Correlated mode
Basic rules are considered reference events and do not directly log to the Apache error_log
Correlation rules in the logging phase analyze inbound/outbound events and generate special events
modsecurity_crs_60_correlation.conf 19

20. Inbound/Outbound Correlation
Couple the inbound with the outbound for increased intelligence
Was there an inbound attack?
Was there an HTTP Status Code Error (4xx/5xx level)?
Was there an application information leak?
Correlation facilitates better incident response
App error without inbound attack -> Contact Ops
Inbound attack + outbound error -> Contact Security 20

21. Event Severity Ratings
Correlated Events
0: Emergency - is generated from correlation (inbound attack + outbound leakage)
1: Alert - is generated from correlation (inbound attack + outbound application level error)
Non-Correlated Events
2: Critical - highest severity level possible without correlation. It is normally generated by the web attack rules (40 level files)
3: Error - is generated from outbound leakage rules (50 level files)
4: Warning - is generated by malicious client rules (35 level files)
5: Notice - is generated by the Protocol policy and anomaly files
6: Info - is generated by the search engine clients (55 marketing file) 21

22. Correlated Event Messages
Message: Pattern match "\;\W*?\bdrop\b" at TX:pm_sqli_data_REQUEST_URI. [file "/opt/wasc-honeypot/etc/rules/base_rules/modsecurity_crs_41_sql_injection_attacks.conf"] [line "262"] [id "959001"] [msg "SQL Injection Attack"] [data "; drop"] [severity "CRITICAL"] [tag "WEB_ATTACK/SQL_INJECTION"] Message: Operator GE matched 0 at TX:anomaly_score. [file "/opt/wasc-honeypot/etc/rules/base_rules/modsecurity_crs_49_enforcement.conf"] [line "30"] [msg "Anomaly Score Exceeded (score 55): SQL Injection Attack Detected"]
Message: Pattern match "\bsupplied argument is not a valid MySQL\b" at RESPONSE_BODY. [file "/opt/wasc-honeypot/etc/rules/base_rules/modsecurity_crs_50_outbound.conf"] [line "259"] [id "971156"] [msg "SQL Information Leakage"] [severity "ERROR"] [tag "LEAKAGE/ERRORS"]
Message: Warning. Operator GE matched 1 at TX. [file "/opt/wasc-honeypot/etc/rules/base_rules/modsecurity_crs_60_correlation.conf"] [line "24"] [msg "Correlated Successful Attack Identified: Inbound Attack (SQL Injection Attack Detected) + Outbound Data Leakage (SQL Information Leakage) - (Transactional Anomaly Score: 85)"] [severity "EMERGENCY"] 22

23. Detection Mechanisms: Protocol Violations
Protocol vulnerabilities such as Response Splitting, Request Smuggling, Premature URL ending
Content length only for non GET/HEAD methods
Non ASCII characters or encoding in headers
Valid use of headers (for example, content length is numerical)
Proxy Access
modsecurity_crs_20_protocol_violations.conf
Attack requests are different due to automation
Missing headers such as Host, Accept, User-Agent
Host is an IP address (common worm propagation method)
modsecurity_crs_21_protocol_anomalies.conf 23

24. Detection Mechanisms: Protocol Policies
Policy is usually application specific
Some restrictions can usually be applied generically
White lists can be build for specific environments
Limitations on Sizes
Request size, Upload size
# of parameters, length of parameter
modsecurity_crs_23_request_limits.conf
Items that can be allowed or restricted
Methods - Allow or restrict WebDAV, block abused methods such as CONNECT, TRACE or DEBUG
File extensions – backup files, database files, ini files
Content-Types (and to some extent other headers)
Modsecurity_crs_30_http_policy.conf 24

25. Detection Mechanisms: Malicious Clients
Not aimed against targeted attacks, but against general malicious internet activity
Offloads a lot of cyberspace junk & noise
Effective against comment spam
Reduce event count
Detection of Malicious Robots
Unique request attributes: User-Agent header, URL, Headers
Black list of IP addresses
Rate based detection
Detection of security scanners
Blocking can confuse security testing software (WAFW00f)
modsecurity_crs_35_bad_robots.conf 25

26. Detection Mechanisms: App Layer Attacks
Detect application level attacks such as those described in the OWASP top 10
SQL injection and blind SQL injection
Cross site scripting (XSS)
OS command injection and remote command access
Remote file inclusion
modsecurity_crs_40_generic_attacks.conf
modsecurity_crs_41_sql_injection_attacks.conf
modsecurity_crs_41_xss_attacks.conf 26

27. Known Vulnerability Signatures
SpiderLabs received authorization from ET to convert their Snort rules and include them in the CRS
Converted the following rule files
emerging-web_server.rules
emerging-web_specific_apps.rules
Identifying attacks against known vulnerabilities does have value
Raised threat level
If done correctly, lessens false positives
CRS combines the what of our generic attack payload detection with the where of ET known vuln data 27

28. Example Emerging Threats Rule
Attack vector location –
URI + Parameter
alert tcp $EXTERNAL_NET any -> $HTTP_SERVERS $HTTP_PORTS (msg:"ET WEB_SPECIFIC_APPS 20/20 Auto Gallery SQL Injection Attempt -- vehiclelistings.asp vehicleID SELECT"; flow:established,to_server; uricontent:"/vehiclelistings.asp?"; nocase; uricontent:"vehicleID="; nocase; uricontent:"SELECT"; nocase; pcre:"/.+SELECT.+FROM/Ui"; classtype:web-application-attack; reference:cve,CVE ; reference:url, reference:url,doc.emergingthreats.net/ ; reference:url, sid: ; rev:5;)
PCRE –
Weak signature 28

29. Converted Emerging Threats Rule
Verify the URI of the request
# (sid ) ET WEB_SPECIFIC 20/20 Auto Gallery SQL Injection Attempt -- vehiclelistings.asp vehicleID
SecRule REQUEST_URI_RAW "(?i:\/vehiclelistings\.asp)" "chain,phase:2,block,t:none,t:urlDecodeUni,t:htmlEntityDecode,t:normalisePathWin,capture,ctl:auditLogParts=+E,nolog,auditlog,logdata:'%{TX.0}',id:sid ,rev:3,msg:'ET WEB_SPECIFIC 20/20 Auto Gallery SQL Injection Attempt -- vehiclelistings.asp vehicleID ',tag:‘web-application-attack',tag:'url,
SecRule &TX:'/SQL_INJECTION.*ARGS:vehicleID/' 0" "setvar:'tx.msg=ET WEB_SPECIFIC 20/20 Auto Gallery SQL Injection Attempt -- vehiclelistings.asp vehicleID
',setvar:tx.sqli_score=+1,setvar:tx.anomaly_score=+20,setvar:tx.%{rule.id}-SQL_INJECTION/SQL_INJECTION-%{matched_var_name}=%{matched_var}"
Verify the attack vector location from saved TX SQL Injection data exists 29

30. Detection Mechanisms: Trojans/Backdoors
Major problem in hosting environments
Uploading is allowed
Some sites may be secure while others not
Upload detection
Check uploading of files containing viruses (i.e. WORD docs)
util/modsec-clamscan.pl
Check uploading of http backdoor page
Access detection
Known signatures (x_key header)
Generic file management output (gid, uid, drwx, c:\)
modsecurity_crs_45_trojans.conf 30

31. Detection Mechanisms: Information Leakage
Monitoring outbound application data
HTTP Error Response Status Codes
SQL Information Leakage
Stack Dumps
Source Code Leakage
Last line of defense if all else fails
Provide feedback to application developers
Important for customer experience
Makes life for the hacker harder (if blocking is used)
modsecurity_crs_50_outbound.conf 31

32. Latest CRS Improvements:32

33. Latest Improvements Lua port of PHPIDS Converter.php code
Advanced normalization functions
More accurate use of PHPIDS Filters
Centrifuge Generic Attack Payload Detection
Experimental Rules
Generic attack payload detection
CRS Demonstration Page
Request Header Tagging 33

34. Lua port of PHPIDS http://phpids.net/
~70 regular expression rules to detect common attack payloads
XSS
SQL Injection
RFI
Filters are heavily tested by the community and updated frequently
Trustwave SpiderLabs worked with PHPIDS lead to port code to Lua for use in ModSecurity’s API
Thanks to Mario Heiderich 34

35. Lua port of PHPIDS Example normalization functions
--[[ Make sure the value to normalize and monitor doesn't contain Regex DoS
--[[ Check for comments and erases them if available ]]
--[[ Strip newlines ]]
--[[ Checks for common charcode pattern and decodes them ]]
--[[ Eliminate JS regex modifiers ]]
--[[ Converts from hex/dec entities ]]
--[[ Normalize Quotes ]]
--[[ Converts SQLHEX to plain text ]]
--[[ Converts basic SQL keywords and obfuscations ]]
--[[ Detects nullbytes and controls chars via ord() ]]
--[[ This method matches and translates base64 strings and fragments ]]
--[[ Strip XML patterns ]]
--[[ This method converts JS unicode code points to regular characters ]]
--[[ Converts relevant UTF-7 tags to UTF-8 ]]
--[[ Converts basic concatenations ]]
--[[ This method collects and decodes proprietary encoding types ]] 35

36. PHPIDS Example Filter <filter> <id>1</id>
<rule><![CDATA[(?:"[^"]*[^-]?>)|(?:[^\w\s]\s*\/>)|(?:>")]]></rule>
<description>finds html breaking injections including whitespace attacks</description>
<tags>
<tag>xss</tag>
<tag>csrf</tag>
</tags>
<impact>4</impact>
</filter> 36

37. Converted PHPIDS Example Filters
SecRule TX:'/^(QUERY_|REQUEST_|ARGS:).*_normalized/' "(?:\<\w*:?\s(?:[^\>]*)t(?!rong))|(?:\<scri)|(<\w+:\w+)" "phase:2,capture,t:none,pass,skip:1,nolog,auditlog,msg:'Detects obfuscated script tags and XML wrapped HTML',id:' ',tag:'WEB_ATTACK/XSS',logdata:'%{TX.0}',severity:'2',setvar:'tx.msg=%{rule.id}-%{rule.msg}',setvar:tx.anomaly_score=+4,setvar:'tx.%{tx.msg}-WEB_ATTACK/XSS-%{matched_var_name}=%{tx.0}'"
SecRule TX:PARANOID_MODE 1" "chain,phase:2,t:none,logdata:'%{TX.0}',severity:'2',pass,nolog,auditlog,msg:'Detects obfuscated script tags and XML wrapped HTML',id:' ',tag:'WEB_ATTACK/XSS'"
SecRule ARGS|REQUEST_BODY|REQUEST_URI_RAW "(?:\<\w*:?\s(?:[^\>]*)t(?!rong))|(?:\<scri)|(<\w+:\w+)" "capture,multiMatch,t:none,t:urlDecodeUni,t:cssDecode,t:jsDecode,t:htmlEntityDecode,t:replaceComments,t:compressWhiteSpace,t:lowercase,setvar:'tx.msg=%{rule.id}-%{rule.msg}',setvar:tx.anomaly_score=+4,setvar:'tx.%{tx.msg}-WEB_ATTACK/XSS-%{matched_var_name}=%{tx.0}'" 37

38. Centrifuge Code
Negative security approach to combating XSS and SQL Injection is doomed to fail…
Unlimited ways to write functionally equivalent code
Obfuscation methods, however often have certain characteristics
PHPIDS has an interesting approach to identify attack payloads through heuristics called Centrifuge
Analysis of the use of special characters
Ratio between the count of the word characters, spaces, punctuation and the non word characters
If <3.49 = malicious
Normalization and stripping of any word character and spaces including line breaks, tabs and carriage returns
Regex check in default_filters.xml catches results 38

39. PHPIDS Lua Port Demo 39

40. Experimental Generic Detection Rules
Two new experimental/beta generic detection rules
Optional_rules/modsecurity_crs_40_experimental.conf
Restricted Character Anomaly Usage
Analyzes the number and type of meta-characters present in a payload
Testing thus far shows that the detection is good except for free- form text fields.
Need to adjust the anomaly scoring for you own site
Repetitive use of non-Word characters
Currently alerts if there are 4 or more special characters found in a row 40

41. Live CRS Demonstration Page41

42. Live CRS Demonstration Page 42

43. CRS Demonstration Page
Request will go through CRS page first and then we proxy the request to the PHPIDS page
We then inspect the inbound with the outbound and provide results
CRS detected an attack
CRS did not find anything malicious but PHPIDS did
Neither CRS nor PHPIDS found anything malicious
A link is provided to report false negatives to our JIRA ticketing system
We have received >6700 attacks thus far 43

44. CRS Demonstration Page44

45. Demonstration page Demo45

46. Request Header Tagging46

47. CRS Demonstration Page
When in a distributed architecture, you can share WAF data with downstream hosts
Similar to SMTP SPAM data added to mime-headers
optional_rules/modsecurity_crs_49_header_tagging.c onf
Maps to AppSensor Detection Point – RP2 (Suspicious External User Behavior) 47

48. Example Header Tagging
GET /path/to/foo.php?test=1%27%20or%20%272%27=%272%27;-- HTTP/1.1
Host:
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv: ) Gecko/ Ubuntu/9.10 (karmic) Firefox/3.5.5
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO ,utf-8;q=0.7,*;q=0.7
X-WAF-Events: TX: / Detects common comment types-WEB_ATTACK/INJECTION-ARGS:test, TX: Detects JavaScript location/document property access and window access obfuscation-WEB_ATTACK/INJECTION-REQUEST_URI_RAW, TX: WEB_ATTACK/SQL_INJECTION-ARGS:test
X-WAF-Score: Total=48; sqli=2; xss=
Connection: Keep-Alive

49. Future Directions 49

50. Future Directions Preventing XSS with Content Injection
Javascript Sandboxing with Active Content Signatures
week-xss-defense-via-content-injection.html
noescape.html
Implementing OWASP AppSensor Detection Points
We currently have some mappings for existing events
Will be using Lua to implement other aggregate/behavioral/trend detection Points 50

51. Call for Community Help
We have made great strides with CRS v2.0 but there is still much work to be done
Test out the CRS demo page and report any issues found either to the mail-list or to JIRA
Need Rule Documentation help
Please sign up on our project mail-list if you want to help
rule-set 51

52. Questions? Email – Ryan.Barnett@owasp.org
Twitter