Till Jaeger / Damian Skeeles EMEA SE Team November 2010 The Joy of Flex Till Jaeger / Damian Skeeles EMEA SE Team November 2010 © 2010 ArcSight, Inc. All rights reserved. ArcSight and the ArcSight logo are trademarks of ArcSight, Inc. All other product and company names may be trademarks or registered trademarks of their respective owners. © 2010 ArcSight Confidential

© 2010 ArcSight Confidential Agenda Presentation So many ways of Joy Useful Tools Categorisation and Normalisation Structure of Properties file Workshop Syslog Regex Parser ID-based Database Flex And using the jTDS driver Multiline Regex with Submessages and Mappings © 2010 ArcSight Confidential

So Many Ways Of Joy Delimiter separated, fixed format text files with wizard “arcsight flexagentwizard” Variable format text files with regex Realtime filereader Folder follower Syslog sub-parser <PRI>HEADER MSG SNMPv1 traps ODBC/JDBC based connectors Use native JDBC whenever possible XML files XQuery

Log Files Number of files Access to the files Static or Dynamic Data So Many Ways Of Joy Log Files Number of files Single file or multiple files? Do we know the exact file names? If file names change, do they have an expressible pattern? Access to the files Do we have access to the files locally or remotely? If not they may have to be copied periodically Static or Dynamic Data Is data in the file growing? Are the files static and generated by the device or copied from the device periodically?

Log File Characteristics So Many Ways Of Joy Log File Characteristics What kind of data is in the file? Delimited Data Free Form Data Key Value Pairs XML

Syslog Data Characteristics So Many Ways Of Joy Syslog Data Characteristics Do the messages have a syslog header? Is the syslog header rfc compliant? If you are using syslog-ng, is it configured to output rfc-compliant header? What Operating System do you plan to run the flex connector on? Syslog Header Actual Message Facility + Priority Host Name Time Stamp

Chaining Parsers Extra Processors Use an extra processor when all or portion of data is suitable for parsing by other parsers No limit in number of chained parsers Extra processor type Which parser is used map Map files delimited Delimited parser regex Regular expression parser keyvalue Key value parser standardkeyvalue Key value parser with standard delimiters (comma and equals to) xml XQuery-based XML parser Parser chaning

© 2010 ArcSight Confidential Useful Tools © 2010 ArcSight Confidential

Useful Tools Regex Buddy (commercial) Regex Coach (donation ware) Squirrel DB Visualiser Notepad ++ - Data VIM - Properties files

Tools: Regex Buddy + Most comprehensive tool + Many Regex dialects, great library + Easy to performance optimize expression - commercial

Tools: Regex Coach Free! Step-through mode / Show tokens Doesn’t work with Java ‘\\’ double-escapes from properties file

Tools For JDBC Connectors: SQuirreL If SQuirreL works, your connector will work! Test Connectivity, Credentials, Drivers, Connection String, etc! [ * Some Configuration Required ]

Categorisation and Normalisation

Categorisation and Normalisation DO NOT ‘throw’ stuff in! Normalisation Existing content is correlating on the contents of those fields! X “I’ll dump the Entrance Door number into TargetHostname” Fields should be consistent for all events from that device “Why is DCS1 sometimes a name, then a time, then a URL?” Right-click Investigate should look ‘right’ “I right click IP on FW events, but don’t see events from the Flex Device?” Beware of massively-varying Indexed Fields X “I’ll put the URL in event.name” We will talk about side tables in a few minutes Consider Additional data fields 14

Categorisation and Normalisation Create a custom Channel/Field Set / Turn on Raw Events

Categorisation and Normalisation If you have time, or is a POC requirement Beware of existing content triggering on this (eg. Config Changes, Brute Forces) https://irock.arcsight.com/message/3915 https://irock.arcsight.com/docs/DOC-2290 ESM 4.0 User Reference Guide / Data Fields ESM 5.0 ESM_UserGuide.pdf 16

Properties File Structure 17

Properties File Structure

Must Have Event Mappings Device Vendor, Device Product, Device Event Class Id Categorization and other modules depend on this information, thus it is a good practice to make sure all events have those fields DeviceVendor = “Cisco” DeviceProduct = “Cisco Pix” DeviceEventClassId=407001

Timestamps Device Receipt Time, Start Time, End Time What is this? Time when the device “detected” the event Should this be set? YES! This MUST be set in the parser Start Time What is this? Time when the actual event detection started Should this be set? Only when device calls it out explicitly End Time What is this? Time when the actual event detection ended or when the actual event ended. Start time: Most devices will NOT send this information Exception: Scanner agents (sets time to when scan started) Start Time and End Time are inferred from deviceReceiptTime.

Event names should not have variables Rules Event Name Event names should not have variables %PIX-3-307001: Denied Telnet login session from 1.1.1.1 on interface inside Good Denied Telnet Several ArcSight features (event graphs, data monitors, etc…) use the event name as a unique identifier of the event type, so when selecting a field to store the event name, make sure you pick a field that contains a fixed name and no variable elements (like ip address, port, etc…) Bad Denied Telnet login session from 1.1.1.1 UgLy %PIX-3-307001: Denied Telnet login session from 1.1.1.1 on interface inside

Rules Device External Id vs. External Id External Id should be an identifier for the event whereas Device External Id should be an identifier for the device Built inbound TCP connection 100888 for device corporate(1.1.1.1/corp.acme.com) Good External Id = 100888 Device External Id = corporate Bad Device External Id = 100888 or External Id = corporate UgLy Device External Id = 100888 and External Id = corporate

Recommendation Regular Expressions Regular expressions should be as specific as possible abc,def,ghi Good [^,]+,[^,]+,[^,]+ Bad \\S+,\\S+,\\S+ .* is EVIL. NEVER use more than one of these in the regex, and that too should preferably be at the end. .*? is almost as evil. NEVER use more than one. a cool regex trick: if you think you'll be receiving a multiline string: try (?s) at the beginning. actually this can be scoped! UgLy .*,.*,.*

Tips 24

The Burden With The Escape Characters In literal Java strings the backslash is an escape character. The literal string "\\" is a single backslash. In regular expressions, the backslash is also an escape character. The regular expression \\ matches a single backslash. This regular expression as a Java string, becomes "\\\\". That's right: 4 backslashes to match a single one.

Some Useful Undocumented Token Operations __divide(integer, integer) __sum(integer, integer [, integer...]) __substract(... __product(... Curious for more? Unzip the arcsight-parserframework<version>.jar Look into \com\arcsight\agent\parsers\operation Want to have these documented? => Open Jira requests ;-)

Syslog Subparser

Syslog Daemon Flex Connector Syslog Flex Connectors Syslog Daemon Flex Connector Choose this connector if You plan to run the connector on Windows or if you plan to run on Unix and do not want to use its native syslog daemon You do not want to lose the <facility+priority> information coming from the original device Raw TCP option is recommended over UDP to guarantee the reliable collection of syslog messages

Syslog Pipe or File Flex Connector Syslog Flex Connectors Syslog Pipe or File Flex Connector Choose this connector if You plan to run the connector on Unix and plan to use its native syslog daemon and configure it to send only the data coming from your devices If <facility+priority> information coming from the original device is not critical to retain and you have similar information in the actual message Syslog file connector is recommended for receiving events reliably at higher event rates or if you are running the connector on Solaris

Syslog Prototyping Avoid capturing/replaying syslog streams! Copy out the \agentdata\*.syslog.N frontside cache file from the operational connector (10MB rollover)

Syslog Prototyping Run your own test connector – once running, open the *.syslog.0 file Copy/paste lines from the sample file into this ‘active’ file. Remember the ‘-’s. Then, save file to ‘send’ the events to the test connector Remember to restart connector with each .properties change – wait until running before adding any events to the syslog.N file

Syslog: Parser Order Flexconnectors in syslog are nearly at the enf of the chain Your event may be matched by a too generic existing parser Typical candidate: Cisco Router Remove all unneeded parsers by defining a custom subagent list in agent.properties Re-ordering the parsers is more complex

Syslog Flex Workshop -> syslog header -> parse with RegEx Nov 08 15:45:40 10.1.1.13 2006:11:07-19:05:29 ulogd[1993]: DROP: IN=eth0 OUT= MAC=00:0c:29:28:fa:4f:00:10:4b:b9:0e:84:08:00 SRC=10.1.1.2 DST=10.1.1.13 LEN=48 TOS=00 PREC=0x00 TTL=128 ID=34476 CE DF PROTO=TCP SPT=2591 DPT=1 SEQ=2195296356 ACK=0 WINDOW=64240 SYN URGP=0 Nov 08 15:45:40 10.1.1.13 -> syslog header 2006:11:07-19:05:29 ulogd[1993]: DROP: -> parse with RegEx And the Rest? Need to animate this

Syslog Workshop – get started Arcsight regex Save file and give it a name Load logfile Treat as syslog Start building your regex

ID-Based FlexConnector….. NOW! with jTDS! For template guidelines or applying this template to an existing presentation, see the ArcSight presentation style guide: https://na5.salesforce.com/sfc/#version?id=06870000000DKLp Questions, contact marcom@arcsight.com 35

ID-Based Flex Workflow DBs: Choose ID- or Time-based Prototype on SQuirreL first – blame the database! Transfer through to FlexConnector

ID-Based Flex What can go wrong? Firewall Rules Credentials (invalid user/pass) Authorisation (access to table) Fields exist, and Datatypes JDBC Drivers (work, and are compatible) Connection String Everything else! Use SQuirreL. Love SQuirreL.

jTDS Why jTDS? Free Driver supporting MS SQL Server Supports Windows Authentication! On ConApp! Apparently faster and more efficient

jTDS Driver download http://jtds.sourceforge.net Driver file goes into <connector_dir>/current/lib/agent/jtds_1.2.5.jar Database JDBC Classpath (into agent.properties) agents[0].JDBCDriver=net.sourceforge.jtds.jdbc.Driver Connection String Mixed Mode: jdbc:jtds:sqlserver://172.16.100.100:1433;database=SQLServer Windows: jdbc:jtds:sqlserver://172.16.100.100:1433;database=SQLServer;domain=WORKGROUP

OK, we’re not out of time yet…? 40

Submessage Parser – Sample Data

Submessage Parser - Analysis

Quiz 44

Question 1 How would you handle this?

Answer 1 A submessage subparser with no ID - Tries matching first expression; if fails, tries the second…. Etc…

Question 2 The token ID is in the middle – with data you need both before and after… How to handle this?

Answer 2 Take everything before the Stop/Start as the main submessage token… and then pass off everything after to an extraprocessor

ArcSight, Inc. Corporate Headquarters: 1 888 415 ARST EMEA Headquarters: +44 (0)844 745 2068 Asia Pac Headquarters: +65 6248 4795 www.arcsight.com