1. Till Jaeger / Damian Skeeles EMEA SE Team November 2010
The Joy of Flex
Till Jaeger / Damian Skeeles
EMEA SE Team
November 2010
© 2010 ArcSight, Inc. All rights reserved.
ArcSight and the ArcSight logo are trademarks of ArcSight, Inc. All other product and company names may be trademarks or registered trademarks of their respective owners.
© 2010 ArcSight Confidential

2. © 2010 ArcSight Confidential
Agenda
Presentation
So many ways of Joy
Useful Tools
Categorisation and Normalisation
Structure of Properties file
Workshop
Syslog Regex Parser
ID-based Database Flex
And using the jTDS driver
Multiline Regex with Submessages and Mappings
© 2010 ArcSight Confidential

3. So Many Ways Of Joy
Delimiter separated, fixed format text files with wizard
“arcsight flexagentwizard”
Variable format text files with regex
Realtime filereader
Folder follower
Syslog sub-parser
<PRI>HEADER MSG
SNMPv1 traps
ODBC/JDBC based connectors
Use native JDBC whenever possible
XML files
XQuery

4. Log Files Number of files Access to the files Static or Dynamic Data
So Many Ways Of Joy
Log Files
Number of files
Single file or multiple files?
Do we know the exact file names?
If file names change, do they have an expressible pattern?
Access to the files
Do we have access to the files locally or remotely?
If not they may have to be copied periodically
Static or Dynamic Data
Is data in the file growing?
Are the files static and generated by the device or copied from the device periodically?

5. Log File Characteristics
So Many Ways Of Joy
Log File Characteristics
What kind of data is in the file?
Delimited Data
Free Form Data
Key Value Pairs
XML

6. Syslog Data Characteristics
So Many Ways Of Joy
Syslog Data Characteristics
Do the messages have a syslog header?
Is the syslog header rfc compliant?
If you are using syslog-ng, is it configured to output rfc-compliant header?
What Operating System do you plan to run the flex connector on?
Syslog Header
Actual Message
Facility + Priority
Host Name
Time Stamp

7. Chaining Parsers
Extra Processors
Use an extra processor when all or portion of data is suitable for parsing by other parsers
No limit in number of chained parsers
Extra processor type
Which parser is used
map
Map files
delimited
Delimited parser
regex
Regular expression parser
keyvalue
Key value parser
standardkeyvalue
Key value parser with standard delimiters (comma and equals to)
xml
XQuery-based XML parser
Parser chaning

8. © 2010 ArcSight Confidential
Useful Tools
© 2010 ArcSight Confidential

9. Useful Tools Regex Buddy (commercial) Regex Coach (donation ware)
Squirrel
DB Visualiser
Notepad ++ - Data
VIM - Properties files

10. Tools: Regex Buddy + Most comprehensive tool
+ Many Regex dialects, great library
+ Easy to performance optimize expression
- commercial

11. Tools: Regex Coach Free! Step-through mode / Show tokens
Doesn’t work with Java ‘\\’ double-escapes from properties file

12. Tools For JDBC Connectors: SQuirreL
If SQuirreL works, your connector will work!
Test Connectivity, Credentials, Drivers, Connection String, etc!
[ * Some Configuration Required ]

13. Categorisation and Normalisation

14. Categorisation and Normalisation
DO NOT ‘throw’ stuff in!
Normalisation
Existing content is correlating on the contents of those fields!
X “I’ll dump the Entrance Door number into TargetHostname”
Fields should be consistent for all events from that device
“Why is DCS1 sometimes a name, then a time, then a URL?”
Right-click Investigate should look ‘right’
“I right click IP on FW events, but don’t see events from the Flex Device?”
Beware of massively-varying Indexed Fields
X “I’ll put the URL in event.name”
We will talk about side tables in a few minutes
Consider Additional data fields 14

15. Categorisation and Normalisation
Create a custom Channel/Field Set / Turn on Raw Events

16. Categorisation and Normalisation
If you have time, or is a POC requirement
Beware of existing content triggering on this (eg. Config Changes, Brute Forces)
ESM 4.0 User Reference Guide / Data Fields ESM 5.0 ESM_UserGuide.pdf 16

17. Properties File Structure17

18. Properties File Structure

19. Must Have Event Mappings Device Vendor, Device Product, Device Event Class Id
Categorization and other modules depend on this information, thus it is a good practice to make sure all events have those fields
DeviceVendor = “Cisco”
DeviceProduct = “Cisco Pix”
DeviceEventClassId=407001

20. Timestamps Device Receipt Time, Start Time, End Time
What is this? Time when the device “detected” the event
Should this be set? YES! This MUST be set in the parser
Start Time
What is this? Time when the actual event detection started
Should this be set? Only when device calls it out explicitly
End Time
What is this? Time when the actual event detection ended or when the actual event ended.
Start time: Most devices will NOT send this information Exception: Scanner agents (sets time to when scan started)
Start Time and End Time are inferred from deviceReceiptTime.

21. Event names should not have variables
Rules Event Name
Event names should not have variables
%PIX : Denied Telnet login session from on interface inside
Good
Denied Telnet
Several ArcSight features (event graphs, data monitors, etc…) use the event name as a unique identifier of the event type, so when selecting a field to store the event name, make sure you pick a field that contains a fixed name and no variable elements (like ip address, port, etc…)
Bad
Denied Telnet login session from
UgLy
%PIX : Denied Telnet login session from on interface inside

22. Rules Device External Id vs. External Id
External Id should be an identifier for the event
whereas Device External Id should be an identifier for the device
Built inbound TCP connection for device corporate( /corp.acme.com)
Good
External Id = Device External Id = corporate
Bad
Device External Id = or External Id = corporate
UgLy
Device External Id = and External Id = corporate

23. Recommendation Regular Expressions
Regular expressions should be as specific as possible
abc,def,ghi
Good
[^,]+,[^,]+,[^,]+
Bad
\\S+,\\S+,\\S+
.* is EVIL. NEVER use more than one of these in the regex, and that too should preferably be at the end.
.*? is almost as evil. NEVER use more than one.
a cool regex trick: if you think you'll be receiving a multiline string: try (?s) at the beginning. actually this can be scoped!
UgLy
.*,.*,.*

24. Tips 24

25. The Burden With The Escape Characters
In literal Java strings the backslash is an escape character.
The literal string "\\" is a single backslash.
In regular expressions, the backslash is also an escape character.
The regular expression \\ matches a single backslash.
This regular expression as a Java string, becomes "\\\\". That's right: 4 backslashes to match a single one.

26. Some Useful Undocumented Token Operations
__divide(integer, integer)
__sum(integer, integer [, integer...])
__substract(...
__product(...
Curious for more?
Unzip the arcsight-parserframework<version>.jar
Look into \com\arcsight\agent\parsers\operation
Want to have these documented? => Open Jira requests ;-)

27. Syslog Subparser

28. Syslog Daemon Flex Connector
Syslog Flex Connectors
Syslog Daemon Flex Connector
Choose this connector if
You plan to run the connector on Windows or if you plan to run on Unix and do not want to use its native syslog daemon
You do not want to lose the <facility+priority> information coming from the original device
Raw TCP option is recommended over UDP to guarantee the reliable collection of syslog messages

29. Syslog Pipe or File Flex Connector
Syslog Flex Connectors
Syslog Pipe or File Flex Connector
Choose this connector if
You plan to run the connector on Unix and plan to use its native syslog daemon and configure it to send only the data coming from your devices
If <facility+priority> information coming from the original device is not critical to retain and you have similar information in the actual message
Syslog file connector is recommended for receiving events reliably at higher event rates or if you are running the connector on Solaris

30. Syslog Prototyping Avoid capturing/replaying syslog streams!
Copy out the \agentdata\*.syslog.N frontside cache file from the operational connector (10MB rollover)

31. Syslog Prototyping
Run your own test connector – once running, open the *.syslog.0 file
Copy/paste lines from the sample file into this ‘active’ file. Remember the ‘-’s. Then, save file to ‘send’ the events to the test connector
Remember to restart connector with each .properties change – wait until running before adding any events to the syslog.N file

32. Syslog: Parser Order
Flexconnectors in syslog are nearly at the enf of the chain
Your event may be matched by a too generic existing parser Typical candidate: Cisco Router
Remove all unneeded parsers by defining a custom subagent list in agent.properties
Re-ordering the parsers is more complex

33. Syslog Flex Workshop -> syslog header -> parse with RegEx
Nov 08 15:45: :11:07-19:05:29 ulogd[1993]: DROP: IN=eth0 OUT= MAC=00:0c:29:28:fa:4f:00:10:4b:b9:0e:84:08:00 SRC= DST= LEN=48 TOS=00 PREC=0x00 TTL=128 ID=34476 CE DF PROTO=TCP SPT=2591 DPT=1 SEQ= ACK=0 WINDOW=64240 SYN URGP=0
Nov 08 15:45:
-> syslog header
2006:11:07-19:05:29 ulogd[1993]: DROP:
-> parse with RegEx
And the Rest?
Need to animate this

34. Syslog Workshop – get started
Arcsight regex
Save file and give it a name
Load logfile
Treat as syslog
Start building your regex

35. ID-Based FlexConnector….. NOW! with jTDS!
For template guidelines or applying this template to an existing presentation, see the ArcSight presentation style guide:
Questions, contact 35

36. ID-Based Flex Workflow DBs: Choose ID- or Time-based
Prototype on SQuirreL first – blame the database!
Transfer through to FlexConnector

37. ID-Based Flex What can go wrong? Firewall Rules
Credentials (invalid user/pass)
Authorisation (access to table)
Fields exist, and Datatypes
JDBC Drivers (work, and are compatible)
Connection String
Everything else!
Use SQuirreL. Love SQuirreL.

38. jTDS Why jTDS? Free Driver supporting MS SQL Server
Supports Windows Authentication! On ConApp!
Apparently faster and more efficient

39. jTDS Driver download http://jtds.sourceforge.net Driver file goes into
<connector_dir>/current/lib/agent/jtds_1.2.5.jar
Database JDBC Classpath (into agent.properties)
agents[0].JDBCDriver=net.sourceforge.jtds.jdbc.Driver
Connection String
Mixed Mode: jdbc:jtds:sqlserver:// :1433;database=SQLServer
Windows:
jdbc:jtds:sqlserver:// :1433;database=SQLServer;domain=WORKGROUP

40. OK, we’re not out of time yet…?40

41. Submessage Parser – Sample Data

42. Submessage Parser - Analysis

44. Quiz 44

45. Question 1
How would you handle this?

46. Answer 1 A submessage subparser with no ID
- Tries matching first expression; if fails, tries the second…. Etc…

47. Question 2
The token ID is in the middle – with data you need both before and after… How to handle this?

48. Answer 2
Take everything before the Stop/Start as the main submessage token… and then pass off everything after to an extraprocessor

49. ArcSight, Inc.
Corporate Headquarters: ARST
EMEA Headquarters: +44 (0)
Asia Pac Headquarters: