1. ArcSight: FlexConnectors
Introduction to the ArcSight FlexConnector Framework
Till Jäger, Solution Architect
CISSP

2. What this is (and what it is not)
This is not
A SQL Training
A Regex Training
A FlexConnector Training
So what is it then?
Broad overview of functionality
Guide how to approach your problem
Help to get started

3. Agenda Connector basics Flexconnectors basics Tools Regex parsers
WUC parsers
DB Connectors
Syslog basics
Connector by example: Syslog
Categorization
Tips & tricks / Best practice
Parser overriding

4. SmartConnector Basics

5. What is a SmartConnector?
A SmartConnector is software that collects events from end-point devices, normalizes the events, send them to an ArcSight destination.

6. SmartConnector Event Flow

7. SmartConnector Types File Connectors Database Connectors
Scanner Connectors
API Connectors
SNMP Connectors
Microsoft Windows Event Log Connectors
Syslog Connectors
FlexConnectors
Model Import Connectors
NetFlow Connector

8. SmartConnector Directory Structure
/current/bin – executables and scripts
Arcsight.bat / arcsight.sh
/current/config/agent – default/base configurations
Agent.defaults.properties
Agent.wrapper.conf.base
/current/logs – SmartConnector generated logs
Agent.log.n
Agent.out.wrapper.n
/current/user
/agent

9. SmartConnector Directory Structure
/current/user/agent – Connector properties and destination-specific configurations, preserve state files
Agent.properties
Hosts.txt
Persisted.properties
<agentid>.xml config files
Syslog.properties
/current/user/agent/agentdata – queue, cache, persistence files
.prstdout.n files
cache.dflt.n files
Syslogd.n files
/current/user/agent/aup – directory for content aup, additional data mappings, zones
Destination specific directories
Additional data mapping files – ngadatamapping.properties
/current/user/agent/acp – directory for categorization files
Categorization files – content aup

10. SmartConnector Directory Structure
/current/user/agent/fcp – directory for parser overrides
Parser overrides
/current/user/agent/flexagent – directory for custom parsers, and regex tester
Custom parsers
/current/user/agent/map – directory for map.n.properties files
Map.n.properties

11. SmartConnector Configuration Files
Location of the Configuration files: /current/user/agent/
agent.properties
contains the global configuration
Agent<id>.xml
contains the destination specific configuration

12. SmartConnector Configuration Files
agent.wrapper.conf
Service configuration file that includes the wrapper configuration
E.g. increasing the heap size, extending ping timeouts,..
agent.defaults.properties
Located in /current/config/agent
Contains the default framework parameters
Contains syntax for enabling debugging and increasing agent log file size, and agent log count

13. Additional Data Mappings / map.x.properties
Often not all fields are explicitly mapped to schema fields
Mapped to additional data
Can be mapped on-demand from Console
Right-click on the SmartConnector, and send the command “get additional data names”
map.x.properties
useful for adding additional details to events
Uses “getters” and “setters”
For example: When source user name is John, set Device Custom String 1 to John’s phone
event.sourceUserName, set.event.deviceCustomString1
John,
/current/user/agent/map

14. Categorization ArcSight Content
Provides additional meaning to the event
Requires 3 fields to be populated to work: Device Event Class ID: Device Product: Device Vendor
Special kind of map file in user/agent/acp/categorizer/current
Required for Foundation Content / System Content to work properly

15. Log Analysis If anything goes wrong: /logs/agent.log Search for ERROR
For log trend analysis use LogFu
LogFu is an ArcSight log Analysis Utility
Part of every SmartConnector install
Open a CMD Prompt CD to /current/logs
\logs>..\bin\arcsight agent logfu -a

16. Log Analysis: LogFu!

17. FlexConnectors Choose The Right Type

18. Before You Get Started Read the Flexconnector Guide
Familiarize with the Event Schema
Download latest Connector build
Get to know the basics of Regex

19. FlexConnector Types File Reader
Multi Folder Reader (Batch and/or Realtime)
Syslog Subagent
SNMP
Database
Vulnerability Scanner
Model Import
WUC (Windows Unified Connector) parser

20. FlexConnector Parser Types
Parser File name for each parser type has a unique extension
Delimited Log Parser (sdkfilereader)
Regex Parser (sdkrfilereader)
Key Value parser (sdkkeyvaluefilereader)
Database parser (sdktbdatabase, sdkibdatabase)
SNMP parser (sdksnmp.X.snmptrap)
XML parser (xqueryparser)
A combination of those (more later)

21. Log File Characteristics
Log File Flex Connectors
Log File Characteristics
Number of files
Single file or multiple files?
Do we know the exact file names?
If file names change, do they have an expressible pattern?
Access to the files
Do we have access to the files locally or remotely?
If not they may have to be copied periodically
Static or Dynamic Data
Is data in the file growing?
Are the files static and generated by the device or copied from the device periodically?

22. Log File Characteristics
Log File Flex Connectors
Log File Characteristics
What kind of data is in the file?
Delimited Data
Free Form Data
Key Value Pairs
XML

23. Choose The Right Type

24. Did You Know that…
You can re-use a connector install by deleting the agent.properties and run „./arcsight agentsetup“ again
You can copy an installed connector to a new location and re-use it
A Connector usually doesn‘t need Administrative/root access
You don‘t need a running ESM/logger to test your Connector

25. FlexConnectors Structure of a properties file

26. Configuration File comments.start.with=# delimiter=,
contains.empty.tokens=true
token.count=9
event.name=Action
event.message=__concatenate(Action,”; ”,Message)
event.deviceReceiptTime=Event_Time
event.attackerAddress=SourceIP
event.attackerPort=SourcePort
event.targetAddress=DestinationIP
event.targetPort=DestinationPort
event.transportProtocol=Protocol
event.deviceAddress=FirewallIP
event.deviceSeverity=Action
event.deviceCustomNumber1=__regexToken(Message,[^.]\. ? Rule (\\d+))
event.deviceCustomNumber1Label=__stringConstant(“Rule Number”)
token[0].name=FirewallIP
token[0].type=IPAddress
token[1].name=Event_Time
token[1].type=TimeStamp
token[1].format=yyyy/MM/dd HH:mm:ss
token[2].name=SourceIP
token[2].type=IPAddress
token[3].name=SourcePort
token[3].type=Integer
token[4].name=DestinationIP
token[4].type=IPAddress
token[5].name=DestinationPort
token[5].type=Integer
token[6].name=Protocol
token[6].type=String
token[7].name=Action
severity.map.high.if.deviceSeverity=drop
severity.map.low.if.deviceSeverity=accept 26

27. Double Underscore Operators
More listed in the flex developers guide
Operator
Meaning
Example
__stringConstant()
Specifies a constant string for event mapping
__stringConstant(“This is the Event Name”)
__concatenate()
Concatenates two or more fields
__concatenate(Token1,Token2)
__regexToken()
Used to parse a field one time
__regexToken($1,User \([^)])\)(.*))
__toUpperCase()
Converts a string to all upper case
__toUpperCase(TokenName)
__createTimeStamp
Creates a timestamp from a date field and a time field
__createTimeStamp(Date,Time) 27

28. FlexConnectors Tools

29. Tools FlexConnector wizard for delimited logs ArcSight Regex Tool
Flexcon wizard in Conapp
Notepad++
We!Analyze
Netsend
Tail for Windows
RegexBuddy
SQuirreL SQL Client

30. Tools: Regex Buddy + Most comprehensive tool
+ Many Regex dialects, great library
+ Easy to performance optimize expression
- commercial

31. Tools For JDBC Connectors: SQuirreL
If SQuirreL works, your connector will work!
Test Connectivity, Credentials, Drivers, Connection String, etc!

32. FlexConnectors Regex Parsers

33. Regex Configuration File
Parser Configuration
Common regular expression to match all entries in the log file
:FW 2002/10/01 18:12: : :80 (tcp) action=drop Message: Rule 25
:VPN 2002/10/01 18:12: : :80 (tcp) action=drop Message: Encryption failed, username jsmith Rule5
regex=(\d+.\d+.\d+.\d+):(\S+) (\d+/\d+/\d+ \d+:\d+:\d+) (\d+.\d+.\d+.\d+):(\d+) (\d+.\d+.\d+.\d+):(\d+) \((\S+)\) action=(\S+) Message:\s(.*)?Rule\s+(\d+)
Token Declaration
Each value in parentheses will be tokenized:
token.count=11
token[0].name=Firewall_IP
token[0].type=IPAddress
token[1].name=ProcessName
token[1].type=String
token[2].name=Event_Time
token[2].type=TimeStamp
token[2].format=yyyy/MM/dd HH\:mm\:ss
token[3].name=SourceIP
token[3].type=IPAddress
token[4].name=SourcePort
token[4].type=Integer
token[5].name=DestinationIP
token[5].type=IPAddress
token[6].name=DestinationPort
token[6].type=Integer
token[7].name=Protocol
token[7].type=String
token[8].name=Action
token[8].type=String
token[9].name=Message
token[9].type=String
token[10].name=RuleNumber
token[10].type=Integer

34. Regex Parser - submessages
Common regex
regex - main or common regular expression, for some parsers, this is sufficient
Submessage Processing
submessage.messageid.token, token that specifies a unique message id
submessage.token, token that should be processed as submessage
submessage.count, total number of submessages
submessage[N].messageid [if omitted, becomes the default submessage]
submessage[N].pattern.count, how many different patterns a submessage can have
submessage[N].pattern[M].regex, regex for the pattern
submessage[N].pattern[M].fields, what fields will be set for the pattern
submessage[N].pattern[M].mappings [optional, what values will be set to fields]
submessage[N].pattern[M].extramappings [optional, any extra mappings]

35. Regex - submesages Subparser regular expressions
Suparsers extract additional information from fields created by the common regular expression
Select the submessage ID Token
Select the token to be parsed
Create a regular expression to parse the submessage
Map the tokens created by the subparser to Arcsight fields
submessage.messageid.token=ProcessName
submessage.token=Message
submessage.count=1
Submessage ID Value: VPN
Submessage token Value: Encryption failed, username jsmith
Submessage Regex: (.*), username (\S+)
submessage[0].messageid=VPN
submessage[0].pattern.count=1
submessage[0].pattern[0].regex=(.*), username (\\S+)
submessage[0].pattern[0].fields=event.name,event.destinationUserName

36. Regex Parser - submessages
2006:11:07-19:05:29 ulogd[1993]: DROP: <1-10 different drop Messages> 2006:11:07-19:05:29 ulogd[1993]: ACCEPT: <1-10 different drop Messages> 2006:11:07-19:05:29 ulogd[1993]: REJECT: <1-10 different drop Messages> Common Regex will cover 2006:11:07-19:05:29 ulogd[1993]: Submessage ID will be DROP – ACCEPT - REJECT

37. Completed Configuration File
# FlexConnector Regex Configuration File
regex=(\\d+.\\d+.\\d+.\\d+)\:(\\S+) (\\d+/\\d+/\\d+ \\d+\:\\d+\:\\d+) (\\d+.\\d+.\\d+.\\d+)\:(\\d+) (\\d+.\\d+.\\d+.\\d+)\:(\\d+) \\((\\S+)\\) action\=(\\S+) Message\:\\s(.*)?Rule (\\d+)
token.count=11
token[0].name=Firewall_IP
token[0].type=IPAddress
token[1].name=ProcessName
token[1].type=String
token[2].name=Event_Time
token[2].type=TimeStamp
token[2].format=yyyy/MM/dd HH\:mm\:ss
token[3].name=SourceIP
token[3].type=IPAddress
token[4].name=SourcePort
token[4].type=Integer
token[5].name=DestinationIP
token[5].type=IPAddress
token[6].name=DestinationPort
token[6].type=Integer
token[7].name=Protocol
token[7].type=String
token[8].name=Action
token[8].type=String
token[9].name=Message
token[9].type=String
token[10].name=RuleNumber
token[10].type=Integer
submessage.messageid.token=ProcessName
submessage.token=Message
event.deviceReceiptTime=Event_Time
event.sourceAddress=SourceIP
event.deviceAddress=Firewall_IP
event.destinationAddress=DestinationIP
event.sourcePort=SourcePort
event.name=Action
event.message=__concatenate(Message," ","Rule:"," ",RuleNumber)
event.destinationPort=DestinationPort
event.deviceSeverity=Action
event.transportProtocol=Protocol
event.deviceProcessName=ProcessName
submessage.count=1
submessage[0].messageid=VPN
submessage[0].pattern.count=1
submessage[0].pattern[0].regex=(.*), username (\\S+)
submessage[0].pattern[0].fields=event.name,event.destinationUserName
severity.map.high.if.deviceSeverity=drop
severity.map.low.if.deviceSeverity=accept

38. Multiline Regex
Configuration parameters identify the start and/or end of each event
multiline.starts.regex
A regular expression that identifies when a multi-line event starts.
Example: multiline.starts.regex=\|\d+/\d+/\d+ \d+:\d+:\d+\|.*
multiline.ends.regex (optional)
A regular expression that identifies when a multi-line event ends.
Example: multiline.ends.regex=.*\|$
multiline.max.count
Over-flow protection. The FlexConnector will truncate the message if it reaches the specified number of lines.
multiline.delimiter
Lines are concatenated with a space by default. This parameter is used to change the default character.

39. Extra Processors (parser chaining)
Use an extra processor when all or portion of data is suitable for parsing by other parsers
Extra processor type
Which parser is used
map
Map files
delimited
Delimited parser
regex
Regular expression parser
keyvalue
Key value parser
standardkeyvalue
Key value parser with standard delimiters (comma and equals to)
xml
XQuery-based XML parser
ntsubparser
Windows event log parsers

40. Parser Chaining Example
Two or more FlexConnector types are needed to parse the same data.
Example Database Log:
A regular expression can be used to parse the fifth column.
04/06/06 13:03:34
2435 80
Firewall accepted a tcp connection
04/06/06 13:03:54 53
Firewall denied a udp connection
04/06/06 13:04:23
2463 22
Firewall denied a tcp connection

41. Conditional Mapping For example, assume the following event:
Event id is 532 type A with parameter
Event id is 533 type A with parameter root
Event id is 534 type A with parameter
The regular expression to parse this event is:
Event id is (\\d+) type (\\S+) with parameter (\\S+)
You can define three tokens: EVENTID, TYPE, and PARAMETER
For event id 532 or 534, set event.sourceAddress to
For Event id 533, set event.sourceUserName to root
Without conditional mappings, two regular expressions needed to match—the IP address and the user name
Feasible in this case, but will not scale well

42. Conditional Mapping
Conditional mappings in properties for the above example:
regex=Event id is (\\d+) type (\\S+) with parameter (\\S+)
token.count=3
token[0].name=EVENTID
token[1].name=TYPE
token[2].name=PARAMETER
#Standard mappings
event.deviceEventClassId=EVENTID
event.deviceEventCategory=TYPE
conditionalmap.count=1
conditionalmap[0].field=event.deviceEventClassId
conditionalmap[0].mappings.count=2
conditionalmap[0].mappings[0].values=532,534
conditionalmap[0].mappings[0].event.sourceAddress=PARAMETER
conditionalmap[0].mappings[1].values=533
conditionalmap[0].mappings[1].event.sourceUserName=PARAMETER
conditionalmap[0].mappings[2].event.destinationAddress=PARAMETER (DEFAULT)

43. WUC (Windows Unified Connector)

44. Windows Unified Connector - WUC
Native Java implementation of Windows event collection based on JCIFS
Architecture requires separate parsers for each type of event
Only certain types of Events are supported by WUC oob

45. WUC - Features and Capabilities
A flexible, scalable and robust mechanism to collect Windows events
Platform independent
Deployable on Windows / Linux / Solaris / Connector Appliance
Device versions supported
Windows Server 2008 R2 / 7 / 2008 / Vista / 2003 / XP / 2000
Event collection supported from varied device configurations
Domain Controllers / Domain Members / Standalone Workgroup hosts
Hosts belonging to multiple different domains
Flexible Architecture and Deployment
Local and Remote Event collection
Distributed vs Centralized Deployment
Scalable to a large number of hosts with low EPS
Bandwidth control
Other Features
NTLMv2 Authentication, Host Browsing, SID/GUID translation
© 2010 ArcSight Confidential 45

46. Event Collection Process
WUC uses multi-threaded event collection, threadcount=10
Each event collection thread is assigned a sub-set of the configured hosts
Each thread
will collect events from its assigned set of hosts in a round-robin manner
For each host, will collect events from all its event logs in a round-robin manner
This constitutes one polling cycle
For aggressive event collection, there is no delay between successive polling cycles, sleeptime=-1
For each polling cycle, the number of events collected per event log per host is 50, eventpollcount=50
If the connection to a host goes down
It attempts to reconnect once more if the reconnection timer has not expired, reconnectinterval=120000
If the connection to a host is up, but no events are retrieved from the event log
It checks for event log rotation since the last check, logrotationcheckinterval=5000
If event log rotation is detected, it will attempt to collect events from that event log maximum 15 times within a period of 15 seconds, whichever happens first, rotationretrycount=15, rotationretryinterval=15000
Multiple hosts can be configured for event collection from multiple event logs
Preservedstatecount=100
Preservedstateinterval=10000
Preservestate=true
Threadcount=10
Sleeptime=-1
Eventpollcount=50
Smbtransporttimeout=30000
Reconnectinterval=120000
Logrotationcheckinterval=5000
Rotationretrycount=15
Rotationretryinterval=15000
© 2010 ArcSight Confidential 46

47. Event Collection vs Event Parsing
WUC provides capability to COLLECT events from all Windows Event Logs
Security Event Log
System Event Log
Common Application Event Log
Custom Application Event Logs
WUC provides different levels of parsing capabilities
All Security, Core System and Application Events: Complete parsing
Other System and Application Events: Event Header Completely parsed by WUC
Event Body/Description: Use WUC Flex Parser Framework to create flex parsers to parse the Event body

48. WUC Flex Parser Framework
Flexible Parser Framework similar to the ArcSight Flex Connector Framework
Provides the power and flexibility to create new parsers to parse custom System and Application events
Why is it needed?
Security events are generated mostly by the OS
System and Application events are generated by other applications
New applications create new Event Logs (Custom Application Event Logs)
Multiple Event Sources can generate events for the same Event Log
How does it parse the System and Application events?
Pre-parses the Event Header fields
Parses the Event Body with a Key Value Parser for each combination of the event’s:
Windows Version
Event Log
Event Source

49. Windows Event Format 2003 / XP / 2000 System Event
Windows Event = Event Header + Event Description/Body
Event Header format similar for all Windows events
Event Body format differs based on
Windows version
Event Log
Event ID
Event Source
2003 / XP / 2000 System Event
2008 / Vista System Event
Event Body
Event Header
Event Header
Event Body

50. What to Parse?
Event Body = Fixed Description String + Variable Place-holders
WUC Raw Event:
EventlogType=System&&EventSource=IIS-FTP&&EventID=10&&EventType=Warning&& User=&&DetectTime= :25:24&&EventCategory=0&&ComputerName=ABC
Windows 2008 Event from Event Log: System, Event Source: IIS-FTP, Event ID: 10
User %1 at host %2 has timed out after %3 seconds of inactivity.

51. How to create a WUC Flex Parser?
Identify the Event Log name, e.g. System, Application, Directory Service, etc…
Identify the Event Source, e.g. Service Control Manager
Create a Key Value Parser file with the following name format:
Format: <Event Log Name>.<Event Source>.sdkkeyvaluefilereader.properties
Normalize all the characters in the parser file name
Example: system.service_control_manager.sdkkeyvaluefilereader.properties
Identify the Windows version of the host
Place the parser file in one of the following parser over-ride sub-folder locations
Windows Version
Parser Over-ride Sub-Folder Location
Windows Server 2000
…\windowsfg\windows_2000
Windows Server 2003
…\windowsfg\windows_2003
Windows XP
Windows Server 2008
…\windowsfg\windows_2008

52. © 2010 ArcSight Confidential
How to Parse?
# Create a Key Value Parser key.delimiter=&& key.value.delimiter== key.regexp=([^&=]+) additionaldata.enabled=true # WUC Flex Parser Framework pre-parses and maps the Windows Event ID field to the ArcSight event.externalId field # Create a conditional map based on the Event ID conditionalmap.count=1 conditionalmap[0].field=event.externalId conditionalmap[0].mappings.count=1 # User %1 at host %2 has timed out after %2 seconds of inactivity. # EventlogType=System&&EventSource=IIS-FTP&&EventID=10&&EventType=Warning&&User=&&DetectTime= conditionalmap[0].mappings[0].values=10 conditionalmap[0].mappings[0].event.destinationUser=Key[0] conditionalmap[0].mappings[0].event.destinationHostAddress=Key[1] conditionalmap[0].mappings[0].event.deviceCustomNumber1=Key[2] conditionalmap[0].mappings[0].event.deviceCustomNumber1Label=__stringConstant(“Inactivity period in seconds”) conditionalmap[0].mappings[0].event.name=__stringConstant(“User timed out”) conditionalmap[0].mappings[0].event.message=__stringConstant(“User has timed out after a period of inactivity”) event.deviceVendor=__getVendor(Microsoft) event.deviceProduct=__stringConstant(Microsoft Windows)
© 2010 ArcSight Confidential 52

53. DB-Based FlexConnector…..
For template guidelines or applying this template to an existing presentation, see the ArcSight presentation style guide:
Questions, contact 53

54. DB Flex Workflow DBs: Choose ID- or Time-based
Prototype on SQuirreL first – blame the database!
Transfer through to FlexConnector

55. DB Flex What can go wrong? Firewall Rules
Credentials (invalid user/pass)
Authorisation (access to table)
Fields exist, and Datatypes
JDBC Drivers (work, and are compatible)
Connection String
Everything else!
Use SQuirreL. Love SQuirreL.

56. jTDS Why jTDS? Open Source Driver supporting MS SQL Server
Supports Windows Authentication! On ConApp!
Apparently faster and more efficient

57. jTDS Driver download http://jtds.sourceforge.net Driver file goes into
/current/lib/agent/jtds_1.2.5.jar
Database JDBC Classpath (into agent.properties)
agents[0].JDBCDriver=net.sourceforge.jtds.jdbc.Driver
Connection String
Mixed Mode:
jdbc:jtds:sqlserver:// :1433;database=SQLServer
Windows:
jdbc:jtds:sqlserver:// :1433;database=SQLServer;domain=WORKGROUP

58. syslog

59. Syslog Characteristics
Do the messages have a syslog header?
Is the syslog header rfc compliant?
Syslog-ng with IETF (RFC5424) is another story
Your parser applies only to the Payload, not to the Syslog Header
Syslog Header
Payload
Facility + Priority
Host Name
Time Stamp

60. Syslog Parsing Based on a Plugin Framework (subparsers)
Syslog connectors has more 40 subagent parser
When a message is received, it compares the pattern of message to all the available subagent parser till a match is found
First match wins!
Once the subagent parser is matched, it will store this information in syslog.properties of which device events matched which subagent parser
If there is no matched pattern, events will be parsed with the generic parser and store the information in Name field and Device Vendor and Product will be “UNIX”

61. Other Syslog Flex Connector Considerations
If you plan to receive events from multiple devices
You need to write multiple property files, one per device type
Property files are loaded and used in alphabetical order
Regexes defined in properties files need to be unique enough to match only the messages from its device type and not from other device types
Names of the property files should be chosen such that more specific regexes appear in properties files with alphabetically higher names
Other tips
If the actual message has data in the delimited, key value or xml format, the regex need only identify the device and an appropriate extra processor can be used for further parsing

62. Syslog by Example

63. Syslog Example
Nov 08 15:45: :11:07-19:05:29 ulogd[1993]: DROP: IN=eth0 OUT= MAC=00:0c:29:28:fa:4f:00:10:4b:b9:0e:84:08:00 SRC= DST= LEN=48 TOS=00 PREC=0x00 TTL=128 ID=34476 CE DF PROTO=TCP SPT=2591 DPT=1 SEQ= ACK=0 WINDOW= SYN URGP=0

64. Syslog Example Start Here
Nov 08 15:45: :11:07-19:05:29 ulogd[1993]: DROP: IN=eth0 OUT= MAC=00:0c:29:28:fa:4f:00:10:4b:b9:0e:84:08:00 SRC= DST= LEN=48 TOS=00 PREC=0x00 TTL=128 ID=34476 PROTO=TCP SPT=2591 DPT=1 SEQ= ACK=0 WINDOW=64240 SYN URGP=0
Syslog subagents are regex parsers
This log seems very easy to parse (key/value pairs) but…
What about optional fields
What about the order of the fields
Yes, we can make tokens optional in a regular expression –> inefficient

65. Parser chaining (extraprocessors)
Use a regex for: 2006:11:07-19:05:29 ulogd[1993]: DROP:
Use a keyvalue parser for the rest of the message IN=eth0 OUT= MAC=00:0c:29:28:fa:4f:00:10 SRC= DST= LEN=48 TOS=00 PREC=0x00 TTL=128 ID= PROTO=TCP SPT=2596 DPT=9 SEQ= ACK=0 WINDOW=64240 SYN URGP=0

66. Keyvalue parser advantages / disadvantages
High performance
Simple Configuration
Arbitrary Key Order
Ignores Missing Keys
Easier to maintain
Only tool available: notepad/vi
More difficult to debug

67. ArcSight Regex Tool
Start building your base Connector with ArcSight Regex tool /bin/arcsight regex
Treat As Syslog
Be as specific as possible
Complete basic mappings
Test with a large sample
Then move on to extraprocessor

68. Now move on to your favorite editor
extraprocessor.count=1 extraprocessor[0].type=keyvalue extraprocessor[0].filename=syslog/astaro/astaro extraprocessor[0].field=event.message extraprocessor[0].clearfieldafterparsing=true extraprocessor[0].flexagent=true

69. astaro.sdkkeyvaluefilereader.properties
Event mappings event.sourceAddress=SRC event.deviceInboundInterface=IN event.deviceOutboundInterface=OUT event.transportProtocol=PROTO event.destinationAddress=DST event.destinationPort=DPT event.sourcePort=SPT
Token definitions
token.count=13
token[0].name=IN
token[0].type=String
token[1].name=OUT
token[1].type=String
token[2].name=SRC
token[2].type=IPAddress
token[3].name=DST
token[3].type=IPAddress
token[4].name=LEN
token[4].type=String
token[5].name=TOS
token[5].type=String
token[X].name=…

70. Test your Connector Set up a syslog deamon
Feed samples into the connector
Watch the agent.log for ERROR and FATAL
[ :26:44,441][FATAL][default.com.arcsight.agent.parsers.k][readFieldMappings] Could not load operation [__stringTESTConstant(Rule)]
[ :26:44,449][FATAL][default.com.arcsight.agent.parsers.k][readFieldMappings]
com.arcsight.agent.parsers.operation.OperationNotSupportedException: Operation [stringTESTConstant] not supported !
at com.arcsight.agent.parsers.operation.OperationLoader.loadOperation(OperationLoader.java:45)
at com.arcsight.agent.parsers.operation.OperationLoader.getOperationIndex(OperationLoader.java:68)
at com.arcsight.agent.parsers.j$e_.<init>(j$e_.java:1283)
at com.arcsight.agent.parsers.j.k(j.java:579)
at com.arcsight.agent.parsers.j.d(j.java:503)
at com.arcsight.agent.parsers.j.u(j.java:347)

71. Categorization
Categorization of FlexConnectors 71

72. What is Categorization?
Categorization allows you to look at an event in a vendor neutral context.
Apply Event Categories — six criteria translate the core meaning of an event into ArcSight’s Event Schema
Object - Entity being targeted
Behavior - What is being done to the object
Outcome - Result of the Behavior on the object (Success, Failure or Attempt)
Technique - Nature of the behavior represented
Device Group - Indicates if event is of one type or another (Such as Firewall Events)
Significance - Indicates security risk based on various data points, information from the device, and the data model

73. Categorisation If you have time, or is a POC requirement
Beware of existing content triggering on this (eg. Config Changes, Brute Forces)
ESM 4.0 User Reference Guide / Data Fields ESM 5.0 ESM_UserGuide.pdf 73

74. Categorization files are map files
Place categorization files in
<connector_home>/user/agent/acp/categorizer/current/<vendor_name>/
Create a file product.csv which will reside inside of the directory
Vendor and product must match what is being assigned in the connector configuration file

75. Categorization File Product.csv File should contain a header like this
event.deviceEventClassId,set.event.categoryObject,set.event.categoryBehavior,set.event.categoryTechnique,set.event.categoryDeviceGroup,set.event.categorySignificance,set.event.categoryOutcome
The key field is deviceEventClassId this is how we assign the categorization to the events

76. Documentation Flexcon Dev Guide sucks but is a good starting point
Review PS / Support Training recordings
Confluence!

77. The Burden With The Escape Characters
In literal Java strings the backslash is an escape character
The literal string "\\" is a single backslash
In regular expressions, the backslash is also an escape character
The regular expression \\ matches a single backslash
This regular expression as a Java string, becomes "\\\\" That's right: 4 backslashes to match a single one

78. Some Useful Undocumented Token Operations
__divide(integer, integer)
__sum(integer, integer [, integer...])
__substract(...
__product(...
Curious for more?
Unzip the arcsight-parserframework<version>.jar
Look into \com\arcsight\agent\parsers\operation

79. Event merging
Event Merger Some devices will send information about a single event in multiple log lines Even though in some cases it would be fine to send each line as a single event, in some other instances it is necessary to merge the information of all the events into a single one. Event Merger vs. Multi-line regex One could argue that a multi-line regex agent could be developed for cases where multiple events have to be merged into a single one In some instances the events sent by the device will not necesarilly be close together, there could be events that will be sent in between other events

80. Event merging Consider the following log lines:
[18/Jul/2005:12:30: ] conn=8 op=0 msgId=82 - BIND uid=admin
[18/Jul/2005:12:30: ] conn=7 op=-1 msgId=-1 - LDAP connection from to  
[18/Jul/2005:12:30: ] conn=8 op=0 msgId=82 - RESULT err=0
Two of those lines refer to a "binding" operation where the user id is admin and the error is 0. This example could NOT be solved using a multiline.
Both events have a connection (conn and a message id (msgId) that is identical for both events
Search Confluence for „Event Merger“

81. Event Name Event names should not have variables The Good The Bad
%PIX : Denied Telnet login session from on interface inside
The Good
Denied Telnet
The Bad
Denied Telnet login session from
thE UgLy
%PIX : Denied Telnet login session from on interface inside
Several ArcSight features (event graphs, data monitors, etc…) use the event name as a unique identifier of the event type, so when selecting a field to store the event name, make sure you pick a field that contains a fixed name and no variable elements (like ip address, port, etc…)

82. ArcSight Regex Tool
Always use the version of the target connector (latest if possible)
Delete tmp and bak files after using it
Stop using it once you have modified properties file outside of the Tool
Rememer to double-escape when editing with external editor (\s+) turns into (\\s+)

83. Regular Expressions
Regular expressions should be as specific as possible
abc,def,ghi
The Good
\\S+?,\\S+?,\\S+? or even better [^,]+,[^,]+,[^,]+
The Bad
\\S+,\\S+,\\S+
thE UgLy
.*,.*,.*
.* is EVIL. NEVER use more than one of these in the regex, and that too should preferably be at the end.
.*? is almost as evil. NEVER use more than one.
a cool regex trick: if you think you'll be receiving a multiline string: try (?s) at the beginning. actually this can be scoped!

84. Regular expressions for Syslog sub-agents
Generic regular expressions may cause legitimate Unix syslog
messages to not be detected
abc,def,ghi
The Good (Order is from most specific to most generic)
abc,def,ghi (exact match)
[^,]+,[^,]+,[^,]+
\\S+?,\\S+?,\\S+?
.*?,.*?,.*
The Bad/thE UgLy (Order is from most generic to most specific)
.*?,.*?,.* (generic catch-all)
abc,deg,ghi
It is critical that syslog flexagents have 'tight' regular expressions. Generic regexes may cause legitimate unix syslog messages to not be detected (since the autoload subagent is higher than unix syslog in the heirarchy).

85. Syslog subagent Flex: Other parser matches first
If possible, remove other subagents from framework
Set usecustomsubagentlist to true
Remove parser from customsubagentlist
If you still need the other parser
Change parser order

86. Fields to map sourceAddress destinationAddress deviceReceiptTime
deviceEventClassId
deviceVendor
deviceProduct
name
Categorization depends on these!

87. Custom Fields
Each Device Custom field has a corresponding Device Custom label, it is a good practice to make sure the labels are also set

88. Device Receipt Time, Start Time, End Time
What is this? Time when the device “detected” the event
Should this be set? YES! This MUST be set in the parser
Start Time:
What is this? Time when the actual event detection started
Should this be set? Only when device calls it out explicitly
End Time:
What is this? Time when the actual event detection ended or when the actual event ended.
Should this be set? Only when device calls it out explicitly
Start time: Most devices will NOT send this information Exception: Scanner agents (sets time to when scan started)
Start Time and End Time are inferred from deviceReceiptTime.

89. Sidetables Only relevant to ESM/Express based on Oracle
Certain fields with highly repetitive content are cached in sidetables
Mapping wrong information into those fields will cause sidetable overflow
Wrong field mapping can kill or slow-down ESM (pre-CORRE)

90. Fixing existing parsers (SmartConnectors)

91. Parser overrides
property file contains the modified information ONLY to replace the corresponding line(s) in the original parser file, in order to change the data mapping
Must have the same parser file name as the original parser
Placed under \current\user\agent\fcp\<folder_ name>\<Parser_file_name>
The <folder_name> must be same as the original parser file’s folder name

92. Unobfuscated property file
# Unobfuscated properties file # Copyright ArcSight, Inc. All Rights Reserved. # This software is the proprietary information of ArcSight, Inc. # Use is subject to license terms. # IMPORTANT: Delete the following property when you modify this # file! ignore.this.file=x # Delete the following property if this file contains only # overrides, or leave it as is if this file is intended to be # complete replace.defaults=x #:::::::::::::::::::::::::::::::::::::::::::::: # Apache sdk regex properties file: For Apache access and error logs regex=(apache_access_log|apache_error_log):\\s*(?:\\[ID (\\S+?) (\\S+?)\\.(\\S+?)\\])?\\s*(.*) token.count=5

93. Parser overrides
Only the changes need to go into the prop file when you remove: “replace.defaults=x” regex=(apache_access_log|apache_error_log):\\s*(?:\\[ID (\\S+?) (\\S+?)\\.(\\S+?)\\])?\\s*(.*)

94. How to get the original parsers
Ask a friend in support 
Unobfuscate yourself
Start Connector on commandline: ./arcsight agents –unobpswd <code>
Parsers will end up in folder: user/agent/aup/fcp
Code changes monthly and has to match either
Month when Connector build was produced
Month of AUP version deployed on top of that Connector
Need a code? Ask a colleague from the ArcSight Specialists team

95. Thank you