Download presentation
Presentation is loading. Please wait.
1
ArcSight: FlexConnectors
Introduction to the ArcSight FlexConnector Framework Till Jäger, Solution Architect CISSP
2
What this is (and what it is not)
This is not A SQL Training A Regex Training A FlexConnector Training So what is it then? Broad overview of functionality Guide how to approach your problem Help to get started
3
Agenda Connector basics Flexconnectors basics Tools Regex parsers
WUC parsers DB Connectors Syslog basics Connector by example: Syslog Categorization Tips & tricks / Best practice Parser overriding
4
SmartConnector Basics
5
What is a SmartConnector?
A SmartConnector is software that collects events from end-point devices, normalizes the events, send them to an ArcSight destination.
6
SmartConnector Event Flow
7
SmartConnector Types File Connectors Database Connectors
Scanner Connectors API Connectors SNMP Connectors Microsoft Windows Event Log Connectors Syslog Connectors FlexConnectors Model Import Connectors NetFlow Connector
8
SmartConnector Directory Structure
/current/bin – executables and scripts Arcsight.bat / arcsight.sh /current/config/agent – default/base configurations Agent.defaults.properties Agent.wrapper.conf.base /current/logs – SmartConnector generated logs Agent.log.n Agent.out.wrapper.n /current/user /agent
9
SmartConnector Directory Structure
/current/user/agent – Connector properties and destination-specific configurations, preserve state files Agent.properties Hosts.txt Persisted.properties <agentid>.xml config files Syslog.properties /current/user/agent/agentdata – queue, cache, persistence files .prstdout.n files cache.dflt.n files Syslogd.n files /current/user/agent/aup – directory for content aup, additional data mappings, zones Destination specific directories Additional data mapping files – ngadatamapping.properties /current/user/agent/acp – directory for categorization files Categorization files – content aup
10
SmartConnector Directory Structure
/current/user/agent/fcp – directory for parser overrides Parser overrides /current/user/agent/flexagent – directory for custom parsers, and regex tester Custom parsers /current/user/agent/map – directory for map.n.properties files Map.n.properties
11
SmartConnector Configuration Files
Location of the Configuration files: /current/user/agent/ agent.properties contains the global configuration Agent<id>.xml contains the destination specific configuration
12
SmartConnector Configuration Files
agent.wrapper.conf Service configuration file that includes the wrapper configuration E.g. increasing the heap size, extending ping timeouts,.. agent.defaults.properties Located in /current/config/agent Contains the default framework parameters Contains syntax for enabling debugging and increasing agent log file size, and agent log count
13
Additional Data Mappings / map.x.properties
Often not all fields are explicitly mapped to schema fields Mapped to additional data Can be mapped on-demand from Console Right-click on the SmartConnector, and send the command “get additional data names” map.x.properties useful for adding additional details to events Uses “getters” and “setters” For example: When source user name is John, set Device Custom String 1 to John’s phone event.sourceUserName, set.event.deviceCustomString1 John, /current/user/agent/map
14
Categorization ArcSight Content
Provides additional meaning to the event Requires 3 fields to be populated to work: Device Event Class ID: Device Product: Device Vendor Special kind of map file in user/agent/acp/categorizer/current Required for Foundation Content / System Content to work properly
15
Log Analysis If anything goes wrong: /logs/agent.log Search for ERROR
For log trend analysis use LogFu LogFu is an ArcSight log Analysis Utility Part of every SmartConnector install Open a CMD Prompt CD to /current/logs \logs>..\bin\arcsight agent logfu -a
16
Log Analysis: LogFu!
17
FlexConnectors Choose The Right Type
18
Before You Get Started Read the Flexconnector Guide
Familiarize with the Event Schema Download latest Connector build Get to know the basics of Regex
19
FlexConnector Types File Reader
Multi Folder Reader (Batch and/or Realtime) Syslog Subagent SNMP Database Vulnerability Scanner Model Import WUC (Windows Unified Connector) parser
20
FlexConnector Parser Types
Parser File name for each parser type has a unique extension Delimited Log Parser (sdkfilereader) Regex Parser (sdkrfilereader) Key Value parser (sdkkeyvaluefilereader) Database parser (sdktbdatabase, sdkibdatabase) SNMP parser (sdksnmp.X.snmptrap) XML parser (xqueryparser) A combination of those (more later)
21
Log File Characteristics
Log File Flex Connectors Log File Characteristics Number of files Single file or multiple files? Do we know the exact file names? If file names change, do they have an expressible pattern? Access to the files Do we have access to the files locally or remotely? If not they may have to be copied periodically Static or Dynamic Data Is data in the file growing? Are the files static and generated by the device or copied from the device periodically?
22
Log File Characteristics
Log File Flex Connectors Log File Characteristics What kind of data is in the file? Delimited Data Free Form Data Key Value Pairs XML
23
Choose The Right Type
24
Did You Know that… You can re-use a connector install by deleting the agent.properties and run „./arcsight agentsetup“ again You can copy an installed connector to a new location and re-use it A Connector usually doesn‘t need Administrative/root access You don‘t need a running ESM/logger to test your Connector
25
FlexConnectors Structure of a properties file
26
Configuration File comments.start.with=# delimiter=,
contains.empty.tokens=true token.count=9 event.name=Action event.message=__concatenate(Action,”; ”,Message) event.deviceReceiptTime=Event_Time event.attackerAddress=SourceIP event.attackerPort=SourcePort event.targetAddress=DestinationIP event.targetPort=DestinationPort event.transportProtocol=Protocol event.deviceAddress=FirewallIP event.deviceSeverity=Action event.deviceCustomNumber1=__regexToken(Message,[^.]\. ? Rule (\\d+)) event.deviceCustomNumber1Label=__stringConstant(“Rule Number”) token[0].name=FirewallIP token[0].type=IPAddress token[1].name=Event_Time token[1].type=TimeStamp token[1].format=yyyy/MM/dd HH:mm:ss token[2].name=SourceIP token[2].type=IPAddress token[3].name=SourcePort token[3].type=Integer token[4].name=DestinationIP token[4].type=IPAddress token[5].name=DestinationPort token[5].type=Integer token[6].name=Protocol token[6].type=String token[7].name=Action severity.map.high.if.deviceSeverity=drop severity.map.low.if.deviceSeverity=accept 26
27
Double Underscore Operators
More listed in the flex developers guide Operator Meaning Example __stringConstant() Specifies a constant string for event mapping __stringConstant(“This is the Event Name”) __concatenate() Concatenates two or more fields __concatenate(Token1,Token2) __regexToken() Used to parse a field one time __regexToken($1,User \([^)])\)(.*)) __toUpperCase() Converts a string to all upper case __toUpperCase(TokenName) __createTimeStamp Creates a timestamp from a date field and a time field __createTimeStamp(Date,Time) 27
28
FlexConnectors Tools
29
Tools FlexConnector wizard for delimited logs ArcSight Regex Tool
Flexcon wizard in Conapp Notepad++ We!Analyze Netsend Tail for Windows RegexBuddy SQuirreL SQL Client
30
Tools: Regex Buddy + Most comprehensive tool
+ Many Regex dialects, great library + Easy to performance optimize expression - commercial
31
Tools For JDBC Connectors: SQuirreL
If SQuirreL works, your connector will work! Test Connectivity, Credentials, Drivers, Connection String, etc!
32
FlexConnectors Regex Parsers
33
Regex Configuration File
Parser Configuration Common regular expression to match all entries in the log file :FW 2002/10/01 18:12: : :80 (tcp) action=drop Message: Rule 25 :VPN 2002/10/01 18:12: : :80 (tcp) action=drop Message: Encryption failed, username jsmith Rule5 regex=(\d+.\d+.\d+.\d+):(\S+) (\d+/\d+/\d+ \d+:\d+:\d+) (\d+.\d+.\d+.\d+):(\d+) (\d+.\d+.\d+.\d+):(\d+) \((\S+)\) action=(\S+) Message:\s(.*)?Rule\s+(\d+) Token Declaration Each value in parentheses will be tokenized: token.count=11 token[0].name=Firewall_IP token[0].type=IPAddress token[1].name=ProcessName token[1].type=String token[2].name=Event_Time token[2].type=TimeStamp token[2].format=yyyy/MM/dd HH\:mm\:ss token[3].name=SourceIP token[3].type=IPAddress token[4].name=SourcePort token[4].type=Integer token[5].name=DestinationIP token[5].type=IPAddress token[6].name=DestinationPort token[6].type=Integer token[7].name=Protocol token[7].type=String token[8].name=Action token[8].type=String token[9].name=Message token[9].type=String token[10].name=RuleNumber token[10].type=Integer
34
Regex Parser - submessages
Common regex regex - main or common regular expression, for some parsers, this is sufficient Submessage Processing submessage.messageid.token, token that specifies a unique message id submessage.token, token that should be processed as submessage submessage.count, total number of submessages submessage[N].messageid [if omitted, becomes the default submessage] submessage[N].pattern.count, how many different patterns a submessage can have submessage[N].pattern[M].regex, regex for the pattern submessage[N].pattern[M].fields, what fields will be set for the pattern submessage[N].pattern[M].mappings [optional, what values will be set to fields] submessage[N].pattern[M].extramappings [optional, any extra mappings]
35
Regex - submesages Subparser regular expressions
Suparsers extract additional information from fields created by the common regular expression Select the submessage ID Token Select the token to be parsed Create a regular expression to parse the submessage Map the tokens created by the subparser to Arcsight fields submessage.messageid.token=ProcessName submessage.token=Message submessage.count=1 Submessage ID Value: VPN Submessage token Value: Encryption failed, username jsmith Submessage Regex: (.*), username (\S+) submessage[0].messageid=VPN submessage[0].pattern.count=1 submessage[0].pattern[0].regex=(.*), username (\\S+) submessage[0].pattern[0].fields=event.name,event.destinationUserName
36
Regex Parser - submessages
2006:11:07-19:05:29 ulogd[1993]: DROP: <1-10 different drop Messages> 2006:11:07-19:05:29 ulogd[1993]: ACCEPT: <1-10 different drop Messages> 2006:11:07-19:05:29 ulogd[1993]: REJECT: <1-10 different drop Messages> Common Regex will cover 2006:11:07-19:05:29 ulogd[1993]: Submessage ID will be DROP – ACCEPT - REJECT
37
Completed Configuration File
# FlexConnector Regex Configuration File regex=(\\d+.\\d+.\\d+.\\d+)\:(\\S+) (\\d+/\\d+/\\d+ \\d+\:\\d+\:\\d+) (\\d+.\\d+.\\d+.\\d+)\:(\\d+) (\\d+.\\d+.\\d+.\\d+)\:(\\d+) \\((\\S+)\\) action\=(\\S+) Message\:\\s(.*)?Rule (\\d+) token.count=11 token[0].name=Firewall_IP token[0].type=IPAddress token[1].name=ProcessName token[1].type=String token[2].name=Event_Time token[2].type=TimeStamp token[2].format=yyyy/MM/dd HH\:mm\:ss token[3].name=SourceIP token[3].type=IPAddress token[4].name=SourcePort token[4].type=Integer token[5].name=DestinationIP token[5].type=IPAddress token[6].name=DestinationPort token[6].type=Integer token[7].name=Protocol token[7].type=String token[8].name=Action token[8].type=String token[9].name=Message token[9].type=String token[10].name=RuleNumber token[10].type=Integer submessage.messageid.token=ProcessName submessage.token=Message event.deviceReceiptTime=Event_Time event.sourceAddress=SourceIP event.deviceAddress=Firewall_IP event.destinationAddress=DestinationIP event.sourcePort=SourcePort event.name=Action event.message=__concatenate(Message," ","Rule:"," ",RuleNumber) event.destinationPort=DestinationPort event.deviceSeverity=Action event.transportProtocol=Protocol event.deviceProcessName=ProcessName submessage.count=1 submessage[0].messageid=VPN submessage[0].pattern.count=1 submessage[0].pattern[0].regex=(.*), username (\\S+) submessage[0].pattern[0].fields=event.name,event.destinationUserName severity.map.high.if.deviceSeverity=drop severity.map.low.if.deviceSeverity=accept
38
Multiline Regex Configuration parameters identify the start and/or end of each event multiline.starts.regex A regular expression that identifies when a multi-line event starts. Example: multiline.starts.regex=\|\d+/\d+/\d+ \d+:\d+:\d+\|.* multiline.ends.regex (optional) A regular expression that identifies when a multi-line event ends. Example: multiline.ends.regex=.*\|$ multiline.max.count Over-flow protection. The FlexConnector will truncate the message if it reaches the specified number of lines. multiline.delimiter Lines are concatenated with a space by default. This parameter is used to change the default character.
39
Extra Processors (parser chaining)
Use an extra processor when all or portion of data is suitable for parsing by other parsers Extra processor type Which parser is used map Map files delimited Delimited parser regex Regular expression parser keyvalue Key value parser standardkeyvalue Key value parser with standard delimiters (comma and equals to) xml XQuery-based XML parser ntsubparser Windows event log parsers
40
Parser Chaining Example
Two or more FlexConnector types are needed to parse the same data. Example Database Log: A regular expression can be used to parse the fifth column. 04/06/06 13:03:34 2435 80 Firewall accepted a tcp connection 04/06/06 13:03:54 53 Firewall denied a udp connection 04/06/06 13:04:23 2463 22 Firewall denied a tcp connection
41
Conditional Mapping For example, assume the following event:
Event id is 532 type A with parameter Event id is 533 type A with parameter root Event id is 534 type A with parameter The regular expression to parse this event is: Event id is (\\d+) type (\\S+) with parameter (\\S+) You can define three tokens: EVENTID, TYPE, and PARAMETER For event id 532 or 534, set event.sourceAddress to For Event id 533, set event.sourceUserName to root Without conditional mappings, two regular expressions needed to match—the IP address and the user name Feasible in this case, but will not scale well
42
Conditional Mapping Conditional mappings in properties for the above example: regex=Event id is (\\d+) type (\\S+) with parameter (\\S+) token.count=3 token[0].name=EVENTID token[1].name=TYPE token[2].name=PARAMETER #Standard mappings event.deviceEventClassId=EVENTID event.deviceEventCategory=TYPE conditionalmap.count=1 conditionalmap[0].field=event.deviceEventClassId conditionalmap[0].mappings.count=2 conditionalmap[0].mappings[0].values=532,534 conditionalmap[0].mappings[0].event.sourceAddress=PARAMETER conditionalmap[0].mappings[1].values=533 conditionalmap[0].mappings[1].event.sourceUserName=PARAMETER conditionalmap[0].mappings[2].event.destinationAddress=PARAMETER (DEFAULT)
43
WUC (Windows Unified Connector)
44
Windows Unified Connector - WUC
Native Java implementation of Windows event collection based on JCIFS Architecture requires separate parsers for each type of event Only certain types of Events are supported by WUC oob
45
WUC - Features and Capabilities
A flexible, scalable and robust mechanism to collect Windows events Platform independent Deployable on Windows / Linux / Solaris / Connector Appliance Device versions supported Windows Server 2008 R2 / 7 / 2008 / Vista / 2003 / XP / 2000 Event collection supported from varied device configurations Domain Controllers / Domain Members / Standalone Workgroup hosts Hosts belonging to multiple different domains Flexible Architecture and Deployment Local and Remote Event collection Distributed vs Centralized Deployment Scalable to a large number of hosts with low EPS Bandwidth control Other Features NTLMv2 Authentication, Host Browsing, SID/GUID translation © 2010 ArcSight Confidential 45
46
Event Collection Process
WUC uses multi-threaded event collection, threadcount=10 Each event collection thread is assigned a sub-set of the configured hosts Each thread will collect events from its assigned set of hosts in a round-robin manner For each host, will collect events from all its event logs in a round-robin manner This constitutes one polling cycle For aggressive event collection, there is no delay between successive polling cycles, sleeptime=-1 For each polling cycle, the number of events collected per event log per host is 50, eventpollcount=50 If the connection to a host goes down It attempts to reconnect once more if the reconnection timer has not expired, reconnectinterval=120000 If the connection to a host is up, but no events are retrieved from the event log It checks for event log rotation since the last check, logrotationcheckinterval=5000 If event log rotation is detected, it will attempt to collect events from that event log maximum 15 times within a period of 15 seconds, whichever happens first, rotationretrycount=15, rotationretryinterval=15000 Multiple hosts can be configured for event collection from multiple event logs Preservedstatecount=100 Preservedstateinterval=10000 Preservestate=true Threadcount=10 Sleeptime=-1 Eventpollcount=50 Smbtransporttimeout=30000 Reconnectinterval=120000 Logrotationcheckinterval=5000 Rotationretrycount=15 Rotationretryinterval=15000 © 2010 ArcSight Confidential 46
47
Event Collection vs Event Parsing
WUC provides capability to COLLECT events from all Windows Event Logs Security Event Log System Event Log Common Application Event Log Custom Application Event Logs WUC provides different levels of parsing capabilities All Security, Core System and Application Events: Complete parsing Other System and Application Events: Event Header Completely parsed by WUC Event Body/Description: Use WUC Flex Parser Framework to create flex parsers to parse the Event body
48
WUC Flex Parser Framework
Flexible Parser Framework similar to the ArcSight Flex Connector Framework Provides the power and flexibility to create new parsers to parse custom System and Application events Why is it needed? Security events are generated mostly by the OS System and Application events are generated by other applications New applications create new Event Logs (Custom Application Event Logs) Multiple Event Sources can generate events for the same Event Log How does it parse the System and Application events? Pre-parses the Event Header fields Parses the Event Body with a Key Value Parser for each combination of the event’s: Windows Version Event Log Event Source
49
Windows Event Format 2003 / XP / 2000 System Event
Windows Event = Event Header + Event Description/Body Event Header format similar for all Windows events Event Body format differs based on Windows version Event Log Event ID Event Source 2003 / XP / 2000 System Event 2008 / Vista System Event Event Body Event Header Event Header Event Body
50
What to Parse? Event Body = Fixed Description String + Variable Place-holders WUC Raw Event: EventlogType=System&&EventSource=IIS-FTP&&EventID=10&&EventType=Warning&& User=&&DetectTime= :25:24&&EventCategory=0&&ComputerName=ABC Windows 2008 Event from Event Log: System, Event Source: IIS-FTP, Event ID: 10 User %1 at host %2 has timed out after %3 seconds of inactivity.
51
How to create a WUC Flex Parser?
Identify the Event Log name, e.g. System, Application, Directory Service, etc… Identify the Event Source, e.g. Service Control Manager Create a Key Value Parser file with the following name format: Format: <Event Log Name>.<Event Source>.sdkkeyvaluefilereader.properties Normalize all the characters in the parser file name Example: system.service_control_manager.sdkkeyvaluefilereader.properties Identify the Windows version of the host Place the parser file in one of the following parser over-ride sub-folder locations Windows Version Parser Over-ride Sub-Folder Location Windows Server 2000 …\windowsfg\windows_2000 Windows Server 2003 …\windowsfg\windows_2003 Windows XP Windows Server 2008 …\windowsfg\windows_2008
52
© 2010 ArcSight Confidential
How to Parse? # Create a Key Value Parser key.delimiter=&& key.value.delimiter== key.regexp=([^&=]+) additionaldata.enabled=true # WUC Flex Parser Framework pre-parses and maps the Windows Event ID field to the ArcSight event.externalId field # Create a conditional map based on the Event ID conditionalmap.count=1 conditionalmap[0].field=event.externalId conditionalmap[0].mappings.count=1 # User %1 at host %2 has timed out after %2 seconds of inactivity. # EventlogType=System&&EventSource=IIS-FTP&&EventID=10&&EventType=Warning&&User=&&DetectTime= conditionalmap[0].mappings[0].values=10 conditionalmap[0].mappings[0].event.destinationUser=Key[0] conditionalmap[0].mappings[0].event.destinationHostAddress=Key[1] conditionalmap[0].mappings[0].event.deviceCustomNumber1=Key[2] conditionalmap[0].mappings[0].event.deviceCustomNumber1Label=__stringConstant(“Inactivity period in seconds”) conditionalmap[0].mappings[0].event.name=__stringConstant(“User timed out”) conditionalmap[0].mappings[0].event.message=__stringConstant(“User has timed out after a period of inactivity”) event.deviceVendor=__getVendor(Microsoft) event.deviceProduct=__stringConstant(Microsoft Windows) © 2010 ArcSight Confidential 52
53
DB-Based FlexConnector…..
For template guidelines or applying this template to an existing presentation, see the ArcSight presentation style guide: Questions, contact 53
54
DB Flex Workflow DBs: Choose ID- or Time-based
Prototype on SQuirreL first – blame the database! Transfer through to FlexConnector
55
DB Flex What can go wrong? Firewall Rules
Credentials (invalid user/pass) Authorisation (access to table) Fields exist, and Datatypes JDBC Drivers (work, and are compatible) Connection String Everything else! Use SQuirreL. Love SQuirreL.
56
jTDS Why jTDS? Open Source Driver supporting MS SQL Server
Supports Windows Authentication! On ConApp! Apparently faster and more efficient
57
jTDS Driver download http://jtds.sourceforge.net Driver file goes into
/current/lib/agent/jtds_1.2.5.jar Database JDBC Classpath (into agent.properties) agents[0].JDBCDriver=net.sourceforge.jtds.jdbc.Driver Connection String Mixed Mode: jdbc:jtds:sqlserver:// :1433;database=SQLServer Windows: jdbc:jtds:sqlserver:// :1433;database=SQLServer;domain=WORKGROUP
58
syslog
59
Syslog Characteristics
Do the messages have a syslog header? Is the syslog header rfc compliant? Syslog-ng with IETF (RFC5424) is another story Your parser applies only to the Payload, not to the Syslog Header Syslog Header Payload Facility + Priority Host Name Time Stamp
60
Syslog Parsing Based on a Plugin Framework (subparsers)
Syslog connectors has more 40 subagent parser When a message is received, it compares the pattern of message to all the available subagent parser till a match is found First match wins! Once the subagent parser is matched, it will store this information in syslog.properties of which device events matched which subagent parser If there is no matched pattern, events will be parsed with the generic parser and store the information in Name field and Device Vendor and Product will be “UNIX”
61
Other Syslog Flex Connector Considerations
If you plan to receive events from multiple devices You need to write multiple property files, one per device type Property files are loaded and used in alphabetical order Regexes defined in properties files need to be unique enough to match only the messages from its device type and not from other device types Names of the property files should be chosen such that more specific regexes appear in properties files with alphabetically higher names Other tips If the actual message has data in the delimited, key value or xml format, the regex need only identify the device and an appropriate extra processor can be used for further parsing
62
Syslog by Example
63
Syslog Example Nov 08 15:45: :11:07-19:05:29 ulogd[1993]: DROP: IN=eth0 OUT= MAC=00:0c:29:28:fa:4f:00:10:4b:b9:0e:84:08:00 SRC= DST= LEN=48 TOS=00 PREC=0x00 TTL=128 ID=34476 CE DF PROTO=TCP SPT=2591 DPT=1 SEQ= ACK=0 WINDOW= SYN URGP=0
64
Syslog Example Start Here
Nov 08 15:45: :11:07-19:05:29 ulogd[1993]: DROP: IN=eth0 OUT= MAC=00:0c:29:28:fa:4f:00:10:4b:b9:0e:84:08:00 SRC= DST= LEN=48 TOS=00 PREC=0x00 TTL=128 ID=34476 PROTO=TCP SPT=2591 DPT=1 SEQ= ACK=0 WINDOW=64240 SYN URGP=0 Syslog subagents are regex parsers This log seems very easy to parse (key/value pairs) but… What about optional fields What about the order of the fields Yes, we can make tokens optional in a regular expression –> inefficient
65
Parser chaining (extraprocessors)
Use a regex for: 2006:11:07-19:05:29 ulogd[1993]: DROP: Use a keyvalue parser for the rest of the message IN=eth0 OUT= MAC=00:0c:29:28:fa:4f:00:10 SRC= DST= LEN=48 TOS=00 PREC=0x00 TTL=128 ID= PROTO=TCP SPT=2596 DPT=9 SEQ= ACK=0 WINDOW=64240 SYN URGP=0
66
Keyvalue parser advantages / disadvantages
High performance Simple Configuration Arbitrary Key Order Ignores Missing Keys Easier to maintain Only tool available: notepad/vi More difficult to debug
67
ArcSight Regex Tool Start building your base Connector with ArcSight Regex tool /bin/arcsight regex Treat As Syslog Be as specific as possible Complete basic mappings Test with a large sample Then move on to extraprocessor
68
Now move on to your favorite editor
extraprocessor.count=1 extraprocessor[0].type=keyvalue extraprocessor[0].filename=syslog/astaro/astaro extraprocessor[0].field=event.message extraprocessor[0].clearfieldafterparsing=true extraprocessor[0].flexagent=true
69
astaro.sdkkeyvaluefilereader.properties
Event mappings event.sourceAddress=SRC event.deviceInboundInterface=IN event.deviceOutboundInterface=OUT event.transportProtocol=PROTO event.destinationAddress=DST event.destinationPort=DPT event.sourcePort=SPT Token definitions token.count=13 token[0].name=IN token[0].type=String token[1].name=OUT token[1].type=String token[2].name=SRC token[2].type=IPAddress token[3].name=DST token[3].type=IPAddress token[4].name=LEN token[4].type=String token[5].name=TOS token[5].type=String token[X].name=…
70
Test your Connector Set up a syslog deamon
Feed samples into the connector Watch the agent.log for ERROR and FATAL [ :26:44,441][FATAL][default.com.arcsight.agent.parsers.k][readFieldMappings] Could not load operation [__stringTESTConstant(Rule)] [ :26:44,449][FATAL][default.com.arcsight.agent.parsers.k][readFieldMappings] com.arcsight.agent.parsers.operation.OperationNotSupportedException: Operation [stringTESTConstant] not supported ! at com.arcsight.agent.parsers.operation.OperationLoader.loadOperation(OperationLoader.java:45) at com.arcsight.agent.parsers.operation.OperationLoader.getOperationIndex(OperationLoader.java:68) at com.arcsight.agent.parsers.j$e_.<init>(j$e_.java:1283) at com.arcsight.agent.parsers.j.k(j.java:579) at com.arcsight.agent.parsers.j.d(j.java:503) at com.arcsight.agent.parsers.j.u(j.java:347)
71
Categorization Categorization of FlexConnectors 71
72
What is Categorization?
Categorization allows you to look at an event in a vendor neutral context. Apply Event Categories — six criteria translate the core meaning of an event into ArcSight’s Event Schema Object - Entity being targeted Behavior - What is being done to the object Outcome - Result of the Behavior on the object (Success, Failure or Attempt) Technique - Nature of the behavior represented Device Group - Indicates if event is of one type or another (Such as Firewall Events) Significance - Indicates security risk based on various data points, information from the device, and the data model
73
Categorisation If you have time, or is a POC requirement
Beware of existing content triggering on this (eg. Config Changes, Brute Forces) ESM 4.0 User Reference Guide / Data Fields ESM 5.0 ESM_UserGuide.pdf 73
74
Categorization files are map files
Place categorization files in <connector_home>/user/agent/acp/categorizer/current/<vendor_name>/ Create a file product.csv which will reside inside of the directory Vendor and product must match what is being assigned in the connector configuration file
75
Categorization File Product.csv File should contain a header like this
event.deviceEventClassId,set.event.categoryObject,set.event.categoryBehavior,set.event.categoryTechnique,set.event.categoryDeviceGroup,set.event.categorySignificance,set.event.categoryOutcome The key field is deviceEventClassId this is how we assign the categorization to the events
76
Documentation Flexcon Dev Guide sucks but is a good starting point
Review PS / Support Training recordings Confluence!
77
The Burden With The Escape Characters
In literal Java strings the backslash is an escape character The literal string "\\" is a single backslash In regular expressions, the backslash is also an escape character The regular expression \\ matches a single backslash This regular expression as a Java string, becomes "\\\\" That's right: 4 backslashes to match a single one
78
Some Useful Undocumented Token Operations
__divide(integer, integer) __sum(integer, integer [, integer...]) __substract(... __product(... Curious for more? Unzip the arcsight-parserframework<version>.jar Look into \com\arcsight\agent\parsers\operation
79
Event merging Event Merger Some devices will send information about a single event in multiple log lines Even though in some cases it would be fine to send each line as a single event, in some other instances it is necessary to merge the information of all the events into a single one. Event Merger vs. Multi-line regex One could argue that a multi-line regex agent could be developed for cases where multiple events have to be merged into a single one In some instances the events sent by the device will not necesarilly be close together, there could be events that will be sent in between other events
80
Event merging Consider the following log lines:
[18/Jul/2005:12:30: ] conn=8 op=0 msgId=82 - BIND uid=admin [18/Jul/2005:12:30: ] conn=7 op=-1 msgId=-1 - LDAP connection from to [18/Jul/2005:12:30: ] conn=8 op=0 msgId=82 - RESULT err=0 Two of those lines refer to a "binding" operation where the user id is admin and the error is 0. This example could NOT be solved using a multiline. Both events have a connection (conn and a message id (msgId) that is identical for both events Search Confluence for „Event Merger“
81
Event Name Event names should not have variables The Good The Bad
%PIX : Denied Telnet login session from on interface inside The Good Denied Telnet The Bad Denied Telnet login session from thE UgLy %PIX : Denied Telnet login session from on interface inside Several ArcSight features (event graphs, data monitors, etc…) use the event name as a unique identifier of the event type, so when selecting a field to store the event name, make sure you pick a field that contains a fixed name and no variable elements (like ip address, port, etc…)
82
ArcSight Regex Tool Always use the version of the target connector (latest if possible) Delete tmp and bak files after using it Stop using it once you have modified properties file outside of the Tool Rememer to double-escape when editing with external editor (\s+) turns into (\\s+)
83
Regular Expressions Regular expressions should be as specific as possible abc,def,ghi The Good \\S+?,\\S+?,\\S+? or even better [^,]+,[^,]+,[^,]+ The Bad \\S+,\\S+,\\S+ thE UgLy .*,.*,.* .* is EVIL. NEVER use more than one of these in the regex, and that too should preferably be at the end. .*? is almost as evil. NEVER use more than one. a cool regex trick: if you think you'll be receiving a multiline string: try (?s) at the beginning. actually this can be scoped!
84
Regular expressions for Syslog sub-agents
Generic regular expressions may cause legitimate Unix syslog messages to not be detected abc,def,ghi The Good (Order is from most specific to most generic) abc,def,ghi (exact match) [^,]+,[^,]+,[^,]+ \\S+?,\\S+?,\\S+? .*?,.*?,.* The Bad/thE UgLy (Order is from most generic to most specific) .*?,.*?,.* (generic catch-all) abc,deg,ghi It is critical that syslog flexagents have 'tight' regular expressions. Generic regexes may cause legitimate unix syslog messages to not be detected (since the autoload subagent is higher than unix syslog in the heirarchy).
85
Syslog subagent Flex: Other parser matches first
If possible, remove other subagents from framework Set usecustomsubagentlist to true Remove parser from customsubagentlist If you still need the other parser Change parser order
86
Fields to map sourceAddress destinationAddress deviceReceiptTime
deviceEventClassId deviceVendor deviceProduct name Categorization depends on these!
87
Custom Fields Each Device Custom field has a corresponding Device Custom label, it is a good practice to make sure the labels are also set
88
Device Receipt Time, Start Time, End Time
What is this? Time when the device “detected” the event Should this be set? YES! This MUST be set in the parser Start Time: What is this? Time when the actual event detection started Should this be set? Only when device calls it out explicitly End Time: What is this? Time when the actual event detection ended or when the actual event ended. Should this be set? Only when device calls it out explicitly Start time: Most devices will NOT send this information Exception: Scanner agents (sets time to when scan started) Start Time and End Time are inferred from deviceReceiptTime.
89
Sidetables Only relevant to ESM/Express based on Oracle
Certain fields with highly repetitive content are cached in sidetables Mapping wrong information into those fields will cause sidetable overflow Wrong field mapping can kill or slow-down ESM (pre-CORRE)
90
Fixing existing parsers (SmartConnectors)
91
Parser overrides property file contains the modified information ONLY to replace the corresponding line(s) in the original parser file, in order to change the data mapping Must have the same parser file name as the original parser Placed under \current\user\agent\fcp\<folder_ name>\<Parser_file_name> The <folder_name> must be same as the original parser file’s folder name
92
Unobfuscated property file
# Unobfuscated properties file # Copyright ArcSight, Inc. All Rights Reserved. # This software is the proprietary information of ArcSight, Inc. # Use is subject to license terms. # IMPORTANT: Delete the following property when you modify this # file! ignore.this.file=x # Delete the following property if this file contains only # overrides, or leave it as is if this file is intended to be # complete replace.defaults=x #:::::::::::::::::::::::::::::::::::::::::::::: # Apache sdk regex properties file: For Apache access and error logs regex=(apache_access_log|apache_error_log):\\s*(?:\\[ID (\\S+?) (\\S+?)\\.(\\S+?)\\])?\\s*(.*) token.count=5
93
Parser overrides Only the changes need to go into the prop file when you remove: “replace.defaults=x” regex=(apache_access_log|apache_error_log):\\s*(?:\\[ID (\\S+?) (\\S+?)\\.(\\S+?)\\])?\\s*(.*)
94
How to get the original parsers
Ask a friend in support Unobfuscate yourself Start Connector on commandline: ./arcsight agents –unobpswd <code> Parsers will end up in folder: user/agent/aup/fcp Code changes monthly and has to match either Month when Connector build was produced Month of AUP version deployed on top of that Connector Need a code? Ask a colleague from the ArcSight Specialists team
95
Thank you
Similar presentations
© 2025 SlidePlayer.com Inc.
All rights reserved.