Flex Connector for importing large Active List Entries Raju Gottumukkala Enterprise Expert August 2010

Flex Connector to import large Active Lists How do you import very large values into Active Lists? You may create a custom archive file and use “arcsight archive” Manager can not handle large archive files Which means you have to split the input into multiple archive files and manage the import process which is a pain You may use the Console and right click on the AL and import Again large files are a pain and the process is manual You can send events to ESM via a flex connector and write a rule that populates the values to AL Excessive firing of rule and overloaded process Creates many internal events also WELCOME to a brand new approach www.arcsight.com © 2009 ArcSight Confidential 2

Example Import of Active List (AL) AL: Black List from SANS Has 1 column with a type of IPAddress

Example Data file that will be imported into AL 222.073.204.093 222.073.044.241 222.073.204.018 198.189.053.081 211.157.113.012 190.210.025.161 217.018.151.006 211.119.115.003 212.065.146.025 207.250.047.137

Flex Connector to import large Active Lists Create any regular flex connector to read the data corresponding to the Active List File, Database, Syslog etc Define Tokens only and do not map to fields Map tokens to additional data Additional Data field name can be anything In this example I am reading only IP Addresses from the file with the token name of IP and mapping it to IP_ADDRESS in additional data Set the Creation Date to Now – converted to milliseconds additionaldata.CREATE_DATE=__concatenate(__longToString(__currentTimestampInSeconds()),"000") www.arcsight.com © 2009 ArcSight Confidential 5

Flex Connector to import large Active Lists Define the properties to invoke Model Import feature Define the property to invoke the custom Velocity Macro file that converts the data into the ArcSight Archive event.deviceCustomString2=__stringConstant(ips.vm) event.deviceVendor=__stringConstant(ArcSight) event.deviceProduct=__stringConstant(FlexArchiveImport) event.deviceCustomString1Label=__stringConstant(model.sender) event.deviceCustomString1=__stringConstant(sans) event.deviceCustomString2Label=__stringConstant(model.template) www.arcsight.com © 2009 ArcSight Confidential 6

Example Flex Properties file comments.start.with=# delimiter=, token.count=1 token[0].name=IP token[0].type=String additionaldata.enabled=true additionaldata.IP_ADDRESS=IP additionaldata.CREATE_DATE=__concatenate(__longToString(__currentTimestampInSeconds()),"000") event.deviceVendor=__stringConstant(ArcSight) event.deviceProduct=__stringConstant(FlexArchiveImport) event.deviceCustomString1Label=__stringConstant(model.sender) event.deviceCustomString1=__stringConstant(sans) event.deviceCustomString2Label=__stringConstant(model.template) event.deviceCustomString2=__stringConstant(ips.vm)

Set Model Import User for the Connector

Create the Velocity Macro Template Create the VM file defined in the flex properties file and place it in the user/agent/fcp directory: In our example it’s called ips.vm <?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE archive SYSTEM "../../schema/xml/archive/arcsight-archive.dtd"> <archive buildVersion="4.5.1.14375.2" buildTime="10-12-2009_18:36:22" user="admin" createTime="11-30-2009_14:41:19.312"> <ActiveList name="Black List from SANS" action="insert" > <insertListEntries> <list> #foreach($ip in $IP_ADDRESS) <map> <count>1</count> <creationTime>$CREATE_DATE.get($velocityCount)</creationTime> <lastModifiedTime>1254785518000</lastModifiedTime> <values> <string>$ip</string> </list> </values> </map> #end </insertListEntries> <childOf> <ref type="Group" uri="/All Active Lists/Personal/admin's Active Lists/"/> </childOf> <eventBound>false</eventBound> <hashBased>false</hashBased> <ttl>0</ttl> </ActiveList> </archive> www.arcsight.com © 2009 ArcSight Confidential 9

Create the Velocity Macro Template Depending upon the Connector build (for example: 5594), you may have to use this without the XML header and closing tag <ActiveList name="Black List from SANS" action="insert" > <insertListEntries> <list> #foreach($ip in $IP_ADDRESS) <map> <count>1</count> <creationTime>$CREATE_DATE.get($velocityCount)</creationTime> <lastModifiedTime>1254785518000</lastModifiedTime> <values> <string>$ip</string> </list> </values> </map> #end </insertListEntries> <childOf> <ref type="Group" uri="/All Active Lists/Personal/admin's Active Lists/"/> </childOf> <eventBound>false</eventBound> <hashBased>false</hashBased> <ttl>0</ttl> </ActiveList> www.arcsight.com © 2009 ArcSight Confidential 10

Explanation of Velocity Macro Template Edit the XML file and change the Active List name and Group URI Change ttl if necessary or you may remove the property also Notice the foreach loop $ip is a local variable and $IP_ADDRESS is the Additional Data field specified in properties file If there are multiple columns in AL then you need to place them within the <values> and <list> loop where the $ip is specified Assuming there are 2 fields in the AL with UserName and UserMachine then: additionaldata.USER_MACHINE=token2 additionaldata.USER_NAME=token1 foreach loop in the vm file may look like this foreach($user in $USER_NAME) Then the values specification will look like this: <string>$USER_MACHINE.get($velocityCount)</string> <string>$user</string> Here $user is defined in the foreach loop hence does not require get VelocityCount specification ipAddress is also a string type for the archive template www.arcsight.com © 2009 ArcSight Confidential 11

© 2009 ArcSight Confidential Other things Make sure to use Agent Software build 5427 or higher Check $managerDir\archive\webservice directory for xml files that are sent from the Agent to Manager You may add “if” statements to the vm file www.arcsight.com © 2009 ArcSight Confidential 12

© 2009 ArcSight Confidential Gotchas! Edit the agent.properties and add the following agent.component[34].maxeventsbeforebuild=20000 agent.component[34].buildmodeldelay=90000 This component (34) could be different for ModelBuilder depending upon the connector build Search config/agent/agent.defaults.properties file for ModelBuilder Play with maxeventsbeforebuild parameter I ran into connector default memory issues with 20000 but 10000 entries was ok Try not to go beyond the 20000 number, otherwise the archive file could become huge and cause problems www.arcsight.com © 2009 ArcSight Confidential 13

ArcSight, Inc. Corporate Headquarters: 1 888 415 ARST EMEA Headquarters: +44 (0)844 745 2068 Asia Pac Headquarters: +65 6248 4795 www.arcsight.com