1. Flex Connector for importing large Active List Entries
Raju Gottumukkala
Enterprise Expert
August 2010

2. Flex Connector to import large Active Lists
How do you import very large values into Active Lists?
You may create a custom archive file and use “arcsight archive”
Manager can not handle large archive files
Which means you have to split the input into multiple archive files and manage the import process which is a pain
You may use the Console and right click on the AL and import
Again large files are a pain and the process is manual
You can send events to ESM via a flex connector and write a rule that populates the values to AL
Excessive firing of rule and overloaded process
Creates many internal events also
WELCOME to a brand new approach
© 2009 ArcSight Confidential 2

3. Example Import of Active List (AL)
AL: Black List from SANS
Has 1 column with a type of IPAddress

4. Example Data file that will be imported into AL

5. Flex Connector to import large Active Lists
Create any regular flex connector to read the data corresponding to the Active List
File, Database, Syslog etc
Define Tokens only and do not map to fields
Map tokens to additional data
Additional Data field name can be anything
In this example I am reading only IP Addresses from the file with the token name of IP and mapping it to IP_ADDRESS in additional data
Set the Creation Date to Now – converted to milliseconds
additionaldata.CREATE_DATE=__concatenate(__longToString(__currentTimestampInSeconds()),"000")
© 2009 ArcSight Confidential 5

6. Flex Connector to import large Active Lists
Define the properties to invoke Model Import feature
Define the property to invoke the custom Velocity Macro file that converts the data into the ArcSight Archive
event.deviceCustomString2=__stringConstant(ips.vm)
event.deviceVendor=__stringConstant(ArcSight)
event.deviceProduct=__stringConstant(FlexArchiveImport)
event.deviceCustomString1Label=__stringConstant(model.sender)
event.deviceCustomString1=__stringConstant(sans)
event.deviceCustomString2Label=__stringConstant(model.template)
© 2009 ArcSight Confidential 6

7. Example Flex Properties file
comments.start.with=#
delimiter=,
token.count=1
token[0].name=IP
token[0].type=String
additionaldata.enabled=true
additionaldata.IP_ADDRESS=IP
additionaldata.CREATE_DATE=__concatenate(__longToString(__currentTimestampInSeconds()),"000")
event.deviceVendor=__stringConstant(ArcSight)
event.deviceProduct=__stringConstant(FlexArchiveImport)
event.deviceCustomString1Label=__stringConstant(model.sender)
event.deviceCustomString1=__stringConstant(sans)
event.deviceCustomString2Label=__stringConstant(model.template)
event.deviceCustomString2=__stringConstant(ips.vm)

8. Set Model Import User for the Connector

9. Create the Velocity Macro Template
Create the VM file defined in the flex properties file and place it in the user/agent/fcp directory: In our example it’s called ips.vm
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE archive SYSTEM "../../schema/xml/archive/arcsight-archive.dtd">
<archive buildVersion=" " buildTime=" _18:36:22" user="admin" createTime=" _14:41:19.312">
<ActiveList name="Black List from SANS" action="insert" >
<insertListEntries>
<list>
#foreach($ip in $IP_ADDRESS)
<map>
<count>1</count>
<creationTime>$CREATE_DATE.get($velocityCount)</creationTime>
<lastModifiedTime> </lastModifiedTime>
<values>
<string>$ip</string>
</list>
</values>
</map>
#end
</insertListEntries>
<childOf>
<ref type="Group" uri="/All Active Lists/Personal/admin's Active Lists/"/>
</childOf>
<eventBound>false</eventBound>
<hashBased>false</hashBased>
<ttl>0</ttl>
</ActiveList>
</archive>
© 2009 ArcSight Confidential 9

10. Create the Velocity Macro Template
Depending upon the Connector build (for example: 5594), you may have to use this without the XML header and closing tag
<ActiveList name="Black List from SANS" action="insert" >
<insertListEntries>
<list>
#foreach($ip in $IP_ADDRESS)
<map>
<count>1</count>
<creationTime>$CREATE_DATE.get($velocityCount)</creationTime>
<lastModifiedTime> </lastModifiedTime>
<values>
<string>$ip</string>
</list>
</values>
</map>
#end
</insertListEntries>
<childOf>
<ref type="Group" uri="/All Active Lists/Personal/admin's Active Lists/"/>
</childOf>
<eventBound>false</eventBound>
<hashBased>false</hashBased>
<ttl>0</ttl>
</ActiveList>
© 2009 ArcSight Confidential 10

11. Explanation of Velocity Macro Template
Edit the XML file and change the Active List name and Group URI
Change ttl if necessary or you may remove the property also
Notice the foreach loop
$ip is a local variable and $IP_ADDRESS is the Additional Data field specified in properties file
If there are multiple columns in AL then you need to place them within the <values> and <list> loop where the $ip is specified
Assuming there are 2 fields in the AL with UserName and UserMachine then:
additionaldata.USER_MACHINE=token2
additionaldata.USER_NAME=token1
foreach loop in the vm file may look like this
foreach($user in $USER_NAME)
Then the values specification will look like this:
<string>$USER_MACHINE.get($velocityCount)</string>
<string>$user</string>
Here $user is defined in the foreach loop hence does not require get VelocityCount specification
ipAddress is also a string type for the archive template
© 2009 ArcSight Confidential 11

12. © 2009 ArcSight Confidential
Other things
Make sure to use Agent Software build 5427 or higher
Check $managerDir\archive\webservice directory for xml files that are sent from the Agent to Manager
You may add “if” statements to the vm file
© 2009 ArcSight Confidential 12

13. © 2009 ArcSight Confidential
Gotchas!
Edit the agent.properties and add the following
agent.component[34].maxeventsbeforebuild=20000
agent.component[34].buildmodeldelay=90000
This component (34) could be different for ModelBuilder depending upon the connector build
Search config/agent/agent.defaults.properties file for ModelBuilder
Play with maxeventsbeforebuild parameter
I ran into connector default memory issues with but entries was ok
Try not to go beyond the number, otherwise the archive file could become huge and cause problems
© 2009 ArcSight Confidential 13

14. ArcSight, Inc.
Corporate Headquarters: ARST
EMEA Headquarters: +44 (0)
Asia Pac Headquarters: