1. IBM Security Guardium: Early Detection, Identification of Root Cause, and Remediation of Full Appliance Issues — John S. Adams IBM Support - Level 2
Think 2018 / Session 8862 / March 19, 2018 / © 2018 IBM Corporation

2. Contents Part One: Detection 03 Guardium Correlation Alerts 04
Examples 05
Part Two: First Response 10
Scope, Urgency & Failover 11
CLI Diagnostics 12
Purge, Archive & clean DAM_data 13
Part Three: Getting to Root Cause 16
Technotes 17
Guardium Reports 20
Fixing the Issue 21
Planning and Prevention 23
When to Call IBM Support 26
Part Four: Questions & Answers 27
Think 2018 / Session 8862 / March 19, 2018 / © 2018 IBM Corporation

3. Detection
Think 2018 / Session 8862 / March 19, 2018 / © 2018 IBM Corporation

4. Guardium Correlation Alerts
Alerts use a query to run a scheduled report. May use any query which has a date field and a count.
If the count exceeds a threshold over a period of time, the alert fires.
Correlation alerts can notify multiple receivers:
Specific s
Groups
SYSLOG
Fully configurable. Pre-defined alerts available out of the box.
Think 2018 / Session 8862 / March 19, 2018 / © 2018 IBM Corporation

5. Guardium Correlation Alerts
Detection
Guardium Correlation Alerts
MySql DB %Used
Sniffer shuts down if the DB reaches 90% of its max size.
Disk %Used (/var)
CLI goes into ‘recovery mode’ if /var is 100% full
Central Manager
Monitor the enterprise using the CM Buffer Usage domain.
Buffer Usage
These alerts rely on the Sniffer Buffer Usage report.
Think 2018 / Session 8862 / March 19, 2018 / © 2018 IBM Corporation

6. MySql DB %Used Set the threshold at 60% DB space used.
Detection
MySql DB %Used
Set the threshold at 60% DB space used.
The MySql database is allowed about 50% of /var, so a 1TB disk should only hold 250GB of audit data.
Schedule hourly.
Think 2018 / Session 8862 / March 19, 2018 / © 2018 IBM Corporation

7. Disk %Used (/var) If /var is 100% full: MySql shuts down.
Detection
If /var is 100% full:
MySql shuts down.
GUI shuts down.
Sniffer shuts down.
CLI goes into ‘recovery mode’.
CLI may become slow or unresponsive.
Keep /var under 60% full to allow for proper data backups.
Disk %Used (/var)
Think 2018 / Session 8862 / March 19, 2018 / © 2018 IBM Corporation

8. Detection
Central Manager
CM Sniffer Buffer Usage is a custom table upload that runs on a schedule.
Schedule is disabled by default:
Custom Table Builder > Upload Data
Think 2018 / Session 8862 / March 19, 2018 / © 2018 IBM Corporation

9. Sniffer Buffer Usage
These alerts only work if the Sniffer Buffer Usage report has good data.
If the report shows all zeros, sniffer is down.
CLI:
start inspection-core
No data at all? The monitor job is down.
restart stopped services
Think 2018 / Session 8862 / March 19, 2018 / © 2018 IBM Corporation

10. First Response
Think 2018 / Session 8862 / March 19, 2018 / © 2018 IBM Corporation

11. First Response
When you get an alert …
Ensure the failover/secondary collector has taken over traffic collection.
Check related collectors; they might be filling up too!
Was a new policy pushed? Roll it back.
New STAPs or Inspection Engines? Turn them OFF.
Make quick diagnostics in CLI and start purge.
Think 2018 / Session 8862 / March 19, 2018 / © 2018 IBM Corporation

12. CLI diagnostic commands
Log into CLI. Is the unit in recovery mode?
support show db-stat used %
support show db-top all
support show db-proc run
support show large_files 100 0
Think 2018 / Session 8862 / March 19, 2018 / © 2018 IBM Corporation

13. clean DAM_data vs. Archive & Purge
First Response
clean DAM_data vs. Archive & Purge
clean DAM_data:
Single, unconnected tables
Policy violations
Messages
Exceptions
Full Details
Archive & Purge:
Normal audit data
GDM_CONSTRUCT_INSTANC E
GDM_SESSION
GDM_FIELD, OBJECT, SENTENCE, or CONSTRUCT
Think 2018 / Session 8862 / March 19, 2018 / © 2018 IBM Corporation

14. “Do not use these clean DAM_data options unless directed by IBM Support.” ** support clean DAM_data agg support clean DAM_Data constructs John S. Adams Using clean DAM_data, YouTube ** … and maybe not then either!
Think 2018 / Session 8862 / March 19, 2018 / © 2018 IBM Corporation

15. clean DAM_data is a hard delete!
Be sure you are authorized to delete this audit data or export/archive before using clean DAM_data.
clean DAM_data is a
hard delete!
Think 2018 / Session 8862 / March 19, 2018 / © 2018 IBM Corporation

16. Getting to Root Cause
Think 2018 / Session 8862 / March 19, 2018 / © 2018 IBM Corporation

17. Technotes Why is my Guardium internal database filling up?
Getting to Root Cause
Technotes
Why is my Guardium internal database filling up?
Resolving DB Full issue using
'support clean DAM_data'
Find and manually delete large files from Guardium
Think 2018 / Session 8862 / March 19, 2018 / © 2018 IBM Corporation

18. The top table usually reveals the root cause!
Basic Troubleshooting
What is your top table? How big is it?
When did it start filling up? Did it fill quickly or over several days?
Based on the top table name, what do you think caused it to fill up?
What changed recently? New STAPs or a new app online? New policy?
Think 2018 / Session 8862 / March 19, 2018 / © 2018 IBM Corporation

19. GDM_POLICY VIOLATIONS_LOG
Logs 1 record every time a policy rule fires.
New policy? Roll it back!
New traffic hitting your policy in new ways?
New STAP or IE? Roll it back!
GDM_EXCEPTION
Logs SQL exceptions captured by STAPs.
New app or big, quarterly job running?
Use reports to find the host and the exact SQL error.
Talk to the DBA!
MESSAGE and/or
MESSAGE_TEXT
Each record is an sent by Guardium for alerts, audits, etc.
Usually a policy issue.
Alert per session rather than per match whenever possible.
GDM
CONSTRUCT_TEXT
Logs 1 record for every SQL matching a LOG FULL DETAILS rule.
Use these rules sparingly, with tight matching conditions.
Think 2018 / Session 8862 / March 19, 2018 / © 2018 IBM Corporation

20. Sort by count & Focus on big numbers!
Get connection profile for suspect sessions:
server/client IP
Db User
Source Program
Service name
Sort by count & Focus on big numbers!
Guardium Reports
Running the right reports can quickly lead you to the root cause and permanent solution.
Pre-defined reports:
Session Details by Server
Exceptions by Server
Policy Violations Details
Walk your policy:
Once you have the connection profile and understand which rules fired, check your rule conditions. Rules may be firing for traffic you want to ignore.
Think 2018 / Session 8862 / March 19, 2018 / © 2018 IBM Corporation

21. Getting to Root Cause
Fixing the Issue
Most full collector issues can be fixed or prevented with a policy change.
Target a specific DB_USER, server and client IP as appropriate.
Use SKIP LOGGING for traffic within a session, or IGNORE STAP SESSION if the connection profile is trusted.
Think 2018 / Session 8862 / March 19, 2018 / © 2018 IBM Corporation

22. Getting to Root Cause
Fixing the Issue
Sometimes the best solution is to move STAPs onto another collector.
If a single STAP or application is flooding the collector even after you SKIP LOGGING on as much traffic as possible, load balancing can divide the load among two or more collectors.
Balancing is round-robin by session.
V10 Enterprise Load-Balancing can help.
Think 2018 / Session 8862 / March 19, 2018 / © 2018 IBM Corporation

23. “An ounce of prevention is worth a pound of cure
“An ounce of prevention is worth a pound of cure.” Benjamin Franklin Pennsylvania Gazette, Feb 4th, 1735
Think 2018 / Session 8862 / March 19, 2018 / © 2018 IBM Corporation

24. Teamwork is critical! Changes on DB hosts can affect Guardium.
Do your change control systems notify the Guardium admin?
Know your DBAs, SysAdmins and network admins.
Know how to contact them if an issue arises.
Think 2018 / Session 8862 / March 19, 2018 / © 2018 IBM Corporation

25. Enable Correlation Alerts
Always test new policies on non-production collectors. Clone your policy. Increment and date the policy name: IBM_prod_ _v5
Periodically review:
Disaster Recovery
Failover schemes
Unit Utilization
Guardium policy
System backups
Archives
Be sure the Guardium admin is listed as a stakeholder for any DB host changes.
New applications
New instances or nodes
OS Upgrades, especially for Linux
Enable/disable encryption
Enable Correlation Alerts
Think 2018 / Session 8862 / March 19, 2018 / © 2018 IBM Corporation

26. When to Call IBM Support
Getting to Root Cause
When to Call IBM Support
Open a case if:
Can’t log into CLI
CLI in recovery mode > 4 hours
(large aggregators may take longer)
/var is 100% full and CLI won’t let you delete the large files.
No data in Buffer Usage report, restart stopped services didn’t help.
You need help interpreting reports or policy. (sev 2 please)
Please Avoid:
Sev 1, Collector full, please send webex!
(Troubleshooting is driven by the Guardium admin. We don’t need root for most cases.)
Requests to drop tables
(Years ago this was common. Now we have better options and technotes. We require written authorization to drop tables or delete audit data.)
Sev 1, GUI down, no other details
(Log into CLI first, try restart gui, include results when you open the case. Recovery mode?)
Think 2018 / Session 8862 / March 19, 2018 / © 2018 IBM Corporation

27. Questions & Answers
Think 2018 / Session 8862 / March 19, 2018 / © 2018 IBM Corporation

28. Thank you John S. Adams IBM Support – IBM Security Guardium —
Think 2018 / Session 8862 / March 19, 2018 / © 2018 IBM Corporation

29. Think 2018 / Session 8862 / March 19, 2018 / © 2018 IBM Corporation