Presentation is loading. Please wait.

Presentation is loading. Please wait.

“IIS Data Mining with Log Parser 2.X” Alexis Eller Program Manager Internet Information Services Microsoft.

Similar presentations


Presentation on theme: "“IIS Data Mining with Log Parser 2.X” Alexis Eller Program Manager Internet Information Services Microsoft."— Presentation transcript:

1 “IIS Data Mining with Log Parser 2.X” Alexis Eller Program Manager Internet Information Services Microsoft

2 What we will cover  Log Parser Basics  Input Formats, Output Formats  Functions  Output Templates  How to get started…  Building on Log Parser  Scripting with LogParser.dll  C# Interop  Advanced Features  New ‘CHART’ output format  CheckPoint

3 Helps to have…  Experience with:  Command Line tools  SQL Query Language  Familiarity with:  IIS Log Files, Event Log  … also nice to have experience with:  VBScript, JScript, C#  SQL Server

4 Log Parser Basics Log Parser is…  Freely downloadable, stand-alone tool  Version 2.2: search for “Log Parser”  Version 2.1: search for “IIS 6.0 Resource Kit Tools”  Developed by Gabriele Giuseppini, former IIS  Not officially supported, try  Provided in two forms:  Command line exe  Scriptable COM object  [no GUI available at this time]

5 Log Parser Basics Key Concepts  Log Parser requires…  Input Format  [+options]  Query  Output Format  [+options]  Query Language:  supports most of the true SQL language syntax  extends the language with additional functions

6 First Walk-Through Log Parser Cmd-Line Help Simple Simple Query Using Input/Output Parameters First Walk-Through Log Parser Cmd-Line Help Simple Simple Query Using Input/Output Parameters

7 Log Parser Basics Functions  To get a list of functions  logparser –h FUNCTIONS  Examples:  Conversion Functions  TO_TIMESTAMP  TO_LOCALTIME  TO_INT  String Handling  STRLEN  SUBSTR  STRCAT  Other  REVERSEDNS  QUANTIZE  EXTRACT_EXTENSION  CASE

8 Log Parser Basics Function Example  Convert log file timestamp from UTC time to local time: SELECT TO_DATE( TO_LOCALTIME( TO_TIMESTAMP(date, time))) AS date, TO_TIME( TO_LOCALTIME( TO_TIMESTAMP(date, time))) AS time, c-ip, cs-username, s-ip, s-port, cs-method, cs-uri-stem, cs-uri-query, sc-status, sc-win32-status, sc-bytes, cs-bytes, time-taken, cs(User-Agent) FROM C:\WINDOWS\system32\Logfiles\W3SVC1\ex log

9 Functions Top 10 URL's Convert Log File Time from UTC Time to Local Time Examine Time Taken Per VDir

10 Log Parser Basics Output Templates  logparser -h -o:TPL  Consists of:  Header  Body  Footer  Best for report-style results Hits/Hour for Ledbury Home Page Hits/Hour for Ledbury Home Page Hour # Hits :00: :00: :00:00 2 HEADER BODY FOOTER TEMPLATE OUTPUT

11 Templates Broken Links Report Event Log Messages Templates Broken Links Report Event Log Messages

12 Log Parser Basics Importing Data into SQL Server  Note: Field names in query output are not important -- the field position determines what column the data is inserted into in the SQL table SELECT 1, TO_TIMESTAMP(date, time) AS LogTimeStamp, s-ip, cs-method, cs-uri-stem, cs-uri-query, s-port, cs-username, c-ip, cs(User-Agent), sc-status, sc-substatus, sc-win32-status INTO W3SVC2 FROM ex log allows for an auto-generated identity column in SQL database table

13 Log Parser Basics How do I get started?  Find the column names in your data source:  LogParser –h –i:  LogParser –h –i:  or… “ SELECT * FROM ”  For TSV or CSV files:  LogParser –h –i:CSV/TSV  LogParser –h –i:CSV/TSV  Understand data provided in the columns  Apply any functions necessary to convert that data into a useful form  –queryinfo Option

14 Log Parser Basics -queryinfo C:\DemoScripts>logparser file:SQLDB_Insert.sql -queryinfo WARNING: Output format not specified - using NAT output format. Query: SELECT 1, TO_TIMESTAMP(date,time) AS LogTimeStamp, [s-ip], [cs-method], [cs-uri-stem], [cs-uri-query], [s-port], [cs-username], [c-ip], [cs(User-Agent)], [sc-status], [sc-substatus], [sc-win32-status], [sc-bytes], [cs-bytes], [time-taken] INTO W3SVC2 FROM ex log Formats selected: Input Format : IISW3C (IIS W3C Extended Log Format) Output Format: NAT (Native Format) Query fields: 1 (I) LogTimeStamp (T) s-ip (S) cs-method (S) cs-uri-stem (S) cs-uri-query (S) s-port (I) cs-username (S) c-ip (S) cs(User-Agent) (S) sc-status (I) sc-substatus (I) sc-win32-status (I) sc-bytes (I) cs-bytes (I) time-taken (I)

15 Building on Log Parser Log Parser COM Architecture  MSUtil.LogQuery  Input and Output Format objects  MSUtil.LogQuery.IISW3CInputFormat  MSUtil.LogQuery.EventLogInputFormat  MSUtil.LogQuery.SQLOutputFormat  MSUtil.LogQuery.CSVOutputFormat  Two ways to run query:  MSUtil.LogQuery.Execute  Returns a LogRecordSet object that allows the script to iterate through the query results  MSUtil.LogQuery.ExecuteBatch  Runs a query with a pre-specified output target (e.g. SQL or CSV)

16 Building on Log Parser Input/Output Format Parameters  Log Parser 2.2  Parameters have same name for both command line tool and COM interface  Log Parser 2.1 and 2.0  Parameters do not always have same name for cmd-line version and COM version, for example:  logparser file:query.sql –o:NAT –rtp:-1  NativeOutputFormat.rowsToPrint  spx?id= spx?id= spx?id=10039

17 Building on Log Parser VBScript  Create LogQuery object and input/output format objects SET objLogQuery = WScript.CreateObject("MSUtil.LogQuery") ' Get the IIS Input and W3C output formats SET w3cInputFormat = WScript.CreateObject("MSUtil.LogQuery.IISW3CInputFormat") SET w3cOutputFormat = WScript.CreateObject("MSUtil.LogQuery.W3COutputFormat") w3cOutputFormat.filemode = 0 ' Create a SQL query query = "SELECT TOP 20 cs-uri-stem, COUNT(*) as Total " & _ "INTO results.log FROM C:\WINDOWS\system32\Logfiles\W3SVC1\ex*.log " & _ "GROUP BY cs-uri-stem ORDER BY Total DESC " objLogQuery.ExecuteBatch query, w3cInputFormat, w3cOutputFormat For an example that uses Execute and LogRecordSet, see \Samples\Scripts\ErrorCodes.js in Log Parser installation directory (also available on TechNet: ex log

18 Script Samples List Error Codes and Reasons Hacker Scan

19 Building on Log Parser C# Interop  Uses System.Reflection and System.Activator Type comLogQueryType = Type.GetTypeFromProgID("MSUtil.LogQuery", true); object comLogQueryObject = Activator.CreateInstance(comLogQueryType); // Get the IIS Input and W3C output formats Type inputFormatType = Type.GetTypeFromProgID("MSUtil.LogQuery.IISW3CInputFormat", true); object inputFormatObject = Activator.CreateInstance(inputFormatType); Type outputFormatType = Type.GetTypeFromProgID("MSUtil.LogQuery.W3COutputFormat", true); object outputFormatObject = Activator.CreateInstance(outputFormatType); // Create a SQL query string query = "SELECT TOP 20 cs-uri-stem, COUNT(*) as Total "; query += "INTO results.log FROM C:\\WINDOWS\\system32\\Logfiles\\W3SVC1\\ex*.log "; query += "GROUP BY cs-uri-stem ORDER BY Total DESC "; // Invoke the ExecuteBatch method object[] inputArgs = { query, inputFormatObject, outputFormatObject }; comLogQueryType.InvokeMember("ExecuteBatch", BindingFlags.InvokeMethod, null, comLogQueryObject, inputArgs);

20 Advanced Features Chart Output Format  Uses Microsoft Office Web Components ChartSpace Object Model  See this link for object model: ary/default.asp?url=/library/en- us/owcvba10/html/octocMSOW CObjectModels.asp ary/default.asp?url=/library/en- us/owcvba10/html/octocMSOW CObjectModels.asp ary/default.asp?url=/library/en- us/owcvba10/html/octocMSOW CObjectModels.asp  You will need:  Licensed version of Microsoft Office Web Components, available in Microsoft Office XP © or better  Use Excel to view different CHART types:  Insert Menu, Chart option – shows the different chart types

21 Charts Status Code Breakdown – Pie Chart Hits Per Hour – Radial Chart

22 Advanced Features CheckPoint – incremental parsing  Parse only what has not been parsed before:  ex*.log… from all log files in a directory  … from all log files for site 1  System… from the System Event Log  Requires a CheckPoint file to store state: logparser "SELECT TimeGenerated, EventTypeName, Strings FROM System WHERE SourceName = 'W3SVC'" -icheckpoint event.lpc

23 CheckPoint Event Log Update

24 Session Summary  Flexible and powerful  As with all data mining, the application starts with a question…  Building Blocks for Auditing and Monitoring  Log Parser can parse almost any data source on your server… with the additional input and output formats in Log Parser 2.2, there’s really nothing more you need to build custom auditing and monitoring for your system  Any Text, Any Time  LogParser can read most log files, text documents, and STDOUT/text streams, and generate results in a HUGE number of formats to make them useful and intelligible Log Parser has as many applications as you can imagine…

25 For More Information  has:  an active forum for questions  KB articles specifically for Log Parser  Articles about Log Parser:  SecurityFocus: Forensic Log Parsing with Microsoft's LogParser  TechRepublic: Consolidating Events with Free Log Parser 2.0 Tool  Windows &.NET Magazine: Using SQL-Like Queries to Extract File-Format Information  ComputerWorld mutopicheads mutopicheads mutopicheads  Presentations:  BlackHat Windows 2004: Forensic Secrets for Windows Servers-  Blog Mentions:  Amazing Log Parser -  Using Log Parser to Read Log Files -  Using LogParser from C# -  michaelw.net -

26


Download ppt "“IIS Data Mining with Log Parser 2.X” Alexis Eller Program Manager Internet Information Services Microsoft."

Similar presentations


Ads by Google