Presentation is loading. Please wait.

Presentation is loading. Please wait.

“IIS Data Mining with Log Parser 2.X”

Similar presentations


Presentation on theme: "“IIS Data Mining with Log Parser 2.X”"— Presentation transcript:

1 “IIS Data Mining with Log Parser 2.X”
Alexis Eller Program Manager Internet Information Services Microsoft

2 What we will cover Log Parser Basics Building on Log Parser
Input Formats, Output Formats Functions Output Templates How to get started… Building on Log Parser Scripting with LogParser.dll C# Interop Advanced Features New ‘CHART’ output format CheckPoint

3 Helps to have… Experience with: Familiarity with:
Command Line tools SQL Query Language Familiarity with: IIS Log Files, Event Log … also nice to have experience with: VBScript, JScript, C# SQL Server

4 Log Parser Basics Log Parser is…
Freely downloadable, stand-alone tool Version 2.2: search for “Log Parser” Version 2.1: search for “IIS 6.0 Resource Kit Tools” Developed by Gabriele Giuseppini, former IIS Not officially supported, try Provided in two forms: Command line exe Scriptable COM object [no GUI available at this time]

5 Log Parser Basics Key Concepts
Log Parser requires… Input Format [+options] Query Output Format Query Language: supports most of the true SQL language syntax extends the language with additional functions

6 First Walk-Through Log Parser Cmd-Line Help Simple Simple Query Using Input/Output Parameters

7 Log Parser Basics Functions
To get a list of functions logparser –h FUNCTIONS Examples: Conversion Functions TO_TIMESTAMP TO_LOCALTIME TO_INT String Handling STRLEN SUBSTR STRCAT Other REVERSEDNS QUANTIZE EXTRACT_EXTENSION CASE

8 Log Parser Basics Function Example
Convert log file timestamp from UTC time to local time: SELECT TO_DATE( TO_LOCALTIME( TO_TIMESTAMP(date, time))) AS date, TO_TIME( TO_LOCALTIME( TO_TIMESTAMP(date, time))) AS time, c-ip, cs-username, s-ip, s-port, cs-method, cs-uri-stem, cs-uri-query, sc-status, sc-win32-status, sc-bytes, cs-bytes, time-taken, cs(User-Agent) FROM C:\WINDOWS\system32\Logfiles\W3SVC1\ex log

9 Functions Top 10 URL's Convert Log File Time from UTC Time to Local Time Examine Time Taken Per VDir

10 Log Parser Basics Output Templates
TEMPLATE OUTPUT <HTML> <HEAD> <TITLE>Hits/Hour for Ledbury Home Page</TITLE> </HEAD> <BODY BGCOLOR="#EFEFFF"> <TABLE BORDER="1" CELLPADDING="2“ CELLSPACING="2"> <TR> <TH COLSPAN="2" ALIGN="CENTER"> Hits/Hour for Ledbury Home Page </TH> </TR> <TH ALIGN="LEFT">Hour</TH> <TH ALIGN="LEFT"># Hits</TH> <TR><TD> :00:00</TD><TD>1</TD></TR> <TR><TD> :00:00</TD><TD>2</TD></TR> <TR><TD> :00:00</TD><TD>2</TD></TR> </TABLE> </BODY> </HTML> logparser -h -o:TPL Consists of: Header Body Footer Best for report-style results HEADER BODY FOOTER

11 Templates Broken Links Report Event Log Messages

12 Log Parser Basics Importing Data into SQL Server
allows for an auto-generated identity column in SQL SELECT 1, TO_TIMESTAMP(date, time) AS LogTimeStamp, s-ip, cs-method, cs-uri-stem, cs-uri-query, s-port, cs-username, c-ip, cs(User-Agent), sc-status, sc-substatus, sc-win32-status INTO W3SVC2 FROM ex log database table Note: Field names in query output are not important -- the field position determines what column the data is inserted into in the SQL table

13 Log Parser Basics How do I get started?
Find the column names in your data source: LogParser –h –i:<IISW3C|IIS|EVT> or… “SELECT * FROM <datasource>” For TSV or CSV files: LogParser –h –i:CSV/TSV <filename> Understand data provided in the columns Apply any functions necessary to convert that data into a useful form –queryinfo Option

14 Log Parser Basics -queryinfo
C:\DemoScripts>logparser file:SQLDB_Insert.sql -queryinfo WARNING: Output format not specified - using NAT output format. Query: SELECT 1, TO_TIMESTAMP(date,time) AS LogTimeStamp, [s-ip], [cs-method], [cs-uri-stem], [cs-uri-query], [s-port], [cs-username], [c-ip], [cs(User-Agent)], [sc-status], [sc-substatus], [sc-win32-status], [sc-bytes], [cs-bytes], [time-taken] INTO W3SVC2 FROM ex log Formats selected: Input Format : IISW3C (IIS W3C Extended Log Format) Output Format: NAT (Native Format) Query fields: 1 (I) LogTimeStamp (T) s-ip (S) cs-method (S) cs-uri-stem (S) cs-uri-query (S) s-port (I) cs-username (S) c-ip (S) cs(User-Agent) (S) sc-status (I) sc-substatus (I) sc-win32-status (I) sc-bytes (I) cs-bytes (I) time-taken (I)

15 Building on Log Parser Log Parser COM Architecture
MSUtil.LogQuery Input and Output Format objects MSUtil.LogQuery.IISW3CInputFormat MSUtil.LogQuery.EventLogInputFormat MSUtil.LogQuery.SQLOutputFormat MSUtil.LogQuery.CSVOutputFormat Two ways to run query: MSUtil.LogQuery.Execute Returns a LogRecordSet object that allows the script to iterate through the query results MSUtil.LogQuery.ExecuteBatch Runs a query with a pre-specified output target (e.g. SQL or CSV)

16 Building on Log Parser Input/Output Format Parameters
Parameters have same name for both command line tool and COM interface Log Parser 2.1 and 2.0 Parameters do not always have same name for cmd-line version and COM version, for example: logparser file:query.sql –o:NAT –rtp:-1 NativeOutputFormat.rowsToPrint

17 Building on Log Parser VBScript
Create LogQuery object and input/output format objects SET objLogQuery = WScript.CreateObject("MSUtil.LogQuery") ' Get the IIS Input and W3C output formats SET w3cInputFormat = WScript.CreateObject("MSUtil.LogQuery.IISW3CInputFormat") SET w3cOutputFormat = WScript.CreateObject("MSUtil.LogQuery.W3COutputFormat") w3cOutputFormat.filemode = 0 ' Create a SQL query query = "SELECT TOP 20 cs-uri-stem, COUNT(*) as Total " & _ "INTO results.log FROM C:\WINDOWS\system32\Logfiles\W3SVC1\ex*.log " & _ "GROUP BY cs-uri-stem ORDER BY Total DESC " objLogQuery.ExecuteBatch query, w3cInputFormat, w3cOutputFormat ex log For an example that uses Execute and LogRecordSet, see \Samples\Scripts\ErrorCodes.js in Log Parser installation directory (also available on TechNet:

18 Script Samples List Error Codes and Reasons Hacker Scan

19 Building on Log Parser C# Interop
Uses System.Reflection and System.Activator Type comLogQueryType = Type.GetTypeFromProgID("MSUtil.LogQuery", true); object comLogQueryObject = Activator.CreateInstance(comLogQueryType); // Get the IIS Input and W3C output formats Type inputFormatType = Type.GetTypeFromProgID("MSUtil.LogQuery.IISW3CInputFormat", true); object inputFormatObject = Activator.CreateInstance(inputFormatType); Type outputFormatType = Type.GetTypeFromProgID("MSUtil.LogQuery.W3COutputFormat", true); object outputFormatObject = Activator.CreateInstance(outputFormatType); // Create a SQL query string query = "SELECT TOP 20 cs-uri-stem, COUNT(*) as Total "; query += "INTO results.log FROM C:\\WINDOWS\\system32\\Logfiles\\W3SVC1\\ex*.log "; query += "GROUP BY cs-uri-stem ORDER BY Total DESC "; // Invoke the ExecuteBatch method object[] inputArgs = { query, inputFormatObject, outputFormatObject }; comLogQueryType.InvokeMember("ExecuteBatch", BindingFlags.InvokeMethod, null, comLogQueryObject, inputArgs);

20 Advanced Features Chart Output Format
Uses Microsoft Office Web Components ChartSpace Object Model See this link for object model: You will need: Licensed version of Microsoft Office Web Components, available in Microsoft Office XP© or better Use Excel to view different CHART types: Insert Menu, Chart option –shows the different chart types

21 Charts Status Code Breakdown – Pie Chart Hits Per Hour – Radial Chart

22 Advanced Features CheckPoint – incremental parsing
Parse only what has not been parsed before: ex*.log … from all log files in a directory <1> … from all log files for site 1 System … from the System Event Log Requires a CheckPoint file to store state: logparser "SELECT TimeGenerated, EventTypeName, Strings FROM System WHERE SourceName = 'W3SVC'" -icheckpoint event.lpc

23 CheckPoint Event Log Update E-mail

24 Log Parser has as many applications
Session Summary Log Parser has as many applications as you can imagine… Flexible and powerful As with all data mining, the application starts with a question… Building Blocks for Auditing and Monitoring Log Parser can parse almost any data source on your server… with the additional input and output formats in Log Parser 2.2, there’s really nothing more you need to build custom auditing and monitoring for your system Any Text, Any Time LogParser can read most log files, text documents, and STDOUT/text streams, and generate results in a HUGE number of formats to make them useful and intelligible

25 For More Information www.logparser.com has: Articles about Log Parser:
an active forum for questions KB articles specifically for Log Parser Articles about Log Parser: SecurityFocus: Forensic Log Parsing with Microsoft's LogParser TechRepublic: Consolidating Events with Free Log Parser 2.0 Tool Windows & .NET Magazine: Using SQL-Like Queries to Extract File-Format Information ComputerWorld Presentations: BlackHat Windows 2004: Forensic Secrets for Windows Servers- Blog Mentions: Amazing Log Parser - Using Log Parser to Read Log Files - Using LogParser from C# - michaelw.net -

26


Download ppt "“IIS Data Mining with Log Parser 2.X”"

Similar presentations


Ads by Google