DB-8: Jump Starting Your OpenEdge® Auditing Solution

DB-8: Jump Starting Your OpenEdge® Auditing Solution
Stephen Ferguson DB-8: Jump Starting Your OpenEdge® Auditing Solution Auditing is one of the essential factors in providing a secure application and in meeting mandatory regulatory compliance. During this session we’ll cover a broad range of topics that are tailored toward getting your auditing solution started sooner and with fewer problems. At the end of this session you’ll understand the best way to configure auditing for your application’s storage, performance and report ability needs. This slide deck has many useful notes for most slides; also, at the end of the slide deck are additional slides with more technical detail which could never be included in a 1 hour time slot. Be sure to check the notes at the slides at the end of the presentation for additional technical information to assist you in your auditing implementation. Stephen Ferguson Progress Software Progress Exchange June, Phoenix, AZ, USA

DB-8: Jump Starting Your OpenEdge Auditing Solution Stephen Ferguson
Agenda OpenEdge Auditing Overview Getting Started with Auditing Staying in Control Creative Reporting OpenEdge Auditing Overview -- What can OpenEdge Auditing do for you -- What can you do for OpenEdge Auditing Getting started with auditing -- The essential steps for enabling and configuring auditing -- Tradeoffs in configuring audit policies -- The importance of auditing application events Staying n control -- It is all about the security of your audit data -- Tuning for short and long term audit data storage -- Keeping the storage of audit data under control Getting creative -- Creative reporting This presentation includes annotations with additional complementary information DB-8: Jump Starting Your OpenEdge Auditing Solution Progress Exchange June, Phoenix, AZ, USA

What is Auditing? “The process of evaluating an organization’s practices for safeguarding electronic information from loss, damage, unintended disclosure, or denial of availability.” The OpenEdge Auditing Core Service gathers, records, and securely maintains the information necessary to perform the auditing process: Who was the client What action took place When did it happen Where did it happen From the auditor’s perspective, Auditing is an impartial evaluation of an organization’s installation, configuration, and administration of software systems to ensure that the electronic data has not been compromised or falsified through faulty or inadequate procedures. To do so, the electronic systems are required to track critical operational actions and/or operations in a way that the information can be trusted, and in a way that the information cannot itself be compromised. This way, when the collected audit data is queried to generate evaluation reports it can be relied upon to be an accurate reflection of what operations occurred, when they occurred, and who was responsible. OpenEdge aids our APs and direct end-users by providing the secure collection and storage of the operational information. To this end, effective audit trails tell you who did what, when, where and how; they must reflect the verifiable identify of the real application user; they must be complete, accurate and non-repudiable; and also show that auditing policies and data have not been tampered with DB-8: Jump Starting Your OpenEdge Auditing Solution Progress Exchange June, Phoenix, AZ, USA

What is a Core Service? Definition Non-domain specific related functions that provide the common infrastructure for a modern application Standard behavior, features and functionality independent of any specific application requirements Typically pre-started and always available OpenEdge Auditing was first implemented as a Core Service in OpenEdge 10.1A. As a core service, it offers advantages of performance, security not possible otherwise. Core Services are the responsibility of the supplier – Progress Software – to maintain and enhance, allowing our customers to focus on their own domain specific solutions; core Services are all about what we do for you, and what you do not have to bother about. It’s about what you do not have to do. Core services typically are infrastructure requirements that apply to most if not all verticals. DB-8: Jump Starting Your OpenEdge Auditing Solution Progress Exchange June, Phoenix, AZ, USA

What Can OpenEdge Auditing Do Out of the Box?
DB-8: Jump Starting Your OpenEdge Auditing Solution Stephen Ferguson What Can OpenEdge Auditing Do Out of the Box? Use the OpenEdge supplied policies and reports ABL & SQL Database connections Security administration User login/logout (needs OpenEdge security) OpenEdge DB Default record level events Schema changes Database and _User administration Audit policy and events administration Ease of reporting Flexibility and ease of reporting - from standard (but secure) tables in the database. Core services represent time and effort savings, and are really about what you do not have to do!! We do it for you. What you don’t have to do is develop, bug fix, maintain, etc, the auditing system. OpenEdge supplied policies and reports can handle a number of your basic regulatory compliance needs. Because most of you write your own application security for user accounts and table/field access controls, those OpenEdge type reports are not useful to you. After loading the OpenEdge policies, deactivate what you do not need. DB-8: Jump Starting Your OpenEdge Auditing Solution Progress Exchange June, Phoenix, AZ, USA

From Schema-Trigger Based Auditing
DB-8: Jump Starting Your OpenEdge Auditing Solution Stephen Ferguson From Schema-Trigger Based Auditing ABL Client Audit Policy Tools Application Code Audit Policy Manager API Policy Data Security Manager Application Data App DB Audit Event Manager (schema triggers) Audit Data Manager Audit Data Archive DB Archive Daemon Manager Offline Audit Data Report Manager Audit Report SQL Client Application Code This slide shows the typical components that provide auditing functionality for a Progress 4GL-based application, using a schema trigger based approach and storing the audit data in a Progress database along with other application data. CONTAINS BUILDS Click 1 – Usually the audit functionality will be controlled through some configuration settings we are referring to as the audit policy. Examples include the enabling of auditing for specific tables and fields in a database. Also shown here is all access to the database going through some security management layer to restrict who can manipulate the audit policy data. Click 2 – Application code that requires auditing is triggered through some event manager typically implemented via schema triggers. Click 3 – There is usually some manager layer that understands the structure of the audit data and automatically records information such as the execution context, the date and time, etc. The security manager layer will then typically supply who the application user is and the audit data will be written to the audit tables in the application database. Click 4 – Most auditing systems have archive functionality that support archiving to some other dedicated auditing database, some off-line storage or simply the bit bucket to delete old audit data that is no longer required. Click 5 – Reports and enquiries on the audit data are provided through some purposed reporting layer that has a knowledge of the audit data DB-8: Jump Starting Your OpenEdge Auditing Solution Progress Exchange June, Phoenix, AZ, USA

To Auditing in OpenEdge
DB-8: Jump Starting Your OpenEdge Auditing Solution Stephen Ferguson To Auditing in OpenEdge ABL Client Database Tools and Utilities Open Tools Audit Policy Tools (APMT) Application Code Audit Policy Subsystem API App DB Policy Data Audit Event Subsystem Audit Data Subsystem Application Data Audit Data Security Subsystem Application Internal Database Archive DB Application Code Archiving Subsystem Reporting Subsystem Note that Database auditing and Internal auditing are ‘automatic’, in that they do not require any code changes. Only Application auditing requires code changes. Archive Daemon SQL Client Audit Data Offline Audit Data Audit Report DB-8: Jump Starting Your OpenEdge Auditing Solution Progress Exchange June, Phoenix, AZ, USA

No Thanks, I Already Got One
DB-8: Jump Starting Your OpenEdge Auditing Solution Stephen Ferguson No Thanks, I Already Got One Why use OpenEdge Auditing over your own solution? Flexible, scalable Core Service Common built-in auditing for both SQL/ABL clients Performance, performance, performance Security Audit system events Utilities, schema changes, etc Flexible, secure reporting Archiving Multi-database, multi-application As well as the above points, an additional saving is in effort – implemented as a Core Service, you do not have to maintain or enhance the OpenEdge Auditing solution – we take responsibility for this. DB-8: Jump Starting Your OpenEdge Auditing Solution Progress Exchange June, Phoenix, AZ, USA

Agenda OpenEdge Auditing Overview Getting Started with Auditing Staying in Control Creative Reporting The following are the basic technical steps involved in getting started with auditing. See the documentation for more in depth details. The goal of this presentation is to inform you about the kind of planning and strategic decisions that have to be considered in order to get the most out of auditing. Use checklists to make sure you do not forget! Not covered here are topics such as: Authentication (who is the user, and how are they reliably identified?) – see DB19-OpenEdge Authentication without the _User Table Planning for disk space – in the production database; in the long term database; for offline storage of audit data. Deployment considerations – what to deploy, what to enable, etc What reporting tools to use – will influence how the data is stored. Separation of duty – of DBA from Audit permissions (Archiver, Audit Inserter, Audit Administrator, Audit Read permissions) What to audit? Get expert advice – from the user management; and especially from the consumers of the audit reports, i.e. the auditors themselves. Document everything!! DB-8: Jump Starting Your OpenEdge Auditing Solution Progress Exchange June, Phoenix, AZ, USA

Step 1: Before you Begin App DB Policy Data Upgrade Databases AND Clients to 10.1+ Application Data Audit Data Add Type II Storage Areas for Auditing Enable Auditing (prepares for auditing) Set database options Assign audit permissions Import shipped audit policies Another reason to go to 10.1B: removal of 2b row limits; and wide key support Large db support: less of a need to keep data and indexes separate, although it is still recommended (originally recommended for size reasons, and also for number of spindles, (this still applies if you can put your storage areas on multiple spindles). Also, can consolidate audit data from multiple application databases in to a single database, and filter by source database when reporting. Wide key support: can store larger audit context in indexed context fields (see also later) so very little reason to ever use 1 rec/field setting. Plan your space requirements. There is no reason to store ‘internal’ fields, i.e. those fields internal to the application that does not need to be recorded in audit data/to produce the audit reports. End users need to understand the environment that their database runs in and performance tune to achieve the optimum. Using their results, they would then tune for performance. DB-8: Jump Starting Your OpenEdge Auditing Solution Progress Exchange June, Phoenix, AZ, USA

Step 2: Define Your Own Audit Policies
DB-8: Jump Starting Your OpenEdge Auditing Solution Stephen Ferguson Step 2: Define Your Own Audit Policies “An Audit Policy is the configuration that controls the recording of audit data into an OpenEdge database” Through Audit Policies you control What audit information is recorded Where to store audited information How to store audited information How much audited information to store Context information to query audit information Security of audit information You will only make effective use of auditing functionality if you know how to control it – and this is achieved with audit policies. The OpenEdge auditing core service is driven by Audit Policies. In today’s complex electronic data systems there are too many data related operations taking place to simply capture everything. First your application would slow to a crawl and you would immediately overflow disk storage. Therefore you have to have a way of controlling: . What to collect . Where to store it . How much to store . Add the application and database context necessary to put data operations into perspective . And secure the data so that it cannot be compromised to hide compromised or falsified primary data storage OpenEdge audits nothing unless a policy tells us to so. There is too much overhead and storage to make any type of assumption. Policies can be maintained online. Audit data is not just for historical purposes. Consider that Auditing could also be used for code coverage analysis; event analysis (looking for sudden changes in event logging, such as failed login attempts as a denial-of-service attempt); to measure application operations and frequency, such as counting number of ‘saves’ performed by groups of users, etc. With online control of enabling/disabling, policies can be switched on or off as desired. DB-8: Jump Starting Your OpenEdge Auditing Solution Progress Exchange June, Phoenix, AZ, USA

Audit Policy Attributes
DB-8: Jump Starting Your OpenEdge Auditing Solution Stephen Ferguson Audit Policy Attributes Stored in audit-enabled OpenEdge databases Contain any number of policies Apply only to the database they are stored in Can have active or inactive state ( on/off ) Active policies are merged at load-time Can be changed and reloaded on-line Has a unique GUID identifier Policies contain event records Audit policies are stored in a OpenEdge database, and any number of policies can be stored. The policies apply only to the database they are stored inPolicies can be on of off. This allows you to generate many policies to cover many types of situations. More on this later. To allow you to specify auditable operations only once, you can make multiple small policies active where they are merged at policy load time. It’s the same principal as breaking an application into many reusable procedures instead of duplicating them into a few huge monolithic procedures. OpenEdge s moving to provide more 7 by 24 hour operations. To this end, OpenEdge auditing allows you to change your audit policies contents and active state during run-time operations and then trigger a reload at run-time. Your application will show a brief pause while the policies are reloaded at a safe-point, and then continue using the new auditing policy configuration. Auditing policy records all utilize primary keys that use a GUID identifier so that policies can be merged, moved, copied and never have conflicts due to name duplication. OpenEdge auditing is Event based. An event is an occurrence of some auditable action taken by the application. Events are defined at development time by OpenEdge and yourselves Each event has a unique numerical number Capturing events involve turning it on or off, and specifying the amount of information to record No event has a default ON state You can define and store any number of audit policies. Like events, they are OFF (inactive) until turned on. You may have multiple active policies which are merged at run-time. Policies only apply to the database they are stored in. You can make online policy changes without stopping and restarting the database or the 4GL or SQL clients. DB-8: Jump Starting Your OpenEdge Auditing Solution Progress Exchange June, Phoenix, AZ, USA

Agenda OpenEdge Auditing Overview Getting Started with Auditing Staying in Control Creative Reporting Steps: These are the typical steps in any development scenario…. It’s the typical process for any new release/anything highly configurable 1.Plan 2.Enable 3.Configure 4.Test Repeat until you get it right Contents of each stage: Plan: What do I plan? Data requirements; reporting requirements; online/offline storage requirements. Enable: What do I have to enable: Configure: Policies; events; application and database; who is the user …. Use checklists for the elements of each step. Each point is a reason for doing so. Trade-offs and implications….. Do the minimum you have to do. “What do I audit?” To answer this question you need input from experts – hire consultants; work with the auditor. Test – test your performance targets; test the reporting (can you get the reports you need?); archiving; DB-8: Jump Starting Your OpenEdge Auditing Solution Progress Exchange June, Phoenix, AZ, USA

Auditing Policy Designs
DB-8: Jump Starting Your OpenEdge Auditing Solution Stephen Ferguson Auditing Policy Designs What type of policy design do I use? Many possible designs No single right design for every application Every application has one best design The policy design is driven by Who generates and runs the reports Who generates and manages the policies Who consumes the reports The next portion of this session will be dedicated to creating audit policies. Where to start? First, OpenEdge gives you a lot of flexibility to develop audit policies that work best for your application. There are many possible designs. Why? In all our research we found that there is not single right-way to organize a policy or policies for all applications. We talk to 10 developers and get 12 different answers. However, we do believe that there is one best design for each of your applications. The problem is that you have to make that determination because only you know your customers and your application. To start on finding that best design: . you identify who will generate audit policies and what their typical skill levels are regarding your application and database . You will also identify who will be responsible for managing the policies at the production site and what their skill levels are for your application and database. Why? By looking at all the various best-practices, standards, government regulations, auditors, consultants, you will find one truth. There are not checklists, no standard technical details that will tell you precisely what the auditors will require. By the best information available today, each production site’s auditors will determine what to check, when, and how. Therefore your application and auditing solution must be adaptable to meet these different installation’s requirements. I would classify this as a mess. DB-8: Jump Starting Your OpenEdge Auditing Solution Progress Exchange June, Phoenix, AZ, USA

Audit Policy Design Goals
DB-8: Jump Starting Your OpenEdge Auditing Solution Stephen Ferguson Audit Policy Design Goals My Audit Policy design needs to Record enough to generate the reports NOT abuse disk space & performance Simplify auditing administration The next big question is how do I determine what to audit? Do I just go in and turn everything on? Please, do not do that. The rule of thumb here is: You will get out of OpenEdge auditing, and be as happy with it as you put in time to understand your customer’s needs and meet them. The whole point of auditing is to generate reports. Nobody captures terabytes of information over years of time just to use up disk space an backup media. So the reports are the key. Use consultants, your customers who have auditors or auditing experience to help guide you.2) Do not abuse disk and cpu resources at the production site by turning on all table auditing unless you have identified the positive need to do so. Just don’t turn it on and walk away. 3) Controlling the auditing details as the auditors provide requirements will probably be necessary. Now if you want to sell a service to the production site to create and make all auditing policy changes, great. Otherwise you need to consider how someone who doesn’t know the details of your application and database can understand what and when to turn things on and off. 4) The last one is a big one. This is an emerging industry, and we are all learning as we go, including the auditors themselves. So I expect that what reports are necessary and therefore what information will need to being saving will evolve over time and eventually stabilize. Until then, the faster that you can react, the bigger hero you will be. DB-8: Jump Starting Your OpenEdge Auditing Solution Progress Exchange June, Phoenix, AZ, USA

Simplifying Audit Administration
DB-8: Jump Starting Your OpenEdge Auditing Solution Stephen Ferguson Simplifying Audit Administration To simplify audit administration for the end-user Focus on who controls what to audit Their level of application and database knowledge? Your ability to do training and documentation? Your ability to support detailed policy administration? Expose only what needs to be configured Hide the detail inside built-in policies Use a management UI to activate and de-activate policies Automate easy switching to/from Auditing primary application “features” Maintenance vs. production mode Different levels of auditing detail Regarding policy administration. How much detail can I hide and should I hide? This goes back to who the policy generators are and who the administrators are. To do detailed policy generation, especially for databases, you probably need to have detailed table and field knowledge. If your end-customers don’t have that, you’ll have to document and teach. That costs you. There are ways to make it easier. The next are what are the operating environments your application will run in. Will your customers need to temporarily disable auditing to do maintenance? Will they need to use different levels and types of auditing during certain processes, such as closing the books? Will maintenance time auditing be needed to guard against hackers during that time, but other auditing should be disabled to not flood the audit trails with false information? All these will go towards telling you what policies to generate and what your support issues will be. DB-8: Jump Starting Your OpenEdge Auditing Solution Progress Exchange June, Phoenix, AZ, USA

Choosing an Audit Policy Strategy
DB-8: Jump Starting Your OpenEdge Auditing Solution Stephen Ferguson Choosing an Audit Policy Strategy What are my audit policy deployment strategy choices ? Do nothing Customer 100% responsible for generating policies Supply audit policies as templates Development supplies 80% knowledge in templates Customer customizes remaining 20% of templates Are there any liability issues? Sell audit administration as a service Developer does remote policy creation and administration Supply turn-key audit policies Developer supplies 100% knowledge Customer site uses UI tool to manage auditing All policy changes are possible ONLINE…. Your choice for auditing support for your application can take different forms. The primary ones we’ve talked with people about is: Do nothing and let the production site do all the work. Of course, they have to be as familiar with the application and database schema as the developer. You can choose to supply templates and allow the production site tailor them to their own specific requirements. Again, a lot of detail about the application and database schema is needed. You can choose to sell “Audit Administration as a service”. You do the work and administer the policies and get paid for it. The last one is what I “think” will be used a lot. You supply the policies, administration UI, and reports and the production site simply turns things on/off. When new requirements arise, you add to the functionality. DB-8: Jump Starting Your OpenEdge Auditing Solution Progress Exchange June, Phoenix, AZ, USA

What is an OpenEdge Auditing Event?
DB-8: Jump Starting Your OpenEdge Auditing Solution Stephen Ferguson What is an OpenEdge Auditing Event? “Audit Events represent the WHAT in auditing.” Each Event definition is a unique action or operation Audit Events fall into three types Database CUD ( OpenEdge ) Internal ( OpenEdge ) Application ( ABL or SQL ) Each Event definition has a Unique positive integer value ( 1 to max integer ) “name” ( “customer.create” ) “description” ( “create customer record” ) Events are the WHAT action or operation in auditing. Each Audit Event definition identifies a single action or operation. OpenEdge has three categories of events: Database Create, Update & Delete record operations Internal OpenEdge run-time and utility actions and operations ABL and SQL application events An Event is defined by a positive integer value. It has a character name, description and type name for human use. DB-8: Jump Starting Your OpenEdge Auditing Solution Progress Exchange June, Phoenix, AZ, USA

Audit Event Types Database Record Events Used for Recording a table’s row operations Create, Update, and Delete Optionally recording selected field values Recorded only in the local database Query by table name OR table and selected field values No automatic “application context” relating the record operation to application operation The first category of events is the database CUD events. As you might think, this audit table-row operations for Create, Update, and Delete. These are recorded only to the local database where the record operation takes place. You can query these events based on table name, or table name with selected field values to uniquely identify the actual record. I get this all the time, “audit database tables”. That’s just about everyone’s first thought. While database auditing is essential, it WILL NOT satisfy your auditing needs on their own. The reason why is that raw database audit records have NO CONTEXT!! Therefore for reporting they have limited usefulness as you cannot know where the database operation was performed from. We provide a method for you to manually add that application context which will be in the slides. The answer is coming next… DB-8: Jump Starting Your OpenEdge Auditing Solution Progress Exchange June, Phoenix, AZ, USA

Audit Event Types Application Events Used for Recording business level, coarse grained, events Events with no corresponding database operation “Read auditing” Applying “application context” to [record] audit events Grouping related audit events for easy queries Triggered by ABL language statements ABL or SQL application code Coded into the application Event number Audit record’s Event Context format and content Audit record’s event detail format and content Application Audit Events are the under-rated as they are essential to good auditing. They are used for recording essential audit events that have no corresponding database operation. They are also essential for providing “READ” auditing of database data. They are also the thing that adds “CONTEXT” to your database audit records. They also allow you to group audit records that are all related to a broad-scope application operation. In OpenEdge auditing, you code the full application event once into your application, because only you know what needs to be recorded, and where it should be recorded from. You do not conditionalize the code, it is not an error to execute an auditing statement when you have no audit enabled database or policy that indicates that it should be recorded. You control all that at run-time at the production site via the audit policies. DB-8: Jump Starting Your OpenEdge Auditing Solution Progress Exchange June, Phoenix, AZ, USA

Application Events and Multiple Databases
DB-8: Jump Starting Your OpenEdge Auditing Solution Stephen Ferguson Application Events and Multiple Databases What happens? Application Events are propagated to all databases Allows for immediate query of events in any database Same Audit record UUID primary index in each database (duplicate) Duplicates removed by archive utility load operation Minimize performance overhead Enable only one database’s Event policy to record the event if immediate queries are not required Application events for the ABL and SQL database clients are distributed to each connected database. The reason being so that you can make immediate queries on any local database’s audit data and get the complete picture of both record events and related application events. One reason for doing this may be an alert service that periodically scans current audit data for detecting an abnormal condition and raise and alert. Most times you will not need this immediate query capability as you will accumulate audit data in the local databases and then archive it to a purposed long term database for formal reporting. In this case, choose which database to capture application events to and enable the policy to capture those events. On all other audit enabled databases, disable capturing application events. DB-8: Jump Starting Your OpenEdge Auditing Solution Progress Exchange June, Phoenix, AZ, USA

Setting Audit Context and Scope
DB-8: Jump Starting Your OpenEdge Auditing Solution Stephen Ferguson Setting Audit Context and Scope “Audit Event Context defines a specific instance of an audit event” Event ID & Context are the primary query filters Used to simplify queries for specific Record changes Application operation or action OpenEdge operations or actions WARNING: avoid format changes at production sites (or you make queries very complex) Event Context is primary query filter for finding instances, or a set of instances of a specific Audit Event. The way you construct the event context will either make those queries simpler or harder. From here you can find which records changed, which application operations took place, and which OpenEdge internal events were triggered. The primary danger is not thinking far enough in advance. You cannot redefine event context information formats without serious consequences. Remember, what you recorded 5 years ago may have to be recovered by using one of those contexts. DB-8: Jump Starting Your OpenEdge Auditing Solution Progress Exchange June, Phoenix, AZ, USA

Event Context Strategy
DB-8: Jump Starting Your OpenEdge Auditing Solution Stephen Ferguson Event Context Strategy Simplify queries for one or more instances of an Audit Event Record Event context Query table changes by [index] field values “PUB.Customer” “PUB.Customerpluto” “PUB.Customerpluto•56 Bone Dr.” Application Event context Use multiple fields of context information c1 [ .c2 [ .c3 … ] ] More context fields yields smaller record subsets “print” “print.audit” “print.audit.users.dduck” Consider the usefulness of a strategy for reporting on context, as shown above. Index size limitations were removed in 10.1B (wide key support), so can structure the context, indexed fields as required, no need to ever go to 1 rec/fields? General strategy for record events is to put in the identifying field information to effectively zoom in on specific audit data. For records, in the Customer table I can use context to find all of the Customer table audit records for a specific operations, or operations. By adding the first identifying field, which is the customer name, I can find all of the audit records for “pluto” for a specific operations. Zooming in, I can then find all of the records for customer “pluto” who lives at 56 Bone Drive. For application event records the same strategy applies, only you get to make up your own formats and fill in the data with your application language statements. So again use multiple fields with some field separator. In my example I use a period (“.”). The more fields, the more specific can be your report queries. For example: I can find all of the print operations, then refine the query for all print operations of auditing, then refine it even more by querying all of the printing of audit information for users and for the user “dduck”. Follow this method and I think you’ll make your job much easier. DB-8: Jump Starting Your OpenEdge Auditing Solution Progress Exchange June, Phoenix, AZ, USA

Assigning Record Operation Audit Events
DB-8: Jump Starting Your OpenEdge Auditing Solution Stephen Ferguson Assigning Record Operation Audit Events Suggested File Policy Event strategy Each table has a block of 10 event numbers Related tables occupy sequential blocks Each table’s events CREATE - record create (table-base + 0) UPDATE - record update (table-base + 1) DELETE - record delete (table-base + 2) VIEW viewed by terminal user * (table-base + 3) IMPORT - electronic transfer in * (table-base + 4) EXPORT - electronic transfer out * (table-base + 5) PRINT paper copy made * (table-base + 6) REPLICA - electronic copy made * (table-base + 7) Controlled in table policies Controlled in event policies * DB-8: Jump Starting Your OpenEdge Auditing Solution Progress Exchange June, Phoenix, AZ, USA

Audit Event Types “Read” Audit Events Regulations audit the “human” data access Only application knows the “human” access OpenEdge reads many records in a query Filtered record set returned to application Read is not the only “human” access Printed reports Electronic copy to removable media Network transport to external application Lets talk about READ auditing for a moment. Our analysis so far indicate the recording physical record reads results in gargantuan numbers of audit records created. This provides a false audit trail. Regulations are generally the same and want to know which HUMANS have seen the information, not what application routine. You may or may not know this, but a number of queries can result in large numbers of records “read”, but very few delivered to your application code. Next, does your application code always make every available record visible to a human? No, it doesn’t. Secondarily, if you think on what human has access, how about if you send information to a Web Service, print it, send on a socket, and so forth. Here it will benefit you to qualify each “read” by “how” it was read of made available to humans outside of the database. DB-8: Jump Starting Your OpenEdge Auditing Solution Progress Exchange June, Phoenix, AZ, USA

Keeping the long-term storage under control
DB-8: Jump Starting Your OpenEdge Auditing Solution Stephen Ferguson Keeping the long-term storage under control Audit Archiving Purposed, Long Term Storage Short Term Storage Offline Storage Audit Archive DB Application DB Audit Data Consider the ‘life cycle’ of auditing: production database(s); to long term storage; to offline storage (may have to keep audited data for many years after the event). Space requirements for all this needs planning; recovery process (from offline storage back to an online storage that supports additional reporting). Plan for the amount of storage space required for long term (single database reporting) and offline storage; how long does the data need to be retained, legally? A single audit archive database can be used for audit data from multiple applications/databases; filter on the source database, held with each record) Database auditing data is initially maintained in the database where the audited event occurred (application events may be recorded directly to the long term storage); think of the application database as the “Short Term Storage” database. Over time it is anticipated that the growth of this data would have a performance impact for the running of applications and database maintenance; as well as for the audit reporting. Consider then the idea of a “Long Term Storage” database. This Long Term Storage database will maintain the audit data for one or more databases. Data in audit tables can only be read and removed by a user with Audit Archival privileges. Data is migrated to the Long Term Storage database either through a process written as a user application and executed by a user with Audit Archive privilege or by the Audit Archive Utility. A user must login as a user with Audit Archive privileges to execute this utility. This means that a user must be granted the Audit Archiver role before this utility can be run. This utility can be executed in a number of ways. It may be run on-demand from a command line by an audit administrator or as a scheduled (CRON) system task. The archival process performs a move operation of audit data. There is also the option to simply delete the audit data rather than moving it to a long term storage database, e.g. if data has been previously archived. Audit data should be archived frequently from the production database. How frequent audit data should be archived depends on the frequency and size of audit data records being written – and this will vary depending on how auditing has been implemented. Database limits have not changed in 10.1A, for example the 2billion row limit still exists and needs to be avoided. See also the document ‘EstimatingAuditingDiskSpaceUsage.doc’. .abd file(s) Audit Archiver(s) Audit Archive Audit Archive Loader(s) Audit Reports DB-8: Jump Starting Your OpenEdge Auditing Solution Progress Exchange June, Phoenix, AZ, USA

Auditing Archive Strategy
DB-8: Jump Starting Your OpenEdge Auditing Solution Stephen Ferguson Auditing Archive Strategy Consider application database as short term storage for audit data Do not enable audit indexes Use separate storage area for audit data Archive often! Use purposed database for audit archive / reporting Enable all indexes Plan for off-line storage DB-8: Jump Starting Your OpenEdge Auditing Solution Progress Exchange June, Phoenix, AZ, USA

Agenda OpenEdge Auditing Overview Getting Started with Auditing Staying in Control Creative Reporting OpenEdge Auditing Overview -- What can OpenEdge Auditing do for you -- What can you do for OpenEdge Auditing Getting started with auditing -- The essential steps for enabling and configuring auditing -- Tradeoffs in configuring audit policies -- The importance of auditing application events Staying n control -- It is all about the security of your audit data -- Tuning for short and long term audit data storage -- Keeping the storage of audit data under control Getting creative -- Creative reporting DB-8: Jump Starting Your OpenEdge Auditing Solution Progress Exchange June, Phoenix, AZ, USA

Generating the required reports
DB-8: Jump Starting Your OpenEdge Auditing Solution Stephen Ferguson Generating the required reports The audit reports drive which Tables need audit policies Which record operations need auditing Fields values need to be recorded Field values need to be indexed Application events are needed and where Application event context formats and values to use Application-context and Audit-event-group to use Where in the application code Spanning which procedures and classes The next few slides address each of the points. Required reports will tell you what application events to define and add to the application. It will tell you what tables will need to be audited, which fields clearly identify an auditable application operation, whether the field’s value is necessary or not, and whether you need to be able to index off the field change to generate a report. What I see so far is that table and field auditing is important when you have a one-to-one relationship between a table/field operation and an auditable event. For example, adding and removing user accounts are auditable events. They just so happen to map 1-to-1 with table operations. So I can economically audit the table and not have to make code changes to my application. DB-8: Jump Starting Your OpenEdge Auditing Solution Progress Exchange June, Phoenix, AZ, USA

Querying Audit Data Reporting Subsystem Secure access to audit data Separation of duty Exposed as standard database tables for ease of reporting Requires knowledge of the implementation Schema and meta-schema Identifying fields How context is formatted (Varies by event id) Audit data searchable by User id, event id, date, context, transaction, audit group, DB connection, client session Only users granted read permission (Except SQL for Open Edge 10.1a) DB-8: Jump Starting Your OpenEdge Auditing Solution Progress Exchange June, Phoenix, AZ, USA

Querying Audit Transactional Data
DB-8: Jump Starting Your OpenEdge Auditing Solution Stephen Ferguson Querying Audit Transactional Data Only record what you need to report Use structured event names _sys.tbl.create _sys.tbl.trig.update Use reporting database Avoids SHARE-LOCK Stringed values always in American format SESSION:DATE-FORMAT = “mdy” SESSION:NUMERIC-FORMAT = “American” Client Session Information Audit Report Audit Transaction Data (FK) Foreign Key (IEx.y) Inversion Entry (non-unique) x = Index Number y = Field Order in Index Modified Values Per field Recursive Join DB-8: Jump Starting Your OpenEdge Auditing Solution Progress Exchange June, Phoenix, AZ, USA

What information is recorded?
DB-8: Jump Starting Your OpenEdge Auditing Solution Stephen Ferguson What information is recorded? Who did it? When did it happen? What event caused it? What was the event on? What was going on at the time? Any other relevant info? DB-8: Jump Starting Your OpenEdge Auditing Solution Progress Exchange June, Phoenix, AZ, USA

Reporting on Application Context and Event Groups
DB-8: Jump Starting Your OpenEdge Auditing Solution Stephen Ferguson Reporting on Application Context and Event Groups Application-Context and Audit-event-groups Are a form of application audit event Normalize applying “application context” to Database record audit events Other application audit events Group related audit records across multiple databases UUID AB627H8 Event Application-context-id Event context “Record visit” UUID G78456U Event Application-context-id AB627H8 Event context “Visit OK Btn” A very powerful type of application event is the Application-Context and Audit-Event-Group events. These events are pre-defined by OpenEdge and live next to the application event number space to facilitate application event queries by event number range. Most times the same exact application context applies to many audit records. Recording all of it in each audit record can really take up a lot of space. So OpenEdge normalizes application context by writing it all in one record and then pointing to that record from the related audit records. Not only can you relate all of the audit records in a single database, but you can relate all of the audit records in multiple databases. For example, if you do a ProDataset UPDATE that includes 4 databases, using these application events you can query all of the events, in all the databases, for the single ProDataset UPDTE operation. These two events are language statements and you control in your code when they start and when they end. They have a lifetime. When the lifetime starts, OpenEdge records the event’s UUID identifier into every audit record it creates, and continues to do so until you turn it off. Of course this has great impact on your coding for error handling. You cannot afford to not end these events if an error is raised. You always have to trap the error and end the event. This may be extra work, but the value that these events give you with regards to adding application context and grouping logically related audit events for query and reporting will justify it. Application events Application-Context event UUID Q2395NL Event Application-context-id AB627H8 Event context “PUB.T1:Jones” Record events DB-8: Jump Starting Your OpenEdge Auditing Solution Progress Exchange June, Phoenix, AZ, USA

Auditing Best Practices
DB-8: Jump Starting Your OpenEdge Auditing Solution Stephen Ferguson Auditing Best Practices Only audit what is absolutely necessary – tune with audit policy maintenance Plan for reporting Group event ids into ranges Structure context consistently Leverage audit event groups Coding style even more important (assigns, record scope, transaction scope) Tune performance through Audit Policy Careful how ASSIGN indexed fields – do in single statement Carefully control record and transaction scope Every database update causes an audit event Consider reporting / query requirements DB-8: Jump Starting Your OpenEdge Auditing Solution Progress Exchange June, Phoenix, AZ, USA

In Summary Auditing is a Core Service One of many new features in OpenEdge 10 Spend time planning your implementation Auditing is a Core Service – it’s all about what you don’t have to do; and doing it faster and more secure, than is possible with triggers; it frees you to focus on your application vertical market. One of many new features in OpenEdge 10 – reduces complexity in general and provides very valuable features; even better management of the audit life cycle and reporting; secure (key to compliance) and scalable. It just gets better – auditing is a rich and powerful feature, very configurable, and time spent planning, implementing and testing your own implementation of features pays dividends in terms of what you can get out of it. DB-8: Jump Starting Your OpenEdge Auditing Solution Progress Exchange June, Phoenix, AZ, USA

Relevant Exchange Sessions
DB-8: Jump Starting Your OpenEdge Auditing Solution Stephen Ferguson Relevant Exchange Sessions DB-19: OpenEdge Authentication Without the _User Table DB-14: OpenEdge run-time database security revealed DB-8: Jump Starting Your OpenEdge Auditing Solution Progress Exchange June, Phoenix, AZ, USA

Education / Documentation References
DB-8: Jump Starting Your OpenEdge Auditing Solution Stephen Ferguson Education / Documentation References Education What's New In OpenEdge 10.1: Auditing Documentation Core Business Services PSDN DB-8: Jump Starting Your OpenEdge Auditing Solution Progress Exchange June, Phoenix, AZ, USA

Questions? DB-8: Jump Starting Your OpenEdge Auditing Solution Progress Exchange June, Phoenix, AZ, USA

Thank you for your time DB-8: Jump Starting Your OpenEdge Auditing Solution Progress Exchange June, Phoenix, AZ, USA

Progress Exchange June, Phoenix, AZ, USA

Preparing for Auditing
DB-8: Jump Starting Your OpenEdge Auditing Solution Stephen Ferguson Preparing for Auditing App DB Preparing for auditing Policy Data Upgrade Databases AND Clients to 10.1A+ Add Type II Storage Areas for Auditing prostrct add <db> addaudit.st Enable Auditing (prepares for auditing) Application Data Audit Data d "Audit_Data":20,32; f 40960 d "Audit_Data":20,32;512 . d "Audit_Index":21,1; f 5120 d "Audit_Index":21,1;64 . Another reason to go to 10.1B: removal of 2b row limits; and wide key support Large db support: no longer need to keep data and indexes separate (originally recommended for size reasons, and also for number of spindles (this still applies if you can pout your stroage areas on multiple spindles). Also, can consolidate audit data from multiple application databases in to a single database, and filter by source database when reporting. Wide key support: can store larger audit context in indexed context fields (see also later) so very little reason to ever use 1 rec/field setting. Plan your space requirements. There is no reason to store ‘internal’ fields, i.e. those fields internal to the application that does not need to be recorded in audit data/to produce the audit reports. End users need to understand the environment that their database runs in and performance tune to achieve the optimum. Using their results, they would then tune for performance. proutil <db> -C enableauditing area “Audit_Data” indexarea “Audit_Index” [deactivateidx] DB-8: Jump Starting Your OpenEdge Auditing Solution Progress Exchange June, Phoenix, AZ, USA

Database Options and Audit Permissions
DB-8: Jump Starting Your OpenEdge Auditing Solution Stephen Ferguson Database Options and Audit Permissions Security Subsystem DB-8: Jump Starting Your OpenEdge Auditing Solution Progress Exchange June, Phoenix, AZ, USA

Application Context and Audit Event Groups
DB-8: Jump Starting Your OpenEdge Auditing Solution Stephen Ferguson Application Context and Audit Event Groups Example usage DEFINE VARIABLE ctxID AS CHARACTER. DEFINE VARIABLE grpID AS CHARACTER. ctxID = AUDIT-CONTROL:SET-APPL-CONTEXT (PROGRAM-NAME(1) + “:Create Order", cOrderData,cExtraStuff). … grpID = AUDIT-CONTROL:BEGIN-EVENT-GROUP (PROGRAM-NAME(1) + “:Create Order Line", cLineData,cExtraStuff). AUDIT-CONTROL:END-EVENT-GROUP. AUDIT-CONTROL:CLEAR-APPL-CONTEXT. Indexed Indexed Include certain slides such as this (i.e. ones showing code); for reference, after the main body of the presentation. It’s useful to know, but not required at this level. Application context record (parent) = _Event-id = 31998 Event group record (parent) = _Event-id = 31999 Requires recursive join on _aud-audit-data to obtain details of context / event group _Application-context-id = parent _Audit-data-guid _Audit-event-group = parent _Audit-data-guid Brackets audit events with a common tag (UUID) Detail record replicated in each database with event id (context / group 31999) enabled Replicated detail records have the same UUID Duplicates eliminated by archive tool Must manually clear context / event group Beware error handling No support for nesting Focus on error handling… Control and responsibility The Application Context and AEG require an Event-context field of character information that also serves as an index to locate certain types of contexts for reporting. Optionally two additional fields of character information, up to a maximum of 10,000 bytes per field, can be supplied. Each time an Application Context or AEG context is begun a single auditing event record is written to the OpenEdge database’s _aud-audit-data table. The application supplied information is placed into the record with OpenEdge filling in the remaining user, date, context, and UUID primary index information automatically. Each database that has an interest in the creation of an application context is sent an appropriate event. The UUID value will also be saved as the APPL-CONTEXT-ID attribute for each database connection. The UUID of the application-context record’s primary index is returned by this method. This UUID will be recorded in all subsequently generated audit event records., until either this method or the CLEAR-APPL-CONTEXT method is called. This method will cause a new application-context audit record to be created in all audit enabled databases whose audit policy includes an interest in this event. If any argument has the unknown value, an error will be generated. DB-8: Jump Starting Your OpenEdge Auditing Solution Progress Exchange June, Phoenix, AZ, USA

Audit Event Types Default Database Record Events Name Event-id Description Type _sys.db.rec.create 5100 “Create record” “schema” _sys.db.rec.update 5101 “Update record” _sys.db.rec.delete 5102 “Delete record” To get developers started, we pre-define 3 events. We DO NOT recommend that you use these for production. It will lead you into query and reporting problems. There are better ways that we’ll talk about later. Demonstration and development purposes Recommend using application defined event IDs for Production auditing DB-8: Jump Starting Your OpenEdge Auditing Solution Progress Exchange June, Phoenix, AZ, USA

Recording Field Values
DB-8: Jump Starting Your OpenEdge Auditing Solution Stephen Ferguson Recording Field Values Selectable via table / field policy Streamed (default) Modified values stored in _Event-detail field of the primary _aud-audit-data record Minimizes performance impact Limited by max record length – auto overflows Arbitrary field order / content <fld-nam> + CHR(6) + <data-typ> + CHR(6) + [<old-val> +] CHR(6) + <new-val> + CHR(7) CHR(8) is used to delimit array elements One Record per Field Query for specific field value changes DB-8: Jump Starting Your OpenEdge Auditing Solution Progress Exchange June, Phoenix, AZ, USA

Controlling the Storage of Audited Field Values
DB-8: Jump Starting Your OpenEdge Auditing Solution Stephen Ferguson Controlling the Storage of Audited Field Values f1 f2 f3 f4 f5 f6 f7 f8 f9 f10 f11 f12 f13 f14 Database record Audit Event Subsystem Audit Policy Subsystem Database _aud-file-policy _aud-field-policy f1 f2 f3 f4 f5 f6 f7 f8 f9 f10 f11 f12 f13 f14 Audited Fields (f2, f6, f9, f14) “1 Field/Record” Audit Data Subsystem “f1:old/new, f3:old/new, f10:old/new” “Streamed Field Values” Controlling where OpenEdge stores the audited field values in two locations: The audit data record itself as event detail information. This is streamed as a delimited character string in American format, ready for reporting. Individual field value records which are indexed so that you can query on specific field changes. When the record arrives at the audit event subsystem, the table’s file and field policies are consulted and fields that will have their values recorded marked. The marked field values are split by the Audit Data subsystem the two locations. The streamed field values are handled firsts until there are no more field values to stream or the Audit record’s detail field is full. Next, the fields marked as “1 field/record” or could not fit into the Audit record’s detail field are written into individual audit data value records. Last, the streamed values are added to the Audit record and the record created. _aud-audit-data _aud-audit-data _aud-audit-data-value _Event-detail DB-8: Jump Starting Your OpenEdge Auditing Solution Progress Exchange June, Phoenix, AZ, USA

Application Event Examples
DB-8: Jump Starting Your OpenEdge Auditing Solution Stephen Ferguson Application Event Examples … /* = Run Menu Option */ AppID = AUDIT-CONTROL:LOG-AUDIT-EVENT (32800, cMenuCode, cDetail, cMore). /* READ auditing = Customer Enquiry */ (32003, STRING(Customer.CustNum), cCustomerDetail, cMore). Indexed Indexed The above examples show how LOG-AUDIT-EVENT can be used to log audit records when a procedure begins and log the name of the procedure, plus any customer data – as represented by the variables cDetail and cUserData; and how read auditing can be implemented using LOG-AUDIT-EVENT. The full syntax for LOG-AUDIT-EVENT is: Ctx-Id = AUDIT-CONTROL:LOG-AUDIT-EVENT ( event-id, event-context [, event-detail [, user-details]]). Where Ctx-Id = The context UUID generated by the client and sent to each database. A base64 encoded value, of length 22. The two trailing base64 pad characters (‘=’) are removed. The context ID is the primary index of the resulting audit record. Character. event-id = The id of the event. Must be greater or equal to and represent a defined audit event in the _aud-event table. Integer expression. event-context = The event description. The content of this argument is an alternative index to the audit record and is used to categorize the source of the audit event. Character expression. event-detail = Expression containing user defined audit information. Optional. Character. user-data = Expression containing user data. Optional. Character. Recall that ‘event-detail’ recording is controlled in any policy by the ‘Event level’ setting on the event policy; and that ‘user-data’ recording is controlled by the ‘Audit custom detail setting’, both of which are accessible using Audit Policy Maintenance. DB-8: Jump Starting Your OpenEdge Auditing Solution Progress Exchange June, Phoenix, AZ, USA

Audit Event Types Internal Audit Events Are a form of application audit event Could not be captured by an application’s bespoke auditing system Are triggered by internal OpenEdge operations ABL & SQL database clients Database utilities Ids are predefined by OpenEdge In OpenEdge controlled event-id space [ 0 – 31,999 ] _pvm.user.login.pass #10510 _sys.audit.data.dump #10310 _sys.tbl.create #5000 _sql.dba.create #210 _sys.area.truncate #10209 The last type is the internal actions and operations internal to OpenEdge ABL, SQL, RDBMS, utilities. All these are pre-defined in the OpenEdge event range 0 to 31,999. DB-8: Jump Starting Your OpenEdge Auditing Solution Progress Exchange June, Phoenix, AZ, USA

Locating Specific Audit Data
DB-8: Jump Starting Your OpenEdge Auditing Solution Stephen Ferguson Locating Specific Audit Data Event context field _aud-audit-data._event-context DEFINE VARIABLE cKey AS CHARACTER NO-UNDO. ASSIGN cKey = "PUB.orderline" + CHR(6) + STRING(orderline.ordernum) + CHR(7) + STRING(orderline.linenum). IF CAN-FIND(FIRST _aud-audit-data NO-LOCK WHERE _aud-audit-data._event-context = cKey) THEN MESSAGE "Audit data exists for " + cKey. By default uses Primary Key Fields Store the code in a later section. Replace with a summary of what can be reported on – context fields, indexed fields, etc. Look at the schema for this? Consider: You need to know how the audit data is stored AND what is stored, in order to plan your reports effectievely. Primary way of locating specific record level events E.g. audit data for customer number 10 Default is to use primary index fields Avoid large character fields (max is 200 chars) Can only use audited table fields (no FKs) Avoid changing fields used / structure over time Makes it difficult to find records <owner>.<table>CHR(6)<id-fld-val>[CHR(7)<id-fld-val>.. ] CHR(8) is used to delimit array elements DB-8: Jump Starting Your OpenEdge Auditing Solution Progress Exchange June, Phoenix, AZ, USA

DB-8: Jump Starting Your OpenEdge® Auditing Solution

Similar presentations

Presentation on theme: "DB-8: Jump Starting Your OpenEdge® Auditing Solution"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

DB-8: Jump Starting Your OpenEdge® Auditing Solution

Similar presentations

Presentation on theme: "DB-8: Jump Starting Your OpenEdge® Auditing Solution"— Presentation transcript:

Similar presentations

About project

Feedback