Presentation is loading. Please wait.

Presentation is loading. Please wait.

User Group Conference 2012 Advanced Tech Track 1.

Similar presentations

Presentation on theme: "User Group Conference 2012 Advanced Tech Track 1."— Presentation transcript:

1 User Group Conference 2012 Advanced Tech Track 1

2 Advanced Tech Track (2012) Introductions Course Materials
Discuss Agenda Advanced Tech Print Automation & Document Management System Administration Fundamentals (Part 1) System Administration Fundamentals (Part 2)

3 Introductions Ron Motley (Host) And you are … CTO/CIO
Director ASP Operations Chief Programmer/Analyst Older than I sound, Younger than I look…  And you are … Managers Administrators Engineers

4 Course Materials Handouts Flash-Drives
Have placed them on the Flash-Drives. Providing notepads instead. Flash-Drives Presentations Supporting Documents Tools & Utilities

5 Agenda Print Automation & Document Management Mon (3pm – 4:15pm)
System Administration Fundamentals (Part 1) Tue (1:15pm – 2:15pm) System Administration Fundamentals (Part 2) Tue (2:45pm – 4:00pm)

6 Print Automation & Document Management
Reform Forms Management Solution What is “Reform” Fabsoft Partnership What does Reform Cost Hosted Solution Forms Modification More Q & A

7 What is “Reform” Fax Archive Print Reform is an electronic Forms application intended to replace pre-printed Forms Reform is a third party software product from Fabsoft Inc. that interfaces with the Élan Software When generating a form from Élan (for example, an Invoice or Statement) you can choose to use an electronic form. If you do so, then instead of loading a pre-printed form in to your printer, the electronic form is overlaid on to the data from Élan

8 Capture & Distribution
Fabsoft Partnership Variable Data Printing (VDP) Spooler Separated From Designer Plug-In Architecture for all Options PDF Capabilities Embedded Domain Support Reform Dashboard End User - Will see on a webpage their documents and status of their documents Help Desk View - Will Allow the help desk personnel to filter the activities of the end user such as a failing device causing the end user's documents to not process correctly Health Monitor Will monitor all aspects of the solution and will alert anyone if something goes wrong with the workflow Capture & Distribution Output & Security Storage & Management

9 What does Reform Cost There are three versions of Reform
Basic Developer Developer+ Reform Hosted at MSG’s cloud site? Yes No Can I Modify Forms? Use Reform with non Élan Apps? Is Training Required? Cost $3,000 $6,000 $9,000 With the Basic version of Reform you are limited to changing the logos on the forms. For Developer+ you will need to run Reform on your own dedicated server. The Pricing of the Developer and Developer+ versions include 4 hours of Web training. Standard Hourly Rate for Customization ($165.00).

10 Hosted Solution Discuss Linked Files! Secure Access 128bit Encryption
Print-Job (PDF) Secure Access 128bit Encryption Any Postscript Device Automation is Simple Forms Printer Fabsoft Virtual Machines Client Dedicated! PDF Generated by Reform Discuss Linked Files! Forms Data Élan Login Élan Gateway (RDP) Server Hosted Élan Database Server Where there are remote client sites, all printers should be networked to the main print server with IP addresses and an Internet connection. If printers are not networked to the main print server each office must have a print server.

11 Forms Modification With Developer or Developer+ versions you can make changes – with some limitations: The data on the form (generated by Élan) can be repositioned. You can also create multiple versions of a form. For example, you may have invoices with different logos for different companies or products. You can change fonts and font sizes on forms. You can add graphics and other formatting to the design layer of a form You can add your own static text, such as Terms and Conditions What would require custom programming? Some examples: Adding fields that are not currently available on the form Adding totals, sub-totals or special calculations that are not already included on the form

12 Forms Modification Some Standard Forms:
Module Form Type Form Name Ad Manager COMBINED ORDER CONFIRMATION MAXCACK SINGLE ORDER CONFIRMATION MAXXACK ORDER ACKNOWLEDGEMENT MAPXACK STANDARD INVOICE MAXXINV Accounts Payable CHECKS - SHORT FORM MAPSCHK CHECKS - LONG FORM MAPXCHK Accounts Receivable CLIENT INVOICE MCXXINV STATEMENT MAXXSTMT Book INVOICE MBWXINV PICKLIST MBWXPICK Conference MCFXINV SPONSORSHIP INVOICE MCFXSPO Exhibits SERVICE INVOICE MEXSINV EXHIBITION INVOICE MEXXINV Internet Ad Manager MINXINV Multi Media Contracts MAXXIST Single Copy Sales MSCSINV What would require custom programming? Some examples: Adding fields that are not currently available on the form Adding totals, sub-totals or special calculations that are not already included on the form

13 Questions & Answers Features Functionality Migration to Hosted Model
There is no upgrade path to model where they have designer – Fabsoft is charging full price, we can take in and setup with no designer for (cost of pieces we need). If you keep designer you have to re buy the version 14 at 6k – but no more hardware costs or maintenance. Basic Developer Developer+ Reform Hosted at MSG’s cloud site? Yes No Can I Modify Forms? Use Reform with non Élan Apps? Is Training Required? Cost $3,000 $6,000 $9,000 $1,500.00 $2,500.00

14 System Administration Fundamentals Part 1
Concepts & Tools Service Management Memory Management Process Management Lock Management Backups & Recovery

15 Concepts & Tools Concepts Tools Client Server Architecture
Unidata DBMS (Overview) Unidata Architecture Unidata Shell (ECL) Tools UniAdmin Extensible Admin Tool Command Line (Introduced as we go!)

16 Client Server Architecture
Unidata Windows TCP/IP (WinSock) Ports Telnet (TCP 23) UniRPC (TCP 31438) SSL (992) Lets take a look at RPC Conduit! Even with Web front-end (Elan.Web) – Backend will be Unidata (Maintenance will not change)

17 Unidata DBMS Overview UniData is a relational database management system (RDBMS) designed specifically for business use. A database management system is software that enables users to control the organization, storage, retrieval, security, and integrity of data in a database. UniData’s technology is based on the nested relational database model. This nested design extends the standard relational model from a redundant, flat table structure to a three-dimensional database. In the nested relational model that UniData employs, data is stored in files, and the records in a file can have attributes that contain values that can be either singlevalued, multivalued, or multi-sub-valued. This allows UniData to store relations within relations. UniData’s method of storing data accomplishes the following: Simplifies the data modeling process Reduces the number of files and indexes in a database Eliminates redundant data storage Maintains relational access to all data in the database

18 Unidata DBMS Overview Natural Database!
Has variable length fields and Files Allows for Tables within Tables Resulting in extremely condensed, efficient data Comparison to “first normal form” database 40% to 60% of SQL data is spaces and zeros (PAD filling for fixed length fields)

19 Unidata DBMS Overview U2’s variable length fields, files & data provide for a very fluid environment If you don’t do anything on the system, it will just run slower! 40 to 60% the size of a comparable first normal form data server It’s compressed data Random –no sequential IO Disk subsystems must be tuned differently! Places the burden on the CPU at the right time After the data has been transferred from disk to memory U2 is a relational database Like all relational DBs, 85% of our Disk I/O is READS (Triggers!) Anything we can do to reduce READS is a good thing Smokin fast!

20 Unidata DBMS Overview The nature of U2 Database (More when we discuss Monitoring/Tuning) Unidata and Universe (U2) are both written in the “C” language U2 uses standard “C” calls to the operating system (MALOC) Whatever is presented via the O/S, Unidata can utilize Disk subsystem architecture (Shadow-Copy API) From it’s variable length records U2 generates millions of 2k – 8k Read/Write (IO) requests (we actually like to align with O/S read/write buffers) U2 will overwhelm a system that was configured for “First Normal Form” Data (Default) Default stripe sizes are too large! – 16k to 64k is best Stripe sizes must be reduced, otherwise entire files will be placed on single disk FNF servers perform “Prefetch”, U2 needs this disabled Selects will use, but not worth having on for just this use Raid5 can compound IO issues in U2 (We will discuss extensively)

21 Unidata Architecture Elements of the Unidata Database
Database (Account) – SBLIVE is software Account A UniData account is a directory that contains a default set of UniData files, including a VOC file and its dictionary. The VOC file identifies commands, paragraphs, and all data files that are used in the UniData account. The data files may be in the same directory as the VOC file, or the VOC file may contain pointers to data files in other directories. File (CLIENTS, PBOOKINGS, SU.SUB) UniData stores your data in hashed files of several different types. UniData also supplies other types of files to support your database, including index files, program files, and directory files. While files contain records, records contain one or more attributes (Fields) that contain specific information about one element of the record. For example: The CLIENTS file contains records about your customers. Each record contains elements of information including; name, address, and phone number, each of which are stored in separate attributes .

22 Unidata Architecture Elements of the Unidata Database Dictionaries
Every UniData data file has a corresponding dictionary file. A dictionary contains a set of records that define the structure of the records in the data file, called D-type records. A dictionary may also contain phrases, called PH-type records, and items that calculate or manipulate data, called virtual fields, or V-type records. Dictionary File (D_CLIENTS) A dictionary file, just like a data file, is a collection of records containing attributes. The purpose of a D-type dictionary record is to define the location of attributes in the data file. Other information, including the conversion code, column display heading, display format. Primary Key A primary key is a unique identifier for a record; it is also called a record ID, Each record must have a unique identifier that differentiates it from other records in the database. UniData uses the primary key to store and locate records in files

23 Unidata Architecture Elements of the Unidata Database
Variable-Length Attributes and Records Unlike other databases that require you to specify an attribute length, UniData does not limit the length of your data within an attribute. Nor does it pad the attribute or record to achieve a fixed length. UniData does not have to rely on fixed lengths to recognize where an attribute begins and ends. Instead, it inserts delimiters between attributes and between records. This is called a dynamic array structure. (nonprinting) 255 Record mark * Marks the end of a record. ~ 254 Attribute mark * Marks the end of an attribute. } 253 Value mark * Marks the end of multivalue. | 252 Subvalue mark * Marks the end of multi-sub-value. { 251 Text mark * Marks the point where values wrap to a new line

24 Unidata Architecture UniData uses several access methods to store, manipulate, and report data. Briefly, these are: UniBasic: UniBasic is a powerful programming language that is used to write sophisticated programs. From within UniBasic, we can run system-level processes or call C programs. SB+ (System Builder 4GL) is written in UniBasic UniQuery: UniQuery is UniData’s nonprocedural query language for creating reports and viewing your database (Abstracted by SB). UniData SQL: UniData’s implementation of the SQL (Structured Query Language) data manipulation language. AE: AE is UniData’s “Alternate Editor”. AE is used to edit records in the database and create UniBasic programs. (Let’s Discuss).

25 Unidata Shell (ECL) UniData provides the Environment Control Language (ECL), a proprietary command language to handle database management functions. ECL commands execute from the UniData colon prompt (:). SB+ Abstracts this functionality (Discuss LOGIN/LOGOUT Paragraphs). ECL commands and keywords install when you install UniData. They are stored in the UniData Vocabulary (VOC) file. UniData also provides system-level commands which you execute from the shell prompt (stored in the udtbin directory). You can execute system-level commands from the UniData colon prompt by entering the ! (bang) command first (for example, :!listuser). Let’s discuss some key ECL commands. LIST SORT/SELECT/SSELECT COUNT

26 Will be your primary tools!

27 Service Management Unidata Database Service 7.2
Shared Basic Code Server (sbcs) Shared Memory Manager (smm) Clean Up (cleanupd) Unidata Telnet Service 7.2 Attaches udt.exe to Database Service Unidata Terminal Service 7.2 (Not needed – Disable) Unidata ObjectCall Service 7.2 (No longer used) UniRPC Service (UniObjects 3.0) Unidata Mail Service (SMTP Bridge) Task Manager

28 Shared Basic Code Server (sbcs)
The shared basic code server (sbcs) manages shared memory used by globally cataloged UniBasic programs. UniData starts sbcs when you start the UniData Database Service (startud), and stops it when you stop the UniData Database Service (stopud). The functions of sbcs include: Loading and tracking globally cataloged programs—sbcs loads globally cataloged programs into shared memory as needed, and keeps track of the programs loaded and the number of processes executing each one. Periodically, sbcs checks shared memory and removes loaded programs that are no longer in use. Controlling shared memory—The sbcs daemon can attach up to 20 shared memory segments. Process cleanup—At periodic intervals, the sbcs process checks the cleanupd service to see if terminated process flags have been set. If sbcs detects a terminated process flag, it removes all messages sent for the process. If the terminated process is the only process using a program in shared memory, it removes the program from shared memory. Sbcsprogs

29 Shared Memory Manager (smm)
The shared memory manager (smm) builds and manages structures and tables within shared memory. UniData starts smm when you start the UniData Database Service (startud), and stops it when you stop the UniData Database Service (stopud). UniData processes (udt processes) communicate with smm to request and return shared memory. The UniData processes request shared memory from smm for the following tasks: License control—The smm process tracks the number of users for which a site is licensed, and prevents more than that number of users from logging on to UniData. smm also displays warning messages when a license is about to expire. User process tracking—When a user logs on to UniData, smm assigns an internal tracking number to the user’s process and records information about the process in tables within UniData. Buffering program variables. Storing query records and intermediate results. Storing select lists. Storing expression buffers. Managing a current modulo table for dynamic files.

30 Clean Up (cleanupd) The clean up process (cleanupd) detects terminated user processes at check time intervals. If cleanupd detects a terminated process, internal flags are set. The smm and sbcs services periodically check to see if cleanupd has set internal flags. If these services detect flags, each service performs the necessary clean up and resets its own flag to zero. The cleanupd service performs clean up that is not handled by smm or sbcs. When the smm and sbcs services have reset their flags to zero, the cleanupd service resets its flag to zero, makes the user process ID available, and frees the local control table. UniData starts smm when you start the UniData Database Service (startud), and stops it when you stop the UniData Database Service (stopud).

31 UniData Telnet Service (udinetd.exe)
The UniData Telnet Service (udinetd.exe) enables multiple users to log on to a single Windows platform to run UniData. With the UniData Telnet Service installed and started, your Windows platform exports a logon prompt to its network so that network users can log on and run UniData. When a user opens a UniData session through udinetd.exe (23 or 992), UniData writes screen output directly to a socket.

32 UniRPC Service Uses Port 31438
The UniRPC service allows the local UniData systems to communicate with remote systems/software. The communicating systems must use TCP/IP networking software to make connections. The UniRPC service waits for a request from a client system to connect to a server process. When it receives a connection request, it checks the unirpcservices files to verify that the client system is allowed to request a particular service. If it can, the UniRPC starts the requested service, then returns to the listening state. Each client process connects to its own server process. Each server process uses the same amount of system resources as a local database user. Note: The terms local and remote refer to client and server programs or systems. However, because client programs can connect to server programs running on the same computer, remote does not necessarily imply that the server is on another physical computer system. Uses Port 31438

33 Log Files (D:\Unidata\Bin)
The sbcs, cleanupd, and smm services each record messages in a pair of logs in the udtbin directory. In addition, the udt process writes messages to a log file called udt.errlog if a UniData process encounters file corruption in a data file. More on logs later! Service/Process Routine Messages Error Messages smm smm.log smm.errlog sbcs Sbcs.log sbsc.errlog cleanupd Cleanupd.log cleanupd.errlog udt N/A Udt.errlog startud Startud.log Startud.errlog UniAdmin is best way to review.

34 Unidata Mail Service Mail Service provides bridge to SMTP service by picking up records written to D:\Unidata\Mail and then formatting for delivery using standard SMTP compliant calls. Much more efficient than MAPI. C:\Program Files (x86)\MSGL\Unidata Mail Service\ EzMailServ.exe D:\UniData\ Mail Mail_History Mail_Attachments Mail_Bad

35 Memory Management Shared Memory Overview
How UniData Uses Shared Memory UniData Shared Memory Errors

36 Shared Memory Overview
What is shared memory? Shared Memory is part of Windows Interprocess Communication facilities (IPC) that allows memory outside of process space to be configured and accessed by multiple processes Why does Unidata use it? Allows udt processes to share specific resources (eg - globally cataloged programs) Allows udt processes to expand and contract memory use without having to grow and shrink the actual OS process size Allows all udt processes to quickly access system-wide tables (eg - record locks, dynamic files current modulo) UniData shared memory implementation in Windows uses memory mapped files

37 How Unidata Uses Shared Memory
UniData interacts with shared memory by using system-level calls, UniData services, and UniData configuration parameters (udtconfig) to build its own structures in shared memory. UniData defines shared memory segments that can be attached by UniData processes. The sbcs (shared basic code server) service creates shared memory structures for storing active globally cataloged UniBasic programs. The smm (shared memory manager) service creates shared memory structures for internal tables required by UniData processes. UniData processes request memory for: Buffering UniBasic variables Storing intermediate results Storing a current modulo table for dynamic files

38 How Unidata Uses Shared Memory
The shared memory manager (smm) creates shared memory segments as needed. The size and characteristics of segments smm or the UniData Database Service create are determined by UniData configuration parameters. Whenever UniData starts, it reads the udtconfig file located in D:\Unidata\Include and stores these values in shared memory. smm subdivides each of its segments into global pages, and subdivides each global page into local pages. . smm also creates and maintains internal tables that track the use of the structures it creates. These internal tables are stored in a shared memory structure called CTL.

39 How Unidata Uses Shared Memory
Segments a udt process attaches Sbcs – if running globally cataloged UniBasic programs Smm (ctl) – global control table Smm (shmbuf) – for storing UniBasic variables and ECL intermediate results. Smm (glm) – for storing group (physical) and record locks. Unidata Shared Memory DOC= ADMINNT.PDF (Unidata and Shared Memory) gstt lstt

40 UniData Shared Memory Errors
Run-Time Shared Memory Errors Error when attaching shm (nnn,nnn,nnn), errno=22 ^ SHM_GPAGESZ Error when creating a shared memory segment (size=nnnn), errno=22 ^ SHM_MAX_SIZE No more entries in MI table in LCT – n ^ SHM_LMINENTS Monitor UDT.ERRLOG (Handled by WinVadm) ipclog – created in the UniData account directory where the application was launched Example name: UDT197860c0.ipclog

41 Process Management List/Remove Users Stop/Start Unidata
Pause/Resume Unidata (7.2.x) Troubleshooting SBClient SBClient Users (udt.exe)

42 List & Remove Users D:\Unidata\Bin UniAdmin (Preferred Method)
Listing User Sessions (udt.exe) listuser.exe ECL = !listuser Remove User Sessions Ask Them to Logout (why?) deleteuser.exe Forces a user out of UniData and removes the user’s entry from the internal tables. stopudt.exe Logs a user out of UniData; a current write completes, but subsequent operations for that udt do not take place D:\Unidata\Bin

43 Stop & Start Unidata Correct Procedures Stop Unidata Start Unidata
Stop Ancillary Systems (Discuss) Stop MSG Schema/SQL Services Stop Unidata Mail Service Stop Phantom Scheduler (pstop.bat) Pause Unidata (dbpause) – prevents new sessions Have Users Logout (check with listuser or UniAdm) Remove User Sessions (UniAdmin or deleteuser) Resume Unidata (dbresume) Stop Unidata (stopud –f) Start Unidata Consider Reboot (shutdown /f /r) startud Start Phantom Scheduler (pstart.bat) Restart all Services (Mail, Schema, SQL) Use the UniData startud and stopud commands, respectively, for normal startup and shutdown. These commands start and stop the sbcs, cleanupd, and smm processes, in the correct order. Must have Admin Privileges!

44 Pause & Resume Unidata dbpause - system-level command that enables you to temporarily block updates to the database. You can use this feature to perform maintenance tasks that require UniData to be stopped, such as backing up your data. Now used by WinVadm scripts to provide non-stop (CONOPS) environment. dbpause blocks most updates to the database that are made within UniData. Writes or transactions in process when you issue the dbpause command are completed before dbpause takes effect. UniData blocks updates until you execute the dbresume command. UniData does not block system-level commands, such as COPY or MOVE. In addition, UniData does not block updates to the _HOLD_ file and the _PH_ file, and does not interrupt report printing.

45 Pause & Resume Unidata Pause Status Resuming Processing
The UniData system-level dbpause_status command returns information about the status of dbpause. Resuming Processing To resume processing after issuing the dbpause command, issue the dbresume command. User processes resume, and writes that were blocked when the dbpause command was issued complete.

46 Troubleshooting Stopping Unidata (stopud.exe)
Makes Windows calls to stop the Unidata Database service after: Checking for Administrative Privileges Checking to see if any Unidata processes are still active NOTE: Stopud –f (force) completes current write and terminates (uses deleteuser on all active sessions) Run guide if you have to force stop Stopud makes calls to the following programs in D:\Unidata\bin: Stopsmm (stops shared memory manager) Stopsbcs (stops shared basic code service Stopcleanupd (stops cleanup service)

47 Troubleshooting Terminating a Unidata Process
As discussed, UniAdmin is best (uses Deleteuser) deleteuser <PID> Sends TERM signal; sleeps; checks if PID still exists If TERM signal is not sufficient to remove PID Create entry in D:\Unidata\Bin\deleteuser.log SIGKILL Signal is sent to process (TASKKILL) Stopudt <PID> SIGHUP is caught by udt.exe Udt process send message (IPC) to SMM to release locks and clear license; aborts Windows hard kill (Task Manager) Like stopudt (timing issues with messages). If a process dies or is terminated from hard kill cleanupd and smm work together to remove all shared memory traces! (cleanupd logs)

48 Troubleshooting Starting Unidata (startud)
Startud calls StartUdSrv.exe to start the Unidata Database service after: Checking for Administrative Privileges Copies all *.log & *.errlog files to D:\Unidata\Bin\Saved_Logs Preserves the last 20 iterations Unidata Dabatase Service Then: smm (starts shared memory manager) sbcs (starts shared basic code service cleanupd (starts cleanup service) Each program makes entries in the log files (smm.log, sbcs.log, and cleanupd.log) that identify the system resources used by the services

49 Lock Management Tool Description UniAdmin
Global Lock Manager (GLM) The GLM is an internal software module that is linked into each udt (UDT.EXE) process to manage logical record locks. UniBasic Locks (Records Locks) A series of UniBasic commands enable programmers to set read-only locks and exclusive locks on UniData files and their contents. Group Locks (Managed by Unidata Processes/Services) Protects the integrity of hashed files groups by controlling access. Tool Description UniAdmin GUI Administration Tool (Best tool for locks) ECL Commands LIST.LOCKS, LIST.READU, LIST.QUEUE

50 Global Lock Manager (GLM)
The GLM manages local lock tables for each udt process and a shared global lock table in shared memory, which can be accessed by multiple udt processes. The lock tables are hashed tables containing linked lists, which contain lock nodes. When a udt process locks a record, UniData writes the file name, record ID, and lock mode to both the local lock table and the global lock table. When a udt process requests a lock, UniData first searches that local lock table for the udt to see if that process is holding the lock, then the global lock table to see if another udt process is holding the lock. Put a note item on each slide section pointing to manuals/documents that are key (for future reference)

51 UniBasic Locking UniBasic locks are advisory rather than physical, meaning that they inform other processes that the file or record is in use, rather than explicitly preventing access. We can set exclusive locks or shared (read-only) locks (Mode). Exclusive (U Type) locks are respected by all lock-checking commands and completely prevent access by all other processes. Shared, or read-only, locks (L Type) can be shared by more than one process. A record locked with an L type lock can be accessed for reading by another “L” command, but cannot be accessed by “U” commands. Put a note item on each slide section pointing to manuals/documents that are key (for future reference)

52 Working With Locks If you break out of a process that is running, if a process is killed, or if a system resource is not unlocked by a UniBasic program, locks can remain after they should have been released. If a lock remains set, other users experience difficulty accessing a record, file, or resource. As other processes attempt to access the locked item, message queue congestion can result if the process that set the lock is no longer logged on. The typical manifestations of unneeded locks are: Users cannot perform expected operations on a file or record. Over a lengthy period of time, users receive messages indicating that the file or record is locked. Performance suffers, either because the item that is locked is heavily used or because a message queue has become clogged due to the lock. Batch jobs (Phantoms) attempting to access a locked item fail. Specific symptoms depend on the type of lock and the frequency of usage of the locked item. Put a note item on each slide section pointing to manuals/documents that are key (for future reference)

53 You should use UniAdmin instead!
Working With Locks Listing Locks UniData offers three ECL commands for listing record and file locks, semaphore locks on system resources, and processes waiting to get locks. LIST.READU LIST.LOCKS LIST.QUEUE Clearing Locks UniData includes two ECL commands that enable an administrator with root access to release locks held by other users. SUPERCLEAR.LOCKS SUPERRELEASE Put a note item on each slide section pointing to manuals/documents that are key (for future reference) You should use UniAdmin instead!

54 Group Locks (mglm) Group Locks are managed by Unidata processes and services Protect the integrity of hashed file groups by controlling access Cannot be directly manipulated by UniBasic or ECL commands Processes reading groups set shared locks Update to a group causes escalation of shared lock to an exclusive lock (blocks access) Processes scanning a file (LIST, SORT, SELECT) will wait (HANG) until an exclusive lock is released In order to resolve deadlock situations, cleanupd will spawn upl_deadlockd every so often to resolve problems (defaults to every 20 seconds) Put a note item on each slide section pointing to manuals/documents that are key (for future reference)

55 Shadow Copy (Volume Snapshot Service or VSS)
Backup & Recovery Shadow Copy (Volume Snapshot Service or VSS) Backup General Backup Procedures/Issues Locked Files Referential Integrity Stopping/Starting Unidata (can now pause/resume) Backup Hardware/Software Recovery Complete (Unidata will have to be re-licensed) An Entire Account (Your Database) Individual Files (May need more than one file) Records (Must be copied using Unidata tools – Binary) vssadmin list providers vssadmin create shadow /for=c: Tool Description listuser Lists all current UniData users. Stopud/Startud Stop/Start Unidata Cleanly Dbpause/dbresume Suspend/Continue Logins deleteuser Forces a user out of UniData

56 Backup & Recovery Backup (Synchronization is key) Stop Services First
MSGSQLInterface MSGSchemaInterface Remove UDAPI Sessions (Should be gone) Server Slaves Remove Phantom Sessions (Schema) Recovery (Synchronization is key) Élan Server SQL Server UniObjects

57 System Administration Fundamentals Part 2
File System Management Maintenance Tools & Automation Monitoring & Tuning

58 File System Management
File Types File Maintenance File Corruption

59 File Types Static Hashed (F) – CLIENTS (And D_CLIENTS)
Single Windows (Binary) file Number of group (MODULO) does not change Overflow blocks are appended to the file as needed Dynamic Hashed (F) – SU.SUB Dynamic files consist of a FILENAME directory with part files Datnnn (Data Part) Overnnn (Overflow Part) Idxnnn (Index Part) Modulo can grow (SPLIT) or shrink (MERGE) as needed Overflow blocks live in a separate file (OVERNNN) Directory (DIR) – Store UniBasic Code – SQL.BP Operating system directories Performance is based on how OS handles files (_HOLD_) Alternate Index File (B+ tree) – X_CLIENTS B+ tree file designed for quick SELECT and UniBasic index browsing

60 File Types Hashed files are binary files that cannot be viewed at the operating system level or read by text editors external to UniData. Each UniData hashed file consists of a file header and one or more groups (MODULO) of data. Each data group contains the following structure: A fixed-length group header A pointer array Record IDs Data A record key is assigned to a group in the file according to a hashing algorithm. Then the precise location of the data is stored in the header of that group. The goal of hashing is to make searching for data more efficient by eliminating the need to search an entire file for a record. In a hashed file, UniData searches only the group where the primary key of the record was assigned.

61 Security Standards Compliance
Payment Card Industry Data Security Standard The standard was created to help organizations that process card payments it is intended to prevent credit card fraud through increased controls around data and its exposure to compromise. The standard applies to all organizations which hold, process, or pass cardholder information from any card branded with the logo of one of the card brands. Currently undergoing compliance certification with “Trustwave” - Done Have implemented new U2 features ADE (Automatic Data Encryption) 256 AES Master Key Controls Clients will manage key storage/control ECP Module Setup and Configuration Backup and Recovery SSL (Secure Socket Layer) – No clear-text SSO (Single Sign-On) – Rotate Passwords Élan 7.5 PCI

62 Hashed File Structure CREATE.FILE CLIENTS 1001,8 STATIC
File divided into groups for storing data Modulo = Number of Groups Block Size = Size of each group (and overflow block) Block Size can be 1k > 16k (multiples of 1k) Suggest (2,4,8,16) – Align with IO buffer sizes Hashing algorithm uses modulo and record key to determine record location in file Unidata reads 1 block size at the calculated offset (Group) Retrieves record key and data location from that block (if not in overflow) File Header block contains file characteristics Modulo, Block Size, Hash Type…..

63 Hashed File Structure Group Layout
Key Space and Data Space are separate Keys are placed at the beginning of the group Data is placed at the end of the group Level1 Overflow As groups fill, record data is moved to overflow blocks All keys remain in primary group Large records trigger level1 overflow (block size >16k) Performance with some level1 overflow is acceptable Level2 Overflow Primary group is filled with Keys Some Keys and all Data records reside in overflow blocks Triggers disk reads of non-continuous blocks Performance will be very poor! – Resize regularly! Includes offset information (Address) to data

64 Hashed File Structure Group Layout Benefits
Operations that require only the record key are optimized SELECT of file based on keys COUNT of records in file Read/Write Operations After hashing to primary group, only that group needs to be read and searched to find the key (assuming no level2 overflow) Only Have to scan keys – not keys and data Level1 overflow does require a secondary read to return data, but offset information results in direct read No block scanning required at this point Keys at front of group

65 Unidata Overflow Handling
Static Files When groups fill, overflow blocks are appended to the file Blocks are linked to a specific group If records are removed from the group, the blocks remain linked to the group, possibly unused Overflow blocks will be reused by that group, but are not available to any other group needing overflow blocks Unused overflow blocks are released by CLEAR.FILE or resize operations (memresize) Dynamic files Overflow blocks are stored in separate file from primary groups – overnnn All overflow blocks for a group will be in one overnnn file Blocks are linked to a group, but if are no longer needed, are available to other groups

66 File Maintenance Monitor Database Real-time errors
D:\Unidata\Bin\udt.errlog WinVadm monitors continuously and s you when errors are detected Level2 Overflow D:\Unidata\Bin\checkover.exe Level1 Overflow D:\Unidata\Bin\Guide.exe (Lets look at the manual) Resize Files D:\Unidata\bin\memresize.exe (lets look at the manual) WinVadm will do this weekly (EzResize) – New tools coming Defragment Drives (Diskeeper) WinVadm runs defrag command

67 File Corruption File Corruption = damage to a Unidata files internal structures Overflow offset points to non-existent block File header block is damaged Record length is wrong Keys hashed into wrong group Logical data integrity issues Interrupted UDT process – not all transactions have been applied Session dies Session killed Disk Errors Monitor event logs Chkdsk /F You must run “guide” in all accounts afterwards!

68 File Corruption What can cause physical corruption of Unidata files?
Killing a process with Task Manager (or similar tool) System Panic or Power Failure Hardware Problems Memory, Disk, File System Cache Failure (battery backed?) Memory to Physical Disk Platters (even drivers) Restoring from backup while data files are being used Overflowed files are more prone to corruption from interruptions as multiple disk writes are required to complete single Unidata write.

69 File Corruption Detecting Unidata file corruption
Corruption encountered by udt process at run-time is recorded in D:\Unidata\Bin\udt.errlog WinVadm scripts continuously monitor this file (will ) D:\Unidata\Bin\guide.exe Reports file corruption in Unidata hashed files GUIDE_ERRORS.LIS D:\Unidata\Bin\guide_ndx.exe Reports physical/logical corruption in index files New version of WinVadm will run this each night (Due in May)

70 File Corruption Fixing Unidata file corruption
D:\Unidata\bin\fixfile –f –D:DUMPFILE Repairs multiple groups in one or more files Used GUIDE_FIXUP.DAT as default input file -D:DUMPFILE = Map of dumped records (what is being repaired) Why you should never try this without MSG staff! Procedure to follow when you suspect/detect errors Have all users logout of the system Stop Unidata (see previous procedures) Start “guide” in account where corruption is suspected/detected Open Helpdesk ticket with MSG (High Priority)

71 File Corruption Fixing errors in Alternate Index files (X_FILENAME)
There are no tools comparable to “fixfile” To fix, you must delete and recreate Index file (ECL Commands) DELETE.INDEX filename ALL You must use the ALL keyword, doing each index individually does not delete the entire index file Corruption can remain if not completely deleted CREATE.INDEX filename DictID (Must do each index individually) BUILD.INDEX Filename ALL You cannot run BUILD.INDEX on an active system The data file is locked until BUILD.INDEX completes

72 Maintenance Tools & Automation
Why should you Automate? What can you Automate? How can you Automate? Putting it all together (WinVadm)

73 Why should you automate?
Efficiency: Once a task is automated, you have a one click solution to a complex problem. Consistency: Exactly the same process repeated, producing reliable results. Dependability: Automating Windows removes the potential for human error. Productivity: Relieves people of repetitive tasks, freeing time for more stimulating work Schedule tasks to run at night and off-peak times to ease the drain on resources during the day Regular tasks can be scheduled to run automatically to reduce workload

74 What can you automate? Data Archive & Backups
Stop, start, and configure services Log Checking (Scanning) Directory & File Cleanup Managing configuration files Server Event Logs (Backup & Clear) Database Maintenance Resize (Takes hours to do manually) Defrag Threshold Events (Disk 90% full) Monitor Performance Information Distribution (Job Success/Failure)

75 How can you automate? Define Strategy (What you need to automate)
Database Administration/Maintenance Daily/Weekly/Monthly Analyze Strategy (Select tools, utilities, and create scripts) Shell Programming Scripting Software/Tools Commercial Applications (MOM/WUG) Execute Strategy (Schedule scripts to run) Windows Task Scheduler (Free) Start->Control Panel>Scheduled Tasks (GUI) Schtasks (Command Line) Batch Scripts Vbscript (WMI) Powershell Perl

76 Putting it all together!
Daily Tasks EOD (End Of Day) – 10pm Service Analysis Event Analysis Performance Analysis Account Analysis Backups Stop SQL Services (For now) Suspend Unidata (DBPause) Shadow Copy (API) Create Mount Point (API) Resume Unidata (DBResume) Start SQL Services RoboCopy (To F: - iSCSI SAN) Remove Mount Point Just my scripts – your requirements may differ!

77 Putting it all together!
Weekly Tasks – Sunday for MSG servers Resize Files in all Accounts Unidata must be running, but users must be off Remove Sessions Stop Telnet Service (uninetd) Block 31438 Ezresize.vbs Defrag Drives C:\Windows\System32\Defrag.exe (Free) UltraDefrag (Open Source) Reboot Server (Cleanup)

78 Monitoring & Tuning Concepts & Tools Hardware Decisions
Windows Monitoring & Tuning Unidata DBMS Monitoring & Tuning

79 Concepts OLTP vs. OLAP (DSS) OLTP (Online Transaction Processing)
High percentage of simple, short transactions from a large number of users generating a large amount of random IO requests. Physical disks spend a measurable percentage of time seeking data from various different parts of the disk for read or write purposes. OLAP (Online Analytical Processing) or (Decision Support System) Stores data for reporting applications which typically have a smaller number of users generating much larger queries, typically resulting in Sequential I/O. Physical disks spend most of their time scanning a range of data clustered together in the same part of the disk U2 Relational Database (OLTP) 40% of Disk IO is transaction WRITE 60% of Disk IO is READS for joins All IO is Random Sequential IO does not exist in U2 Compressed Data U2 generate millions of 2k – 8k disk reads/writes U2 places the burden of expanding the record on the CPU Much more efficient use of resources! U2 is basically a string manipulation engine

80 Tools Windows Event Logs provide early warning of issues Task Manager
Perfmon (know your counters,) – I have provided Unidata Logs provide early warning of issues D:\Unidata\Bin\*.log and *.errlog D:\Unidata\Include\loglevelconfig UniAdmin (interface to specific Unidata counters) Udtmon_remote Udtdiag script (diagnose issues) Provides a snapshot of the system while an issue is occurring Dumps Unidata shared memory in re-playable format Udtconfig (settings must match resources available in OS) PORT.STATUS (FILEMAP & CALL.STACK)

81 Hardware Decisions CPU Memory Disk Subsystem Network Cards

82 Hardware Decisions Processor Cards (Not really CPU’s anymore)
More is better than faster! 64bit is a requirement now U2 may not see benefit of more cores Less cores – more physical processors L3 Cache is shared for sure 1 processor per 25 users (1 for system) – no matter the cores It’s all about the cache The more the better

83 Hardware Decisions Windows 2008 R2 (64bit) Memory Order memory is used
Disk Cache – This is our gold mine! Reduces number of READS Most used data is hiked & higher into memory, never paged out When memory is sized properly Change system into Disk writing machine Will make SELECTS up to 10 times faster Local variables for each user Why we used globally cataloged code, is a waste of memory Globally cataloged programs Unidata run engine Operating System Lab benchmarks have proven servers built with memory sized to 25% of the Dynamic Data Set (the data that is changing),the server will literally turn into a disk writing machine. SELECT and reports have run in 1/10ththe time Literally, the majority of the data that is used is hiked higher and higher into memory.

84 Hardware Decisions Disk Subsystem:
Random IO requires more disk seeks and IOPS Use small transfer/stripe/block sizes Number of disks is important Use disks with fast seek times Don’t double cache reads, use disk cache for writes! Drives (15,000 RPM) We now use SSD (Let’s discuss) Configuration: O/S (C:\) Raid 1 – 2 x 15,000 RPM Drives Swap performs well on Raid1 Database (D:\) Raid to 6 x 15,000 RPM Drives Logs/Temp (L:\) SQL.MAPJRNL SQL.UDTJRNL TMP (udtconfig) – UDTSORT uses

85 Hardware Decisions Network Cards Not likely to see network bottleneck
Client/Server (Telnet) OLTP Single 1gb Adapter will usually handle the load Speed is more important than capacity Stability is key (State based connections) If you are using UniObject interface, add a second adapter and isolate traffic Consider Teaming!

86 Hardware Decisions Summary:
Global Catalog Programs – in Cache (L1 is 50/50 code/data) – Spatial and Temporal locality difficult for Unidata – so more is better. L3 cache is good since it is shared by all cores it works well for items in global catalog space, bad for data sets as constant scavenging (FSB Contention). 64k Chunking for Stripes is best (no less than 16k will cause issues) – get what we need into cache. FBWC – Flash Backed Write Cache – uses Super Capacitors 1024mb – much better than BBWC (which dumps after 48 hours anyway).

87 Windows Monitoring & Tuning
Event Logs (you would be surprised how many people ignore) Perfmon (Separate Slide) Tuning It is very important to try to match the block/stripe/segment sizes to a multiple of the I/O size for workloads (proportion of writes). Windows AUS is 4kb Raid stripe is best at 64kb or less (default is usually 128) – I like 16kb Random I/O workload occurs in very small blocks (2 kilobyte though 8 kilobyte), and each I/O to/from the storage device requires a seek and an I/O because we are reading only 1 file system block at a time (OLTP). Caching file system blocks into memory can make a big difference to application performance, since we can avoid many of those expensive and slow I/O's. We want to avoid Read/Modify/Write cycles. Disable pre fetch and read ahead, or limit read ahead to the size of each I/O.

88 Windows Monitoring & Tuning
System Adjust for best performance (Visual effects Tab) Processor Scheduling – Adjust for programs (not background services) Memory Usage – Adjust for System Cache Discuss windows 2008 and dynamic cache manager Virtual Memory (Fixed Page File Size) – 1.5 to 2 times Memory (Discuss) DEP – Essential Windows services and programs only Automatic Updates (Careful) – Lets discuss Local Security Policy Idle timeout (90 Minutes) – is 15 by default – Set Keyboard Timeout Instead Printers/Faxes Log Spooler Error events only (Will overwhelm logs otherwise) Virus Protection Disable real-time scan (interferes with IO) Ignore D:\Unidata and D:\IBM Lets discuss running other applications on this server!

89 Windows Monitoring & Tuning
Performance Monitoring (Perfmon) IO Performance IOPS (130/Sec for 15,000rpm Disks) Disk Reads/Sec Disk Writes/Sec Latency (Look at with Queue and IO Size) – Want 5ms-20ms Average Disk sec/Read Average Disk sec/Write Disk Throughput (Bandwidth) Disk Read Bytes/Sec Disk Writes Bytes/sec IO Size (Usually 2k-8k) – Helps us determine config Average Disk Bytes/Read Average Disk Bytes/Write IO Queue (IO Latency) - <= 2 per physical disk Average Disk Queue Length Current Disk Queue Length Cache hit ratio!

90 Unidata Monitoring/Tuning
File Systems (checkover/guide) Logs (D:\Unidata\bin\*.log and *.errlog) Tune Use the PHANTOM scheduler to batch jobs at night! Use Globally Cataloged code (sbcs) Look for private segments (GSTT/LSTT) Resize Regularly Dynamic defaults may not be correct for your DB Split/Merge Defrag Tune Unidata to use all the available resources UDTCONFIG (Lets look at udtconfig) Open file handles (rotating file pool) NFILES max in Windows 2008 is 2035 Shared Memory PORT.STATUS is key to determining PID Files and Stack activity!

Download ppt "User Group Conference 2012 Advanced Tech Track 1."

Similar presentations

Ads by Google