Presentation is loading. Please wait.

Presentation is loading. Please wait.

1 © 2008 Julian Dyke juliandyke.com Automatic Storage Management Julian Dyke Independent Consultant Web Version - December 2008.

Similar presentations


Presentation on theme: "1 © 2008 Julian Dyke juliandyke.com Automatic Storage Management Julian Dyke Independent Consultant Web Version - December 2008."— Presentation transcript:

1 1 © 2008 Julian Dyke juliandyke.com Automatic Storage Management Julian Dyke Independent Consultant Web Version - December 2008

2 © 2008 Julian Dyke juliandyke.com 2 Objectives 1. Understand how Oracle database files are stored in ASM 2. Calculate how long ASM rebalance operations will take

3 © 2008 Julian Dyke juliandyke.com 3 Agenda ASM Instances ASM Disk Groups Metadata Extent Distribution Rebalancing Redundancy

4 © 2008 Julian Dyke juliandyke.com 4 ASM Instances

5 © 2008 Julian Dyke juliandyke.com 5 ASM Single Instance Architecture Oracle Clusterware ASM Instance RDBMS Instance Server OCSSD Daemon Only Dedicated Storage

6 © 2008 Julian Dyke juliandyke.com 6 ASM Single Instance Background Processes PMONSMONCKPTLGWRDBW0VKTMPSP0MMANRBALGMONX000DIAGDIA0 Fixed Area Variable Area ASM Cache Oracle 11.1

7 © 2008 Julian Dyke juliandyke.com 7 ASM RAC Architecture Oracle Clusterware ASM Instance RDBMS Instance Node 1 Oracle Clusterware ASM Instance RDBMS Instance Node 2 Oracle Clusterware ASM Instance RDBMS Instance Node 3 Oracle Clusterware ASM Instance RDBMS Instance Node 4 Storage Network Private Network Public Network Shared Storage

8 © 2008 Julian Dyke juliandyke.com 8 ASM RAC Architecture PROD1TEST1 CLUSTERWARE +ASM1 PROD2TEST2 CLUSTERWARE +ASM2 PROD TEST Clusterware ASM Instances RDBMS Instances Database Files

9 © 2008 Julian Dyke juliandyke.com 9 ASM RAC Instance Background Processes PMONSMONCKPTLGWRDBW0VKTMPSP0MMANRBALGMONX000DIAGDIA0 Fixed Area Variable Area ASM Cache Oracle 11.1 LMONLMS0LMD0LCK0DIAGMARKKATE

10 © 2008 Julian Dyke juliandyke.com 10 ASM Disk Groups

11 © 2008 Julian Dyke juliandyke.com 11 ASM Disk Groups and Disks Disk Group 1 Disk Group 2 Disk Group 3 Disk 1Disk 4Disk 2Disk 6Disk 5Disk 7Disk 3

12 © 2008 Julian Dyke juliandyke.com 12 ASM Disk Groups and Disks Disk Group 1 Disk 1Disk 4 Disk Group 2 Disk 2Disk 5Disk 6Disk 7Disk 3 Disk Group 3

13 © 2008 Julian Dyke juliandyke.com 13 ASM Disk Groups, Disks and Database Files Disk Group 1 Disk 1Disk 4 Disk Group 2 Disk 2Disk 5Disk 6Disk 7 Disk 3 Disk Group 3 File 1 File 2 File 3 File 4 File 5 File 6

14 © 2008 Julian Dyke juliandyke.com 14 File Extents versus Allocation Units File Extent Logical unit of ASM file Map to allocation units One to many mapping Allocation Unit Physical unit of ASM disk Oracle 10.2 and below Always 1MB Can be increased using _asm_ausize Oracle 11.1 and above Variable size 1MB, 2MB,4MB, 8MB, 16MB, 32MB, 64MB

15 © 2008 Julian Dyke juliandyke.com 15 X$KFFXP Maps file extents to allocation units Only populated in ASM instance Columns include Column NameDescription GROUP_KFFXPDisk Group Number NUMBER_KFFXPFile Number COMPOUND_KFFXPDisk Group Number || File Number INCARN_KFFXPIncarnation Number PXN_KFFXPPhysical Extent Number (within file) XNUM_KFFXPLogical Extent Number (within file) LXN_KFFXP0=primary, 1=first mirror, 2=second mirror DISK_KFFXPDisk Number AU_KFFXPAllocation Unit Number (within disk) SIZE_KFFXPSize (# allocation units)

16 © 2008 Julian Dyke juliandyke.com 16 ASM Metadata Metadata is stored in first 256 files in ASM disk group Space is initially allocated when disk group is created Can be subsequently extended Metadata allocation units are divided into blocks Each block is 4096 bytes Block size specifed using _asm_blksize Metadata files include File#Description 0Metadata Header 1File Directory 2Disk Directory 3Active Change Directory 4Continuing Operations Directory 5Template Directory 6Alias Directory 9Attribute directory (optional) 12Staleness registry (optional)

17 © 2008 Julian Dyke juliandyke.com 17 ASM Metadata Active Change Directory Continuing Operations Directory Template Directory Alias Directory Disk Directory File Directory Partner Status Table Metadata Header Free Space Table Allocation Table Disk Header Metadata Header

18 © 2008 Julian Dyke juliandyke.com 18 ASM Metadata Initial Allocation (Single Instance) File#AUDescription# AUs 00Disk Header, Free Space Table, Allocation Table1 1Partner Status Table1 1File Directory1 2Disk Directory1 3Active Change Directory42 4Continuing Operations Directory2 5Template Directory1 6Alias Directory1 Active Change Directory Records changes to metadata Used during recovery of instance or operation failures Continuing Operations Directory Maintains state of active operations

19 © 2008 Julian Dyke juliandyke.com 19 ASM Metadata Block Types TypeDescription 1KFBTYP_DISKHEAD 2KFBTYP_FREESPC 3KFBTYP_ALLOCTBL 4KFBTYP_FILEDIR 5KFBTYP_LISTHEAD 6KFBTYP_DISKDIR 7KFBTYP_ACDC 8KFBTYP_CHNGDIR 9KFBTYP_CODBGO 10KFBTYP_TMPLTDIR 11KFBTYP_ALIASDIR 12KFBTYP_INDIRECT TypeDescription 13KFBTYP_PST_NONE 14KFBTYP_HASHNODE 15KFBTYP_COD_RBO 16KFBTYP_COD_DATA 17KFBTYP_PST_META 18KFBTYP_PST_DTA 19KFBTYP_HBEAT 20KFBTYP_SR 21KFBTYP_STALEDIR 22KFBTYP_VOLUMEDIR 23KFBTYP_ATTRDIR

20 © 2008 Julian Dyke juliandyke.com 20 KFED Utility In Oracle 10.2 and above the kfed utility can be used to inspect and edit the contents of ASM blocks ~]$ $ORACLE_HOME/bin/kfed -h as/mlibASM Library [asmlib='lib'] aun/umAU number to examine or update [AUNUM=number] aus/zAllocation Unit size in bytes [AUSZ=number] blkn/umBlock number to examine or update [BLKNUM=number] blks/zMetadata block size in bytes [BLKSZ=number] ch/ksumUpdate checksum before each write [CHKSUM=YES/NO] cn/tCount of AUs to process [CNT=number] d/evASM device to examine or update [DEV=string] o/pKFED operation type [OP=READ/WRITE/MERGE/NEW/FORM/FIND/STRUCT] p/rovnmName for provisioning purposes [PROVNM=string] s/eekAU number to seek to [SEEK=number] te/xtFile name for translated block text [TEXT=string] ty/peASM metadata block type number [TYPE=number] This utility should only be used under the guidance of Oracle Support

21 © 2008 Julian Dyke juliandyke.com 21 KFED Utility For example to dump blocks in aliases directory in DISKGROUP1 Find group number SELECT disk_kffxp, au_kffxp FROM x$kffxp WHERE group_kffxp = 1 AND number_kffxp = 6 AND lxn_kffxp = 0; DiskAllocation Unit 049 Find disk name SELECT group_number FROM v$asm_diskgroup WHERE name = 'DISKGROUP1'; Alias directory is stored in file number 6 SELECT path FROM v$asm_disk WHERE group_number = 1 AND disk_number = 0; Path /dev/oracleasm/disks/VOL1

22 © 2008 Julian Dyke juliandyke.com 22 KFED Utility Example (continued) Allocation unit is 1MB Block size is 4096 Therefore there are 256 blocks per allocation unit Starting block offset = 256 * 49 = for (( f = ; f blk${f} done

23 © 2008 Julian Dyke juliandyke.com 23 Extent Distribution

24 © 2008 Julian Dyke juliandyke.com 24 Extent Distribution Creating a disk group: CREATE DISKGROUP diskgroup1 EXTERNAL REDUNDANCY DISK '/dev/oracleasm/disks/VOL1'; Dropping a disk group: DROP DISKGROUP diskgroup1 INCLUDING CONTENTS

25 © 2008 Julian Dyke juliandyke.com 25 Extent Distribution 1 disk Disk 0 Metadata Data

26 © 2008 Julian Dyke juliandyke.com 26 Extent Distribution 2 disks Disk 0Disk 1 Metadata MetadataData

27 © 2008 Julian Dyke juliandyke.com 27 Extent Distribution 4 disks Disk 0Disk 1Disk 2Disk 3 Metadata MetadataData

28 © 2008 Julian Dyke juliandyke.com 28 Extent Distribution 1 large disk - 1 small disk Disk 0Disk 1 Metadata MetadataData

29 © 2008 Julian Dyke juliandyke.com 29 Extent Distribution 1 large disk - 3 small disks Disk 1Disk Metadata Disk 0Disk MetadataData

30 © 2008 Julian Dyke juliandyke.com 30 Extent Distribution 2 large disks - 2 small disks Disk 1Disk Metadata Disk 0Disk MetadataData

31 © 2008 Julian Dyke juliandyke.com 31 Rebalancing

32 © 2008 Julian Dyke juliandyke.com 32 Extent Distribution Adding a disk: ALTER DISKGROUP diskgroup1 ADD DISK '/dev/oracleasm/disks/VOL2' REBALANCE POWER 0; Dropping a disk: ALTER DISKGROUP diskgroup1 DROP DISK 'DISKGROUP1_0002' REBALANCE POWER 0; Rebalancing a disk group: ALTER DISKGROUP diskgroup1 REBALANCE POWER 1;

33 © 2008 Julian Dyke juliandyke.com 33 Rebalancing Adding disks - 1 disks to 2 disks Disk 0 Disk STOP MetadataData

34 © 2008 Julian Dyke juliandyke.com 34 Rebalancing Adding disks - 1 disks to 4 disks Disk 0Disk 1Disk 2Disk STOP MetadataData

35 © 2008 Julian Dyke juliandyke.com 35 Rebalancing Adding disks - 2 disks to 3 disks Disk 0Disk 1Disk MetadataData

36 © 2008 Julian Dyke juliandyke.com 36 Rebalancing Adding disks - 2 disks to 4 disks Disk 0Disk 1Disk 2Disk STOP MetadataData

37 © 2008 Julian Dyke juliandyke.com 37 Rebalancing Dropping disks - 3 disks to 1 disk Disk 0Disk 1Disk STOP MetadataData

38 © 2008 Julian Dyke juliandyke.com 38 Rebalancing Moving disks - 2 disks to 2 disks Disk 0Disk 1Disk STOP MetadataData

39 © 2008 Julian Dyke juliandyke.com 39 Rebalancing V$ASM_OPERATION Contains details of ongoing rebalance operations Column NameData Type GROUP_NUMBERNUMBER OPERATIONCHAR(5) STATEVARCHAR2(4) POWERNUMBER ACTUALNUMBER SOFARNUMBER EST_WORKNUMBER EST_RATENUMBER EST_MINUTENUMBER ERROR_CODEVARCHAR2(44) Estimate of remaining time

40 © 2008 Julian Dyke juliandyke.com 40 Rebalancing Power Limit Power limit can be 0 to 11 0 disables rebalance operation 1 to 11 specifies number of ARBn background processes used for rebalance In Oracle 10.2 RBAL manages rebalance operation Each ARBn background process is allocated a range of 128 allocation units to rebalance When complete another range is requested AD lock is taken while an allocation unit is being rebalanced Rebalance operations take much longer than theoretically necessary. Possible reasons include: Locking GES updates with other ASM instances Updates to RDBMS instance

41 © 2008 Julian Dyke juliandyke.com 41 Rebalancing Summary EST_MINUTES column of V$ASM_OPERATION is reasonably accurate Allow a few minutes for SAN cache to stabilize Check regularly for changes to estimate ASM rebalance operations do not affect workload Locks are only taken briefly Lock mechanism has changed in Oracle 11.1 SAN cache and I/O performance will be affected In Oracle 10.2 rebalancing is fastest if Other ASM instances are shutdown RDBMS instance is shutdown Estimated completion time will be affected by: Use of SAN cache and I/O by rest of workload Rate of change by applications to blocks in ASM files being rebalanced

42 © 2008 Julian Dyke juliandyke.com 42 Redundancy

43 © 2008 Julian Dyke juliandyke.com 43 Redundancy ASM Supports three levels of redundancy External Redundancy Implemented externally using storage layer Most common configuration in production Normal Redundancy Two copies of each extent maintained in separate failure groups Used with extended clusters Used occasionally in production e.g. CERN Increases CPU overhead on servers High Redundancy Three copies of each extent maintained in separate failure groups Very rare in production

44 © 2008 Julian Dyke juliandyke.com 44 ASM Failure Groups - External Redundancy Disk Group Disk 1Disk 2Disk 3

45 © 2008 Julian Dyke juliandyke.com 45 ASM Failure Groups - Normal Redundancy Failure Group 1FailureGroup 2 Disk Group Disk 1Disk 2Disk 3Disk 4Disk 5Disk 6

46 © 2008 Julian Dyke juliandyke.com 46 ASM Failure Groups - High Redundancy Failure Group 1 Disk Group Disk 1Disk 2 Failure Group 2 Disk 1Disk 2 Failure Group 3 Disk 1Disk 2

47 © 2008 Julian Dyke juliandyke.com 47 Normal Redundancy 1 Disk Per Failure Group Disk 0Disk Failure Group Failure Group 2 PrimarySecondary Metadata Data PrimarySecondary

48 © 2008 Julian Dyke juliandyke.com 48 Normal Redundancy 2 Disks per Failure Group Disk 0Disk 1Disk 2Disk Failure Group 1 Failure Group PrimarySecondary Metadata Data PrimarySecondary

49 © 2008 Julian Dyke juliandyke.com 49 High Redundancy 1 Disk per Failure Group Disk 0Disk 1Disk Failure Group 1 00 Failure Group 2Failure Group PrimarySecondary Tertiary Metadata Data PrimarySecondary Tertiary

50 © 2008 Julian Dyke juliandyke.com 50 References Oracle Automatic Storage Management (Oracle Press) Nitin Vengurlekar Murali Vallath Rich Long What ASM and ZFS Can Do For You Jason Arneil - Nominet A Closer Look Inside Oracle ASM Luca Canali - CERN Implementing ASM Without HW Raid Luca Canali - CERN

51 © 2008 Julian Dyke juliandyke.com 51 Thank you for listening


Download ppt "1 © 2008 Julian Dyke juliandyke.com Automatic Storage Management Julian Dyke Independent Consultant Web Version - December 2008."

Similar presentations


Ads by Google