File Organizations Sept. 2012Yangjun Chen ACS-3902/31 Outline: File Organization Hardware Description of Disk Devices Buffering of Blocks File Records on Disk Operations on Files Files of Unordered Records (Heap Files) Files of Ordered Records (Sorted Files)

File Organizations Sept. 2012Yangjun Chen ACS-3902/32 Hardware Description of Disk Devices -Magnetic disks: used for storing amounts of data data: bits and bytes (characters) -Two kinds of disks: floppy disks and hard disks floppy disk: 400 Kbytes Mbytes hard disk: hundred Mbytes - Gbytes -single-sided or double-sided -disk pack: disks assembled together -track, cylinder and sector

File Organizations Sept. 2012Yangjun Chen ACS-3902/33... actuator track cylinder sector (arc of a track)

File Organizations Sept. 2012Yangjun Chen ACS-3902/34 - Sector: Each track is divided into sectors by the disk management system. It is hard-coded and cannot be changed. Not all disks have their tracks divided into sectors. - Block: Each track is divided into blocks by the operating system during disk formatting (or initialization). Block size is fixed during formatting and cannot be changed dynamically. Block size: bytes. A disk with hard-coded sectors has the sectors subdivided into blocks. Blocks are separated by fixed-size interlock gaps.

File Organizations Sept. 2012Yangjun Chen ACS-3902/35 - Data Transfer (Read/Write) data transfer unit: a block or a cluster (several contiguous blocks) - Hardware address: <surface number, track number, block number> - Buffer: a contiguous area reserved in main memory that holds a block.

File Organizations Sept. 2012Yangjun Chen ACS-3902/36 Buffering of Blocks -Several buffers can be reserved in main memory. While one buffer is being read or written, the CPU can process data in the other buffers. -concurrency: interleaved or in parallel time t1t1 t2t2 t3t3 t4t4 A A B B C D

File Organizations Sept. 2012Yangjun Chen ACS-3902/37 -Double Buffering The CPU can start processing a block once its transfer to main memory is completed; at the same time the disk I/O processor can be reading and transferring the next block into a different buffer. i fill A i+1 fill B i+2 fill A i+3 fill B i+4 fill A i process A i+1 process B i+2 process A i+3 process B i+4 process A time Disk block: I/O: Disk block: Processing:

File Organizations Sept. 2012Yangjun Chen ACS-3902/38 File Records on Disk -Record: a collection of related data values or items, where each value is formed of one or more bytes and corresponds to a particular field of the record. -Record type: a collection of field names and their corresponding data types. A data type, associated with each field, specifies the type of values a field can take. Example: struct employee { char name[30]; char SSN[9]; intsalary; int jobcode; char department[20]; };

File Organizations Sept. 2012Yangjun Chen ACS-3902/39 -Fixed-length records and variable-length records Fixed-length records: NameSSN Salary JobCode Department Hire-Date Variable-length records: Smith, John xxxx Computer name SSN salary JobCode department NAME=Smith, John SSN= DEPATMENT=Computer 

File Organizations Sept. 2012Yangjun Chen ACS-3902/310 -Record blocking: spanned versus unspanned records The records of a file must be allocated to disk blocks because a block is the unit of data transfer between disk and memory. Unspanned records: record 1 record 2record 3 unused record 4 record 5record 6 Spanned records: block i block i+1 record 1 record 2record 3 block i block i+1 record 4 P record 4 (rest) record 5record 6 record 7 P

File Organizations Sept. 2012Yangjun Chen ACS-3902/311 -Allocating file blocks on disk Contiguous allocation: file block 1 file block 2 …... disk track Linked allocation: file block 1 file block 2 file block 3 …... -File header: disk addresses of blocks, record format description (field length, order of fields, field type code, separator characters, record type code, …)

File Organizations Sept. 2012Yangjun Chen ACS-3902/312 Operations on Files operations general OP. retrieval OP. update OP. combined OP. record-at-a-time set-at-a-time

File Organizations Sept. 2012Yangjun Chen ACS-3902/313 -general operations Open: Prepares the file for reading or writing. Allocates appropriate buffers (typically at least two) to hold file blocks from disk, and retrieves the file header. Sets the file pointer to the beginning of the file. file header buffer file pointer file on disk: main memory

File Organizations Sept. 2012Yangjun Chen ACS-3902/314 Reset: Sets the file pointer of an open file to the beginning of the file. Close: Completes the file access by releasing the buffers and performing any other needed cleanup operation (e.g., cleanup the information of the file header which is maintained in main memory).

File Organizations Sept. 2012Yangjun Chen ACS-3902/315 -retrieval operations Find (or Locate): Searches for the first record that satisfies a search a search condition. Transfers the block containing that record into a main memory buffer (if it not already there). The file pointer points to the record in the buffer and it becomes the current record. search conditions: SSN = ‘ ’ DEPARTMENT = ‘Research’ SALARY > 30000

File Organizations Sept. 2012Yangjun Chen ACS-3902/316 -retrieval operations (continued) Read (or get): Copies the current record from the buffer to a program variable in the user program. This command may also advance the current record pointer to the next record in the file, which may necessitate reading the next file block from disk. Find Next: Searches for the next record in the file that satisfies the search condition. Transfers the block containing that record into a main memory buffer (if it is not already there). The record is located in the buffer and becomes the current record.

File Organizations Sept. 2012Yangjun Chen ACS-3902/317 -update operations Delete: Deletes the current record and (eventually) updates the file on disk to reflect the deletion. Modify: Modifies some field values for the current record and (eventually) updates the file on disk to reflect the modification. Insert: Inserts a new record in the file by locating the block where the record is to be inserted, transferring that block into a main memory buffer (if it is not already there), writing the record into the buffer, and (eventually) writing the buffer to disk to reflect the insertion.

File Organizations Sept. 2012Yangjun Chen ACS-3902/318 -combined operations Scan (a combination of Find, FindNext, and Read): If the file has just been opened or reset, Scan returns the first record that satisfies the search condition; otherwise, it returns the next record. file header buffer file pointer file on disk: main memory current record next record If the file is just opened or reset, then execute Find(); Then Find-next().

File Organizations Sept. 2012Yangjun Chen ACS-3902/319 set-at-a-time: FindAll: Locates all the records in the file that satisfies a search condition. FindOrdered: Retrieves all the records in the file in some specified order (e.g., descendent key values). Reorganization: Starts the reorganization process.

File Organizations Sept. 2012Yangjun Chen ACS-3902/320 -Records are placed in the file in the order in which they are inserted, so new records are inserted at the end of the file. -Inserting a record is efficient: Files of Unordered Records (Heap Files) takes the last disk block into a buffer; adds the new record to it; rewrites the block back to the disk -Searching a record is not efficient. It needs scanning the file record by record. -Deleting a record: find the corresponding block; copy the block into a buffer; delete the record from the buffer; rewrite the record back to the disk.

File Organizations Sept. 2012Yangjun Chen ACS-3902/321 -Records are ordered on the values of one of their fields - called the ordering field. Files of ordered Records (Sequential Files, Sorted Files) NAMESSNBIRTHDATE JOBSALARYSEX Aaron, Ed Abbott, Diane Acosta, Marc …... block 1 Adams, John Adams, Robin Akers, Jan …... block 2 …...

File Organizations Sept. 2012Yangjun Chen ACS-3902/322 Files of ordered Records (Continued) -Searching a record can be very efficient. Binary search on the values of the ordered field: Algorithm Binary search on an ordering key of a disk file l  1; u  b; (*b is the number of file blocks*) while (u  l) do begin i  (l + u)/2; read bock i of the file into the buffer; if k < (ordering key field value of the first record in i) then u  i - 1; else if k > (ordering key field value of the last record in i;) then l  i + 1; else if the record with ordering_key = k is in the buffer then goto found else goto notfound; end goto notfound; k is the value to be found

File Organizations Sept. 2012Yangjun Chen ACS-3902/323 Files of ordered Records (Continued) -Searching a record can be very efficient. But searching on the non-ordering field values is the same as in the unordered file. -Deletion can be done efficiently: find the record; delete it and mark the place; periodically reorganize the file. -Insertion is expensive: find the position to insert the record with k; make space for the record with k; put the record with k in the place.

File Organizations Sept. 2012Yangjun Chen ACS-3902/324 Files of ordered Records (Continued) -Two methods for making insertion more efficient 1.Keep some unused space in each block. 2.Maintain a temporary unordered file called an overflow or transaction file. Periodically, the overflow file is merged with the main file insertion AB A ordered B + A’ merging main file overflow file