1. 06 File Objects and Directory Handling
PGI520S
Programming for Geoinformatics
06 File Objects and Directory Handling

2. File Objects and Directory Handling
Open and Close Files
Access Modes
Buffering
Properties
Read, Write and Tell
CSV Module
Introduction to CSV Files
Using the Module
Directory Handling
Import
Methods

3. File Class: Open and close files
Opening a file (creating a file object) is done with the open() method, which has the general form:
file_object = open(file_name [, access_mode] [, buffering])
file_name: mandatory, String with a valid filename
access_mode: optional, Read, write, append standard value 'r'
buffering: optional, buffersize standard value 0 – no buffering
Closing file (and flushing the buffer) is done with the close() method, which has the general form:
file_object.close()
see: Python Documentation: Built-in Functions

4. File Objects: Access modes
Description
Remark r
read only
file pointer at start of file works only on existing files rb
binary read only r+
read / write
rb+
binary read write w
write only
creates a new file will overwrite existing files wb
binary write only w+
read / write
wb+
binary read write a
append
Creates a new file if not existing, appends to existing file file Pointer at the end of file ab
append binary a+
append and read
ab+
binary append and read

5. File Objects: Buffering
Mode
Description
Remark
no buffering 1
buffers one line at a time
also known as row buffering
> 0
buffering with indicated buffer size
buffering with indicated buffer size using buffers may significantly increase performance.
< 0
buffering with system specific buffer size
uses settings from the operating system

6. File Objects: Properties
Property
Description
f.closed
Returns True if file is closed otherwise False
f.mode
Returns the mode the file was opened with
f.name
Returns the file name

7. File Objects: Read, write and tell
Method
Description
f.read()
reads file content (binary or text)
f.read([number_of_bytes])
reads number_of_bytes bytes starting at the current position of the file pointer
f.readline()
Reads a line from the file
f.readlines()
Reads all lines from file from the current file position and puts them in a list object
f.write(some_string)
writes some_string to file (binary or text)
f.tell()
Returns the current position of the file pointer

8. CSV Module: Introduction to CSV Files
CSV is a file format for plain text, tabular data, which is widely used for data exchange, supported by a large variety of programmes including ArcGIS, QGIS and EXCEL.
No | Name | Amount | Price
----| | |
1 | Jim | |
2 | Tim | |
3 | Rim | |
4 | Vim | |

9. CSV Module: Introduction to CSV Files
This tabular data can be stored in a simple text file:
example.csv
No,Name,Amount,Price
1,Jim,15,20.00
2,Tim,10,1.30
3,Rim,9,17.98
4,Vim,12,11.15
No,Name,Amount,Price\n1,Jim,15,20.00\n2,Tim,10,1.30\n3,Rim,9,17.98\n4,Vim,12,11.15
End of Line / End of Record / End of Row marker (OS dependent)
No,Name,Amount,Price\r\n1,Jim,15,20.00\r\n2,Tim,10,1.30\r\n3,Rim,9,17.98\r\n4,Vim,12,11.15
Unix/Linux/Android/OS X: \n
Windows, DOS, OS/2: \r\n
Warning:

10. CSV Module: An Example
Use the table of Page 10 and create a new csv file, which has an additional column called 'Total'. The values of the new column are obtained by multiplying the associated values in the Amount and Price columns.
No | Name | Amount | Price | Total
----| | | |
1 | Jim | | |
2 | Tim | | |
3 | Rim | | |
4 | Vim | | |

11. CSV Module: An Example
Using file objects, we found a way to solve this problem:
with open('data/example1.csv') as f: headline = f.readline() headers = headline[:-1].split(',') + ['Total'] table = [] for row in f.readlines(): fields = row[:-1].split(',') fields += ['{:.2f}'.format(float(fields[2]) * float(fields[3]))] table.append(fields) with open('data/output1.csv', 'w') as g: g.write(','.join(headers) + '\n') for row in table: g.write(','.join(row) + '\n')
with open('data/example1.csv') as f: headline = f.readline() headers = headline[:-1].split(',') + ['Total']
table = [] for row in f.readlines(): fields = row[:-1].split(',') fields += ['{:.2f}'.format(float(fields[2]) * float(fields[3]))] table.append(fields)
with open('data/example2.csv', 'w') as g: g.write(','.join(headers) + '\n') for row in table: g.write(','.join(row) + '\n')
data/example1.csv managed by file object f data/output1.csv managed by file object g
No,Name,Amount,Price 1,Jim,15, ,Tim,10, ,Rim,9, ,Vim,12,11.15
No,Name,Amount,Price,Total 1,Jim,15,20.00, ,Tim,10,1.30, ,Rim,9,17.98, ,Vim,12,11.15,133.80

12. CSV Module: An Example A more elegant way to achieve the same is:
from csv import DictReader, DictWriter
with open('data/example1.csv') as f, open('data/output1.csv', 'wb') as g: i_file = DictReader(f)
o_file = DictWriter(g, i_file.fieldnames + ['Total'])
table = [] for row in i_file: row['Total'] = '{:.2f}'.format(float(row['Amount']) * float(row['Price'])) table.append(row)
o_file.writeheader()
for row in table: o_file.writerow(row)
data/example1.csv managed by i_file data/output1.csv managed by o_file
No,Name,Amount,Price 1,Jim,15, ,Tim,10, ,Rim,9, ,Vim,12,11.15
No,Name,Amount,Price,Total 1,Jim,15,20.00, ,Tim,10,1.30, ,Rim,9,17.98, ,Vim,12,11.15,133.80

13. CSV Module: An Example To avoid any memory issues rewrite as:
from csv import DictReader, DictWriter
with open('data/example1.csv') as f, open('data/output1.csv', 'w') as g:
i_file = DictReader(f)
o_file = DictWriter(g, i_file.fieldnames + ['Total'])
o_file.writeheader()
for row in i_file: row['Total'] = '{:.2f}'.format(float(row['Amount']) * float(row['Price'])) o_file.writerow(row)
data/example1.csv managed by i_file data/output1.csv managed by o_file
No,Name,Amount,Price 1,Jim,15, ,Tim,10, ,Rim,9, ,Vim,12,11.15
No,Name,Amount,Price,Total 1,Jim,15,20.00, ,Tim,10,1.30, ,Rim,9,17.98, ,Vim,12,11.15,133.80

14. CSV Module: An Example
For CSV files without header replace DictReader and DictWriter by:
from csv import reader, writer
with open('data/example2.csv') as f:, open('data/output2.csv', 'w') as g:
i_file = reader(f) o_file = writer(g) for row in in_file: row += ['{:.2f}'.format(float(row[2]) * float(row[3]))] o_file.writerow(row)
data/example2.csv managed by i_file data/output2.csv managed by o_file
1,Jim,15, ,Tim,10, ,Rim,9, ,Vim,12,11.15
1,Jim,15,20.00, ,Tim,10,1.30, ,Rim,9,17.98, ,Vim,12,11.15,133.80

15. CSV Module: Additional Parameters
For reader, DictReader, writer and DictWriter objects, we can add additional parameters concerning the style of a csv file:
delimeter: ',', ';',' ', '\t'
quotechar: '"' or "'"
quoting: QUOTE_ALL
QUOTE_MINIMAL
QUOTE_NONE
QUOTE_NONNUMERIC

16. CSV Module: Example – Additional Parameters
Adding Parameters to the reader() and writer() methods:
import csv
with open('data/example2.csv') as f, with open('data/output3.csv', 'w') as g:
i_file = csv.reader(f, delimiter=',', quotechar="'", quoting=csv.QUOTE_MINIMAL)
o_file = csv.writer(g, delimiter=':', quotechar='"', quoting=csv.QUOTE_ALL) for row in i_file: row += ['{:.2f}'.format(float(row[2]) * float(row[3]))] o_file.writerow(row)
data/example2.csv managed by i_file data/output3.csv managed by o_file
1,Jim,15, ,Tim,10, ,Rim,9, ,Vim,12,11.15
"1":"Jim":"15":"20.00":"300.00" "2":"Tim":"10":"1.30":"13.00" "3":"Rim":"9":"17.98":"161.82" "4":"Vim":"12":"11.15":"133.80"

17. Directory Handling: Import
Methods for directory handling are available in the os module, which needs to be imported before you can use them. Import
either:
import os
or alternatively:
from os import getcwd, mkdir, rmdir, chdir, listdir

18. Directory Handling: Import
Methods for directory handling are available in the os module.
Method
Description
getcwd()
Returns current working directory
mkdir("NewDir")
Creates a new directory "NewDir".
Fails if "NewDir" already exists.
rmdir("OldDir")
Removes existing directory "OldDir".
Fails if "Old Dir" does not exist.
chdir("OtherDir")
Changes current working directory to "OtherDir".
Fails if "OtherDir" does not exist.
listdir("TheDir")
Returns a list of files in directory "TheDir".
Fails if "TheDir" does not exist.