1. Tool Building: Libraries
CS 360
Tool Building: Libraries
tool3

2. Libraries

3. Software Structure of Unix, Again
users
shell
programs
unix libraries
unix kernel
Kernel: essential basic services
only one kernel
general
Libraries: useful additional services
many libraries
specific
today, we consider the "stdio" library

4. Libraries Are a Powerful Technique
your program
pixel library
animation
library
2D library
3D library
decreasing
generality,
but
increasing
specificity
and
convenience
reach down to a level that is useful
kernel facilities
Save development time by reusing debugged routines
Save maintenance time by evolving a coherent capability

5. When To Build a Library Have a set of useful reusable routines
Any particular program uses only a subset
Helpful to think of the routines as a coherent whole
Will invest effort to make the library good ...

6. Characteristics of Good Libraries
Complete
all elements needed for some specific problem
Concise
less is more
Coherent
pieces fit together well without undue mental stress
Correct
pieces work in all combinations and usages
Configurable
users can tune and extend as needed
Changeable
maintainers can evolve as environment changes

7. Key Files During Compiling and Linking
Source files:
C language
in text format
Object files:
machine instructions
data initialization
external names
in binary format
Libraries:
collections of object files
Executable files:
instructions, data
entry point ("main") & other details
x.c
y.c
z.c
compile
compile
compile
x.o
y.o
z.o
/usr/lib/libm.a
/usr/lib/lm.o
/usr/lib/lm.o
link
foo

8. Linker Details
The linker combines separate object files into a single executable file
The executable format is as follows:
cookie magic number that identifies this as binary executable file.
header size in bytes.
code combined code sections; always begins with standard C startup routine from system library.
data combined data sections; contains only initialized external and static internal data; room for uninitialized data set aside during loading; that includes numbers and strings ("abc\n").

9. Dynamic Linking
Some external references can be left unresolved
"static" linking - every needed library routine included in executable
"dynamic" linking - library routine references left to resolve during execution
How dynamic linking works:
call to routine (e.g. "printf") goes to general "loader" routine with "printf" as arg
loader copies needed routine into memory from library & redirects the call there
the copying is skipped if the needed routine is already present in memory
Advantages of dynamic linking:
size of executable reduced (library routine not present in executable)
overall system speed increased (library routines loaded just once & then shared)
program robustness increased (changes to library don't require relinking)
Disadvantages:
Library version control, can the executable find the right library routine version?
“Thrashing” of library routines in a memory constrained environment
Libraries prepared for dynamic linking are called "shared libraries" or "dynamic linking libraries" (DLL's)

10. Implementing a Library
Combine separate object files into a library file:
libnet.a : x.o y.o z.o
ar cru libnet.a x.o y.o z.o
ranlib libnet.a
x.o y.o z.o : useful.h
makefile
Use the library when you link:
program : main.o a.o b.o c.o libnet.a
gcc -o program main.o a.o b.o c.o libnet.a -lcs360
put libraries after the objects
Remember compile/link conventions:
#include <net.h>
gcc -o ... -lnet
#include "/usr/include/net.h”
gcc -o ... /usr/lib/libnet.a

11. External Variables static local to this file
#include "xyz.h"
int main (...) {
if (foo > ...) bar (...); }
main.c
static local to this file
extern defined in another file
nothing defined in this file
typedef ...
#define ...
extern int foo;
extern void bar ( ... );
xyz.h
#include "xyz.h"
#define ...
int foo = ...;
makefile
x.c
program : main.o libxyz.a
cc -o program main.o libxyz.a -lcs360
libxyz.a : x.o y.o
ar -cr libxyz.a x.o y.o
x.o y.o z.o : xyz.h
#include "xyz.h"
static char f ( ... );
static int g = ...;
void bar ( ... ) {
... }
static char f ( ... ) {
y.c

12. Standard I/O Library Motivation

13. Why Want More Than "read" & "write"?
Formatting:
recognize & produce human-readable text
e.g.: strings & decimal numbers (integer, floating)
Performance:
process by 1 character chunks efficiently
Convenience:
read 1 line
backup 1 char
remember fewer flags
Portability:
program on an OS without specific Unix kernel capabilities

14. Comparison: File Copy Using Kernel
Copy "/a/foo" to /b/bar" using kernel i/o:
#include <fcntl.h>
#include <unistd.h>
#define BUFMAX 100 /* size of read/write area */
char buffer[BUFMAX]; /* read/write area */
int fd1, fd2; /* file descriptors */
int n; /* read/write actual chunk size */
fd1 = open ("/a/foo", O_RDONLY, 0);
if (fd1 < 0) { ... handle error ... }
fd2 = open ("/b/bar", O_WRONLY|O_TRUNC|O_CREAT, 0666);
if (fd2 < 0) { ... handle error ... }
while ((n = read (fd1, buffer, BUFMAX)) > 0) {
write (fd2, buffer, n); }
close (fd1);
close (fd2);
one chunk at a time

15. Comparison: File Copy Using Stdio
Copy "/a/foo" to /b/bar" using stdio:
#include <stdio.h>
#define LINEMAX 100 /* size of get/put area */
char line[LINEMAX]; /* get/put area */
FILE *fp1, *fp2; /* file pointers */
fp1 = fopen ("/a/foo", "r");
if (fp1 == NULL) { ... handle error ... }
fp2 = fopen ("/b/bar", "w");
if (fp2 == NULL) { ... handle error ... }
while (fgets (line, LINEMAX, fp1) != NULL) {
fputs (line, fp2); }
fclose (fp1);
fclose (fp2);
one line at a time

16. Comparison: File Copy Using Stdio
Copy "/a/foo" to /b/bar" using stdio:
#include <stdio.h>
int c; /* get/put area */
FILE *fp1, *fp2; /* file pointers */
fp1 = fopen ("/a/foo", "r");
if (fp1 == NULL) { ... handle error ... }
fp2 = fopen ("/b/bar", "w");
if (fp2 == NULL) { ... handle error ... }
while (c = fgetc (fp1), c != EOF) {
fputc (c, fp2); }
fclose (fp1);
fclose (fp2);
one char at a time

17. Comparison: Data Formatting Using Stdio
A program to sum numbers in stdin:
% sum < numbers
sum = 48
1 2 3
numbers
#include <stdio.h>
int x, sum = 0;
while (fscanf (stdin, "%d", &x) == 1) {
sum += x; }
fprintf (stdout, "sum = %d\n", sum);
Code using stdio:
one number at a time
Notes:
"stdin" & "stdout" refer (somehow) to file descriptors 0 & 1
similarly, "stderr" refers to file descriptor 2
these names are not variables and not integers

18. Comparison: Differences
Use the routines that match your problem

19. chars may not be written to disk immediately
What is a "Buffer"?
buffer n. Thing that lessens, absorbs, or protects against impact.
buffer
stdio routines
disk
kernel routines
chars may not be written to disk immediately

20. Stdio Usage
Opening & Closing
Reading & Writing
Miscellaneous

21. Stdio Big Picture

22. Opening To get a new file pointer, we "open" a file
we supply pathname and an open "mode" (read or write)
library returns a new file pointer
returns NULL if it can't open the pathname per your request
don't know yet to what this points!
Example to open "/home/roger/foo" for reading:
#include <stdio.h>
...
FILE *fp;
fp = fopen ("/home/roger/foo", "r");
if (fp == NULL) {
... handle error .. }
pathname
mode: "r", "w", "a", ...
Notes on opening for writing:
"w" means to truncate to 0 size if exists; "a" means to append if exists
both "w" and "a" create the file if it doesn't exist
in that case, the mode used is read/write access for everyone (0666)
S_IRUSR | S_IWUSR | S_IRGRP | S_IWGRP | S_IROTH | S_IWOTH

23. Closing It's pretty simple: Notes on buffering for "write" opens:
fclose (fp);
Notes on buffering for "write" opens:
buffers are periodically "flushed" (calls to "write" occur)
when buffer is full
when file descriptor is closed
when you do it explicitly (via "fflush" routine)
when a process terminates normally, the library closes all open streams
thus buffers are flushed
normal: main routine returns or "exit" is called
if process terminates abnormally, the library doesn't get chance to close streams
thus the buffers are not flushed
e.g., material in stdout buffer (e.g. via "fprintf") may not get to the terminal
however, stderr has no buffer, so it's material always gets to terminal

24. Don't Confuse File Pointers and Descriptors
stdio library
kernel table
disk
your code
buffer pointer
struct {
} FILE
18978 3
file descriptor fp
data blocks
inode 4529
buffer:
file pointer:
points to a FILE structure
inode:
index into the
disk management
area
file descriptor:
index into a
table that
contains inodes
FILE:
the kind of structure manipulated by the
stdio library; includes
a file descriptor and
a buffer pointer
pointer!
integer!

25. Reading and Writing
Four families of routines are provided to read/write streams:
In following examples, we assume
fpi is an open input stream
fpo is an open output stream

26. Reading & Writing by Characters
An example that copies fpi stream to fpo stream while converting lower case characters to upper case:
int c;
...
while (c = fgetc (fpi), c != EOF) {
if (('a' <= c) && (c <= 'z')) {
c = c - 'a' + 'A'; }
fputc (c, fpo);
why?
Notes:
"c" is declared as "int"

27. Reading & Writing by Lines
An example that prints the n-th line from stdin:
#define BUFSIZE 1000
int main (int argc, char *argv[]) {
char buffer[BUFSIZE]; /* area for input line */
int n; /* line wanted 1, 2, ... */
int i; /* lines read so far */
n = atoi (argv[1]); i = 0;
while ((i < n) && (fgets (buffer, BUFSIZE, stdin) != NULL)) {
++ i; }
if (i == n) fputs (buffer, stdout);
return 0;
fgets appends '\0'
'\n' is included (if fits)
fputs writes until '\0'; doesn't add a '\n'

28. Use "fprintf" to Produce Formatted Output
Example:
Operation:
fprintf writes characters in the output string one at a time
a % marks a format code, which consumes and prints a data value
%s data value is a string (pointer to first char in null terminated sequence)
%c data value is a character
%d data value is an integer to be printed in decimal
%x data value is an integer to be printed in hex
the data values are consumed left-to-right, each matching a format code
"Printf" is the same, but uses "stdout"
fprintf (stderr, "This is x=%d and y=%d right now\n", x, y);
where output should go
output string and format codes
data values that the codes will consume
lots of options!
printf ("This is x=%d and y=%d right now\n", x, y);

29. Use "fscanf" to Process Formatted Input
Example:
Operation:
fscanf moves thru format string left-to-right, interpreting each char as a code
the codes cause input to be converted; the results are stored using the pointers
a % marks a conversion code, which "eats" some input
%d decimal integer (possibly signed)  int *
%s sequence of non-whitespace chars  char *
%c single character (no whitespace skip)  char *
%[^chars] like %c but input must not be any char  char *
any other char must appear in the input
whitespace causes input whitespace to be skipped
Value:
returns number of stores (stops at first conflict or end of format) or EOF (no conflicts)
"scanf" is the same, but uses "stdin"
pointers!
fscanf (stdin, "%d %d", &x, &y);
where input comes from
format string
where data values will be stored
(all values MUST be pointers)
eat: v. skip over white space; parse input (greedily); convert input; store result using pointer
scanf ("%d %d", &x, &y);

30. Formatted Input/Output Example
Find the most popular item:
% popular < item
pear = 10
1 apple 10
pear 3 peach
2 berry 12
item
Code using scanf & printf:
int vote, maxVote; /* current and best vote so far */
char item[64], maxItem[64]; /* current and best item so far */
maxVote = 0;
while (scanf ("%d %s", &vote, item) == 2) {
if (vote > maxVote) {
vote = maxVote;
strcpy (maxItem, item); }
if (maxVote > 0) {
printf ("%s = %d\n", maxItem, maxVote);
note carefully the pointers!

31. Positioning a Stream
The "fseek" function changes the "current offset":
The "ftell" operation reports it's value:
Nice & simple, since Unix makes no distinction between text & binary files ...
#include <stdio.h>
long li=...
fseek(fp, li, SEEK_SET);
fseek(fp, li, SEEK_CUR);
fseek(fp, li, SEEK_END);
new offset = i
new offset = old offset + i
new offset = size of file + i
how could you code this routine?
li = ftell(fp);

32. Binary vs. Text Files On Unix, there is no distinction
a newline = 1 char
Some other OS's make a distinction
a newline = CR/LF pair
You can open in a portable way, by indicating you are reading/writing a binary or a text file:
But, positioning will still be a problem ...
fp = fopen (..., "r"); /* convert cr/lf to '\n' as read */
fp = fopen (..., "w"); /* convert '\n' to cr/lf as written */
fp = fopen (..., "rb"); /* no such conversion; rarely used */
fp = fopen (..., "wb"); /* no such conversion; rarely used */

33. (details not guaranteed!)
The FILE Structure
The stdio routines use a "FILE" structure to maintain those buffers:
typedef struct {
int _file; /* file descriptor */
char *_base; /* start of buffer */
int _bufsiz; /* size of buffer */
char *_ptr; /* next char to get */
int _cnt; /* # chars left in buffer */
int _flag; /* misc info */
...
} FILE;
#include <stdio.h>
(details not guaranteed!)
You communicate with the stdio routines using a pointer to this structure:
FILE *fp;
fp = fopen ("/a/foo", "r");
while (fgets (line, LINEMAX, fp) != NULL) {
... }
fclose (fp);
<stdio.h> has other useful stuff:
stdin, stdout, stderr each a "FILE *" w/ file descriptor 0, 1, 2 respectively
NULL a pointer to nothing
EOF (-1) ... what fgetc returns on end-of-file

34. Macros
Some operations are so common and so lightweight that the overhead of a function call seems excessive
get 1 char
put 1 char
do above on stdin or stdout
# define getc(fp) (*((fp)->_ptr)++)
# define getchar() getc(stdin)
stdio.h
So, some macros are defined for these operations
similarly, "putc" and "putchar"
the code above can't be the full story - why?
putchar ('a');
fputc ('a', stdout); 
The function forms are always correct
we will not discuss the macro forms further since they are simple abbreviations

35. Terminology: Files & Streams
Note that the word "file" is used in many ways:
a collection of disk blocks
an inode that manages such a collection
a file descriptor that is associated with an inode
a FILE structure (which includes a file descriptor)
a pointer to such a structure (also called a "file pointer")
any of the above that is not a "directory" (i.e. a "regular file")
The term "stream" is often used in connection with stdio because it suggests a useful viewpoint about how files can be processed:
sequence of units (chars, lines, ...), which ...
flow into program in order, and ...
flow out of program in order
We often say that when we create a file pointer and associate it with a file descriptor, we are "creating a stream"
i.e. "stream"  "file pointer" (in this context)

36. disk
cache
buffer
your program
Buffer Details
The <stdio.h> library uses three buffering disciplines:
Use "fflush" to explicitly flush a buffer:
More control is possible with "setvbuf"
however, the big picture above is usually all you need to remember
fflush (fp);

37. how is that implemented?
Temporary Names
Sometimes, your program needs a temporary file
Several functions are available, but the simplest is "tmpfile":
the fp is open for both reading and writing
the file will be removed at program termination (normal or abnormal)
fp = tmpfile ();
how is that implemented?

38. Some Simple Formatting Tricks
To create a formatted string, use "sprintf":
you can then process the string (which is terminated by '\0')
receiving area must be big enough!
To parse a string, use "sscanf", which is similar
char area[64]; /* receiving area */
sprintf (area, "%s = %d", vote, item);

39. Summary The stdio library is a very successful design
clear separation of concerns between kernel & library
good choice of function and clear grouping of functions
limited intellectual size
Should be part of every system programmer's toolkit
can make some tasks easier
can help organize program structure
available in all C implementations
It's not perfect
double-copy performance overhead
inconsistent naming and argument conventions (fputs vs. fprintf)
cumbersome eof indication (fgetc)

40. Stdio Implementation

41. Stdio.h Content Typedefs FILE Defines EOF and NULL
Defines stdout, stdin, and stderr (somehow)
Declares routines (big 6 are below)
error handling:
NULL if error
EOF if error, 0 if OK
EOF if error or end of file
NULL if EOF, line if not
extern FILE *fopen (char *filename, char *mode);
extern int fclose (FILE *fp);
extern int fgetc (FILE *fp);
extern int fputc (char c, FILE *fp);
extern char *fgets (char *line, int n, FILE *fp);
extern int fputs (char *line, FILE *fp);

42. Problem: Boundary Crossings
Reads and writes must cross the kernel/user boundary
disk
kernel space
user space
your program
cache
read
write
Entering kernel space (a "syscall") is quite expensive
If we read/write in chunks  disk block size, we cross the boundary without actually touching the disk

43. An intermediate area is inserted between your program and the kernel
Solution: Buffers
An intermediate area is inserted between your program and the kernel
kernel space
user space
fgetc fgets
...
disk
cache
read
your program
fputc fputs
...
buffer
write
Characters are taken/placed from/into the buffer
If buffer is empty/full, then read/write actually done (aka "refresh"/"flush")
Therefore, fewer boundary crossings
... But, more copying
no buffer: only one copy operation needed to make data available in user space
with buffer: two copy operations are needed

44. (details not guaranteed!)
The FILE Structure
The stdio routines use a "FILE" structure to maintain buffers:
typedef struct {
...
} FILE;
extern FILE * fopen ( ... );
extern int fclose ( ... );
extern char * fgets ( ... );
extern int fputs ( ... );
(details not guaranteed!)
/usr/include/stdio.h
#include <stdio.h>
FILE * fp;
fp = fopen ("/a/foo", "r");
while (fgets (line, LINEMAX, fp) != NULL) {
... }
fclose (fp);
You communicate with the stdio routines using a pointer to this structure:
mine.c

45. Buffer Big Picture: Open and Close
Opening:
malloc a buffer
malloc a FILE
open and get a file descriptor
fill in the FILE fields
return pointer to the FILE
Closing:
if open for writing:
write any chars stored in buffer
close the file descriptor
free the buffer
free the FILE fp
a FILE pointer
a FILE
various
fields
a buffer
some #defined size

46. Buffer Big Picture: Get and Put
fputc (c, fp)
place c at next avail position
(i.e. store c, increment position)
already placed
avail
if buffer is full, flush it then place c (i.e. write contents, reset position, do logic on left) fp
a FILE pointer
fields
a FILE
contents of the buffer associated with fp
c = fgetc (fp)
contents of the buffer associated with fp
take c from next avail position (i.e. take c, increment position)
already taken
not yet taken
empty
if buffer is empty, refresh it then take c (i.e. read more content, reset position, do logic on left)
eof?

47. Design: FILE Our design:
The FILE structure must keep track of the following:
the open file descriptor
whether the file descriptor is open for writing or reading
a buffer area and the size of that area
if writing:
where in the buffer the next fputc should place its character
how much room remains before the buffer is full and must be flushed
if reading:
from where in the buffer the next fgetc should take a character
how many characters remain before the buffer is empty and must be refreshed
fixed at fopen time
Our design:
typedef struct {
int fd; /* the open file descriptor */
char *buffer; /* the buffer area */
int size; /* size of the buffer area */
int writing; /* are we writing? */
int i; /* buffer[i] is next char to be placed or taken */
int n; /* full/empty counter used by fputc/fgetc */
} FILE;

48. Design: fputc Here is a FILE in action writing: fputc (c, fp): 4 100 6
typedef struct {
...
int i; /* buffer[0..i-1] have already been placed */
int n; /* buffer[i..n-1] are still available */
} FILE;
fputc (c, fp): 4
100 6
100 fd
buffer
size i n
= size (via fopen)
the buffer can hold 100 chars
reading using file descriptor 4
next xputc will replace buffer[6]
figure it out! n
abcde
size

49. Design: fgetc Here is an FILE in action reading: 4 100 3 10
typedef struct {
...
int i; /* buffer[0..i-1] have already been taken */
int n; /* buffer[i..n-1] are still available */
} FILE; 4
100 3 10
fgetc (fp):
the last read put 10 chars into the buffer fd
buffer
size i n
the buffer can hold 100 chars
reading using file descriptor 4
next fgetc will take buffer[3] n
abcdef\nxyz
figure it out!
size

50. Design: fopen Here is a diagram of a FILE in action for opening: 100 ?
typedef struct {
int fd; /* the open file descriptor */
char *buffer; /* the buffer area */
int size; /* size of the buffer area */
int writing; /* are we writing? */
int i; /* buffer[i] is next char to be placed or taken */
int n; /* full/empty counter used by fputc/fgetc */
} FILE;
100 ? ?
fopen (filename, mode):
malloc a new FILE chunk
malloc a new buffer of size 1024
open filename per supplied modes
set all the FILE fields appropriately
return pointer to the new FILE chunk
return NULL if anything goes wrong fd
buffer
size i n
size

51. Design: fclose Here is a diagram of a FILE in action for closing: 4
typedef struct {
int fd; /* the open file descriptor */
char *buffer; /* the buffer area */
int size; /* size of the buffer area */
int writing; /* are we writing? */
int i; /* buffer[i] is next char to be placed or taken */
int n; /* full/empty counter used by fputc/fgetc */
} FILE; 4
100 ? ? fd
buffer
size i n
fclose (fp):
if open for writing, flush the buffer:
write ? chars
close fd
size
figure it out!