Review

• Files
• Types of Files
• Identifying a File
• Storing Data on a Hard Disk
• Files and the Kernel
• Working with a File in Python
• Objects
• Accessing the Data in Files
• Opening a File
• Writing Data to a File
• Reading Data From a File

New Material

• Reading a File Line By Line
• Getting Rid of Newline Characters
• Writing and Reading Numbers
• Using Loops to Process Files
• Reading Files with while and readline
• Reading Files with a for Loop
• Professional Responsibility
• Codes of Conduct

Homework 8

I have posted homework 8 here.

It is due this coming Sunday at 11:59 PM.

Quiz 6

I have posted the answers to Quiz 6 here.

Questions

Are there any questions before I begin?

Review

Files

• When processing large amount of data we need to use files
• A file is simply a linear arrangement of data on some long term storage device
• That storage device might be a
  • Hard disk
  • Flash drive
  • CD ROM
  • SSD card

Types of Files

• All files consist of binary numbers
• but those numbers can be interpreted in two ways
  • Text
  • Binary
• Consider the following text file

  $ cat fruit.txt 
  grapes
  pears
  oranges
  cranberries
  apples
  melons
  blueberries

• To see how this file is really stored on disk I can use the Unix od (octal dump) command

  $ od -b fruit.txt 
  0000000 147 162 141 160 145 163 012 160 145 141 162 163 012 157 162 141
  0000020 156 147 145 163 012 143 162 141 156 142 145 162 162 151 145 163
  0000040 012 141 160 160 154 145 163 012 155 145 154 157 156 163 012 142
  0000060 154 165 145 142 145 162 162 151 145 163 012
  0000073

• To see the characters, I can run od with the -c option

  od -c fruit.txt 
  0000000   g   r   a   p   e   s  \n   p   e   a   r   s  \n   o   r   a
  0000020   n   g   e   s  \n   c   r   a   n   b   e   r   r   i   e   s
  0000040  \n   a   p   p   l   e   s  \n   m   e   l   o   n   s  \n   b
  0000060   l   u   e   b   e   r   r   i   e   s  \n
  0000073

• A binary file, such as a JPEG image also consists of numbers
• But the numbers do not represent characters
• Text files have a simple structure
• Each file is a collection of lines
• A line is a series of characters with the newline character, \n, at the end
• We will only be dealing with text files in this course

Identifying a File

• Every file has two attributes that uniquely identify it
  • Name
  • Location
• The location of a file is given by a path
• A path is a list of directories
• The last directory in a path is the one that holds the file

Storing Data on a Hard Disk

• Hard disk consists of rapidly spinning disks
• The disk is made up of material that can be magnetized
• Small regions on the disk are magnetized to point up or down
• Each of these regions stands for a binary digit called a bit
• The surface of the disk is broken up into rings called tracks
• Each track is broken up into sectors
• Most files consist of many sectors scattered throughout the disk

Files and the Kernel

• At the core of every operating system is a program called the kernel
• The kernel handles all interactions with hardware
• To read a file, the interpreter must use the kernel
• The kernel will then call a special program called a device driver
• The device driver handles communication between the hardware and the kernel
• Each hardware device attached to the computer must have a device driver
• The device driver is created by the manufacturer of the device

Working with a File in Python

• A file lives on a long term storage device
• But a program works with the file in RAM
• RAM is fast and expensive
• External storage is slower and cheap
• RAM is limited and files can be very large
• So there is usually only a small part of the file in RAM
• A file object contains the bookkeeping data ...
• needed to work with the file

Objects

• A variable holds only one piece of information
• Objects can hold many such pieces
• The data is contained in variables inside the object
• Objects also hold functions that interact with the data
• The functions inside an object are called methods
• Objects have no name
• Just a location in RAM
• We create an object by using a special function called a constructor
• The constructor returns a pointer to the object in RAM
• We store this value of this pointer in a variable
• Then we use the variable to work with the object

Accessing the Data in Files

• There are two ways to get data from a file
  • Sequential access
  • Direct access
• Sequential access is like a tape recorder
• It starts at the beginning and runs to the end
• It can only go in one direction
• Direct access goes to a particular place in the file
• We will only be using sequential access in this class

Opening a File

• To work with a file we have to create a file object
• We do this using the built-in function open
• The open function does two things
  • It creates a file object
  • It returns a reference to the object
• The reference is stored in a variable
• The picture in memory looks like this

• We then use the variable to work with the object
• Here is the format for using the open function

  FILE_VARIABLE = open(FILENAME, MODE)

• FILE_VARIABLE is the name of the variable that holds the reference
• FILENAME is a string specifying the file
• MODE determines what we can do with the file
• The three most commonly used access modes are

  Mode	Description
  "r"	Open a file for reading only
  "w"	Open a file for writing
  "a"	Open a file for writing to the end of the file

• The a in the last mode stands for "append"
• Text is added to the end of the file so nothing is lost
• A file open for reading cannot be changed
• If you try to open a file for reading ...
• and the file does not exist ...
• you will get an error
• If you try to open a file for writing or appending ...
• that does not exists ...
• the kernel will create the file for you
• If you open a file for writing that already exists ...
• its contents will be replaced by whatever you write to it

Writing Data to a File

• To work with a file in Python, we must first create a file object
• This object contains methods that allow us to talk to the file
• The functions inside objects are called methods
• To call a method you must use dot notation
• Dot notation for methods has the following format

  OBJECT_VARIABLE.METHOD_NAME

• We first open the file for writing

  teams_file = open("teams.txt", "w")

• Now we can send text to the file using write method of this object

  $ cat write_teams.py
  # writes the names of the teams in the American League East
  
  teams_file = open("teams.txt", "w")
  
  teams_file.write("Boston Red Sox\n")
  teams_file.write("Baltimore Orioles\n")
  teams_file.write("Toronto Blue Jays\n")
  teams_file.write("Tampa Bay Rays\n")
  teams_file.write("New York Yankees\n")
  
  
  $ python3 write_teams.py 
  
  $ cat teams.txt 
  Boston Red Sox
  Baltimore Orioles
  Toronto Blue Jays
  Tampa Bay Rays
  New York Yankees

• Notice that I had to put the newline character, \n
• At the end of each line
• Otherwise all the text would be on one line

Reading Data From a File

• There is more than one method that can read a line
• The simplest is read
• Which returns a string containing everything in the file

  $ cat read_teams.py
  # reads the contents of the file teams.txt
  
  teams_file = open("teams.txt", "r")
  
  teams = teams_file.read()
  
  print(teams)
  
  $ python3 read_teams.py 
  Boston Red Sox
  Baltimore Orioles
  Toronto Blue Jays
  Tampa Bay Rays
  New York Yankees

• It turns out that read is not very useful
• Normally we want to read a file line by line

Attendance

New Material

Reading a File Line By Line

• What if you had a file with numbers, one to each line ...
• that you needed to add?
• The read method cannot do this
• We need to read in one line at a time ...
• to get each number ...
• and the add it to the running total
• One way to do this is to use the readline method of the file object
• Remember, objects have functions built inside them called methods ...
• that work with the data inside the object
• Each time you call readline it returns the next line in the file
• By calling readline many times ...
• we can read the entire contents of a file
• So if wanted to read and print the text file teams.txt created in the last class

  $ cat teams.txt 
  Boston Red Sox
  Baltimore Orioles
  Toronto Blue Jays
  Tampa Bay Rays
  New York Yankees

• We can do this calling readline 4 times

  $ cat readline_teams.py 
  # reads and prints the lines in teams.txt
  
  file = open('teams.txt', 'r')
  
  line_1 = file.readline()
  print(line_1)
  line_2 = file.readline()
  print(line_2)
  line_3 = file.readline()
  print(line_3)
  line_4 = file.readline()
  print(line_4)
  
  $ python3 readline_teams.py 
  Boston Red Sox
  
  Baltimore Orioles
  
  Toronto Blue Jays
  
  Tampa Bay Rays

Getting Rid of Newline Characters

• Why are there blank lines between the teams in the output above?
• That's because readline reads in the entire line ...
• including the newline character, \n, at the end
• There is a simple fix to this problem using a string method
• String have methods because they are objects
• I didn't mention this earlier because I didn't want to confuse you
• The string method we need is strip() ...
• which produces a copy of the string ...
• with spaces removed from both ends
• Here is an improved version of the script

  $ cat readline_teams_2.py 
  # reads and prints the lines in teams.txt
  
  file = open('teams.txt', 'r')
  
  line_1 = file.readline()
  print(line_1.strip())
  line_2 = file.readline()
  print(line_2.strip())
  line_3 = file.readline()
  print(line_3.strip())
  line_4 = file.readline()
  print(line_4.strip())
  
  $ python3 readline_teams_2.py 
  Boston Red Sox
  Baltimore Orioles
  Toronto Blue Jays
  Tampa Bay Rays

Writing and Reading Numbers

• Let's say we wanted to create a file of pseudorandom numbers ...
• one number to a line
• We might write code like this

  # creates a series of random numbers and writes them to a file
  # this script will not work
  
  import random
  
  filename = input("Filename: ")
  how_many = int(input("How many numbers: "))
  highest  = int(input("Highest number: "))
  
  file = open(filename, "w")
  for i in range(how_many):
      file.write(random.randint(1, highest))

• But when we run it we get an error

  $ python3 random_numbers_create.py 
  Filename: numbers.txt
  How many numbers: 10
  Highest number: 5
  Traceback (most recent call last):
    File "random_numbers_create.py", line 11, in <module>
      file.write(random.randint(1, highest))
  TypeError: write() argument must be str, not int

• The problem lies with the access mode
• The three access modes we use in this class
  • r   Read
  • w   Write
  • a   Append
• Only work with text files
• So write method only accepts text as an argument
• The integers created by calling randint are not strings
• If we wanted to write integers to a file ...
• we would need an access mode that works on binary files
• These access modes exist ...
• but we won't be using them in this class
• We will only be using text files in this course
• So we must convert any numbers we write into strings
• Here is a corrected version of the script

  # creates a series of random numbers and writes them to a file
  
  import random
  
  filename = input("Filename: ")
  how_many = int(input("How many numbers: "))
  highest  = int(input("Highest number: "))
  
  file = open(filename, "w")
  for i in range(how_many):
      file.write(str(random.randint(1, highest)) + "\n")

• We can run this script without getting an error

  $ python3 random_numbers_create_2.py 
  Filename: numbers.txt
  How many numbers: 10
  Highest number: 5

• And the results are what we expect

  $ cat numbers.txt 
  2
  2
  4
  2
  5
  4
  3
  3
  1
  5

• When we read in a text file of numbers ...
• those numbers will be strings
• and we need to use either int or float to convert them back into numbers

Using Loops to Process Files

• In the example above we knew how many times to call readline ...
• because we knew how many lines there were in the file
• But that situation does not occur often
• So we need a better way to proceed
• And that involves using loops
• We can do this because file objects know when they have reached the end of a file ...
• and will not let us go beyond that point
• There are two ways to loop through a file
• First I will show you a technique using readline ...
• and while loops
• But this method is seldom used
• There is a much better way
• So why do I show you while loops first?
• Because it uses the information in the file object more directly
• I want you to get comfortable with the concept of a file object
• Then I can show you a better way to process a file

Reading Files with while and readline

• Normally when you call readline ...
• it returns the next line in the file ...
• including the newline character, \n, at the end
• But readline can't do this when it reaches the end of the file
• It needs some way to tell us that there are no more lines ...
• and it does this is a very simple way
• When readline gets to the end of a file ...
• the next time it is called it will return the empty string
• The empty string is a perfectly good string
• It serves the same purpose for strings as 0 does for integers
• With this in mind we can use the following algorithm

  open the file for reading
  call readline to get the first line
  while the line is not the empty string:
      process the last line from the file
      read in the next line

• Notice that I had to call readline twice
• Once before the loop ...
• and once inside the loop
• The while loop needs a line to test ...
• before entering the loop
• So I need to call readline before the loop
• Here is a script to read a file and print each line

  $ cat while_loop_print_file.py
  # reads and prints the lines in a file
  # using readline and a while loop
  
  filename = input("Filename: ")
  file = open(filename, "r")
  
  line = file.readline()
  while line != "":
      print(line.strip())
      line = file.readline()
  
  $ python3 while_loop_print_file.py
  Filename: teams.txt
  Boston Red Sox
  Baltimore Orioles
  Toronto Blue Jays
  Tampa Bay Rays

Reading Files with a for Loop

• Most programming languages read a file using a while loop
• But Python has a better and simpler way
• We can use a for loop
• Here is the format for a Python for loop

  for LOOP_VARIABLE in LIST_OF_SOME_KIND:
      STATEMENT
      STATEMENT
      ...

• Python has objects called lists that we will discuss in a few weeks
• But a for loop can use anything which is a collection of things
• A text file is collection of lines
• So we can run through it with a for loop
• But what do we put in the header of the loop ...
• after the keyword in?
• We use a file object
• Here is a for loop that reads a file and prints each line

  $ cat for_loop_print_file.py
  # reads and prints the lines in a file
  
  filename = input("Filename: ")
  file = open(filename, "r")
  
  for line in file:
      print(line.strip())
  
  $ python3 for_loop_print_file.py
  Filename: teams.txt
  Red Sox
  Patriots
  Celtics
  Bruins

• Let's compare the two scripts that read a file and print each line
• I can do in 2 lines with a for loop

  for line in file:
      print(line.strip())

• What takes me 4 lines with a while loop

  line = file.readline()
  while line != ":
      print(line.strip())
      line = file.readline()

• Short and simple is always better than long and complicated
• This is why I usually use a for loop when reading files

Class Exercise

Class Quiz