Review

• Pathname Expansion
• The ? Meta-character
• The * Meta-character
• The [ and ] Meta-characters
• Builtins
• Help for Builtins
• Ways a Shell Can Be Created
• The Login Shell
• Shells Created at the Command Line
• Shells Created to Run a Script
• Creating Startup Files
• Different Shell Versions
• Using a Different Shell
• POSIX
• Shells for Running Scripts

New Material

• Running a Startup File after a Change has been Made
• Commands that are Symbols
• File Descriptors
• Redirecting Standard Error
• Shell Scripts
• Making a Shell Script Executable
• How Unix Runs a Program
• Running a Shell Script
• Specifying Which Shell Will Run a Script
• Comments in Shell Scripts

Questions

Are there any questions before I begin?

Readings

If you have the textbook you should read chapter 8, The Bourne Again Shell.

Homework 8

I have posted Homework 8 here.

As usual, it will be due next Sunday at 11:59 PM.

Graded Quiz Today

After I finish talking I will pass out the papers for today's graded quiz.

Today's Class

Today I'll discuss certain symbols that are really commands, how to make script executable, and what happens when you run a script.

Midterm Scores

If you believe I have scored one of your Midterm answers incorrectly you can request a rescore for that question on Gradescope.

If your score on the Final is significantly better than your score on the Midterm, I will replace your Midterm score with that of the Final when calculating your grade for the course.

If you want to discuss how you can improve your grade in this course, come see me during Office Hours, either in person or on Zoom.

You do not have to make an appointment to see me during Office Hours, but you may have to wait your turn if I am helping another student.

I want students to pass this course, and will do what I reasonably can to make that happen.

Your Current Standing in This Course

If you want to know your grade as of this moment, send me an email.

I have written a script that shows you all your scores and your grade as of today.

In performing this calculation, I will assume that your score on the final exam will be the same as that of you Midterm score.

I can do this only for your grade as of the Midterm.

I can't do this for your grade later in the semester.

Answers to Midterm Questions

I will post the answers to the midterm questions tonight.

Final Exam

The final for this course will take place on Tuesday, December 15th from 3:00 PM - 6:00 PM.

It will NOT take place in this room, but in M03-0730, the Pomplun Lab just down the hall.

You will also find this information at the top of the class web page.

Review

Pathname Expansion

• Pathname expansion allows you create a pattern of all the pathnames ...
• you need on the command line
• You create this patern using a combination of regular characters and meta-characters
• Meta-characters are also sometimes called wildcards
• They allow you to have one or more pathnames on the command line ...
• without a lot of typing
• For example if I wanted to list all files with a ".txt" extension ...
• I would write

  $ ls *.txt
  cars.txt        foo4.txt  fruit.txt           numbers_jumbled.txt
  empty.txt       foo5.txt  lines.txt           red_sox.txt
  five_lines.txt  foo6.txt  log.txt             sample.txt
  foo10.txt       foo7.txt  numbered_lines.txt  test.txt
  foo1.txt        foo8.txt  numbers1.txt        windows_lines.txt
  foo2.txt        foo9.txt  numbers2.txt
  foo3.txt        foo.txt   numbers3.txt

• You can use as many meta-characters as you want to form a pattern
• Pathname expansion is different from pathname completion ...
• which you get by hitting Tab ...
• and only creates one pathname

The ? Meta-character

• The question mark, ?, meta-character ...
• stands for a single occurance of any character
• ? can be used with any command
• Even those that don't normally deal with files

  $ echo dir?
  dir1 dir2 dir3 dir4

• The ? meta-character does not match a leading period in a filename
• You must explicitly enter a leading period, . when specifying an "invisible" file

The * Meta-character

• An asterisk, * , will match any number of characters in a pathname
• It will even match no characters
• * can be used with any command
• Even those that don't normally deal with files

  $ echo dir*
  dir dir1 dir10 dir2 dir3 dir4

• * cannot be used to match the initial period in a hidden filename

The [ and ] Meta-characters

• The square brackets, [ and ], are also meta-characters
• They work somewhat like ?
• They only match a single character in a pathname
• But the pathname character must match one of the characters ...
• within the brackets
• There can be many characters within the brackets ...
• but they only match a single character
• You can use a range of characters
• to avoid listing all the individual characters
• A range is created by listing the first and last characters of a sequence ...
• separated by a dash, -
• The range must be writen in alphabetical order
• To match all lower case characters you coud use

  [a-z]

• An exclamation mark, ! , or a caret, ^ has special meaning inside the brackets ...
• but only when it appears as the first character
• This construction matches any character that is NOT included within the brackets

Builtins

• Not all commands can be found on disk as executable files
• Some are actually contained within the shell itself
• Such commands are called builtins
• When you run a builtin the shell does not have to create a new process
• Instead, the shell calls a procedure in its own code to perform the task
• This means no subprocess is created ...
• which makes execution faster
• There is a Linux command that will tell you whether a command is a builtin or not
• The command is type, and you use it like this

  type COMMAND_NANE

• Here are some examples

  $ type bg
  bg is a shell builtin
  
  $ type cat
  cat is /bin/cat
  
  $ type pwd
  pwd is a shell builtin
  
  $ type cp
  cp is /bin/cp
  
  $ type type
  type is a shell builtin

Help for Builtins

• If you run man on a builtin command you will get nothing

  $ man bg
  No manual entry for bg

• For help information on builtins run help
• which is itself a builtin ...

  $ type help
  help is a shell builtin

• If you follow help with the name of a builtin ...
• it will print some information on that command

  $ help bg
  bg: bg [job_spec ...]
      Move jobs to the background.
  
      Place the jobs identified by each JOB_SPEC in the background, as if they
      had been started with `&".  If JOB_SPEC is not present, the shell"s notion
      of the current job is used.
  
      Exit Status:
      Returns success unless job control is not enabled or an error occurs.

• If you run help with no argument ...
• it will show all the builtin commands

  $ help 
  GNU bash, version 4.3.11(1)-release (x86_64-pc-linux-gnu)
  These shell commands are defined internally.  Type `help" to see this list.
  Type `help name" to find out more about the function `name".
  Use `info bash" to find out more about the shell in general.
  Use `man -k" or `info" to find out more about commands not in this list.
  
  A star (*) next to a name means that the command is disabled.
  
   job_spec [&]                     history [-c] [-d offset] [n] or history -anrw [filename>
   (( expression ))                 if COMMANDS; then COMMANDS; [ elif COMMANDS; then COMMA>
   . filename [arguments]           jobs [-lnprs] [jobspec ...] or jobs -x command [args]
   :                                kill [-s sigspec | -n signum | -sigspec] pid | jobspec >
   [ arg... ]                       let arg [arg ...]
   [[ expression ]]                 local [option] name[=value] ...
   alias [-p] [name[=value] ... ]   logout [n]
   bg [job_spec ...]                mapfile [-n count] [-O origin] [-s count] [-t] [-u fd] >
  ...

Ways a Shell Can Be Created

• There are three ways a user can create a shell
  • Logging in
  • On the command line
  • When running a scritp
• There are subtle differences between these three types of shells

The Login Shell

• When you first login to Unix, a shell is created for you
• This shell is your login shell
• The absolute pathname of this shell is contained in the variable SHELL
• On pe15 the default login shell version is Bash
• When your login shell starts up, it runs the commands found in /etc/profile
• This is a file customized by the system administrator for all users
• You can create your own customizations in a startup file ...
• in you home directory
• The file must have one of these names
  • .bash_profile
  • .bash_login
  • .profile
• We will use .bash_profile

Shells Created at the Command Line

• The shell is a program, just like cat or ls
• You can run another shell as a sub-shell of your current shell
• You do this by typing the name of the shell at the command line

   $ ps
     PID TTY          TIME CMD
   12778 pts/1    00:00:00 bash
   12969 pts/1    00:00:00 ps
  
  $ bash
  
  $ ps
    PID TTY          TIME CMD
  12778 pts/1    00:00:00 bash
  12970 pts/1    00:00:00 bash
  12973 pts/1    00:00:00 ps

• Notice that the first entry for Bash ...
• has the same process ID each time you run ps
• This is your login shell
• The second Bash process is the sub-shell
• So your login shell is still running ...
• but you are talking to a sub-shell
• Your login shell is sleeping
• It is waiting for the sub-shell to finish
• This sub-shell is the second type of shell
• It is not a login shell
• But you can interact with it
• So it is an interactive non-login shell
• An interactive non-login shell is a shell that you create ...
• without having to enter a password
• Interactive non-login shells have their own startup file called .bashrc
• It must be in your home directory

Shells Created to Run a Script

• A shell script is a text file with Unix commands
• You have been creating shell scrips for the Class Exercises ...
• and homework assignments
• A shell script contains Unix commands ...
• which only the shell can understand
• So when a process is created to run a script ...
• two files are loaded into the process RAM
  • The text file containing the script
  • The Bash binary executable file
• The picture in memory looks like this

• So you are always running another shell ...
• when you are running a Bash script

Creating Startup Files

• A startup file contains Unix commands that are run ...
• just before you get a prompt
• The system administrator creates a startup file for all users in /etc/profile
• After your login shell runs this file ...
• it looks for a startup file in your home directory
• The user customizable startup file normally used by Bash is .bash_profile
• This file must be placed in your home directory
• We'll discuss this more when we talk about shell variables and aliases

Different Shell Versions

• The shell we have been using is Bash
• Bash stands for Bourne again shell
• The first Unix shell was written by Steve Bourne at AT&T's Bell Laboratories
• You can still run this shell by typing sh
• System V Unix introduced the Korn shell, ksh ...
• which was written by David Korn
• It introduced aliases and command line editing
• Machines running Ubuntu have another shell called dash
• dash is a stripped down shell with very few features
• It was designed to run shell scripts
• Another popular shell is the C shell
• It is popular with C programmers ...
• because it's scripting language looks like C
• There are two versions of this shell
  • csh
  • tcsh
• But the difference between them is slight
• Both the Korn and C shells cannot run Bash scripts ...
• because of slight differences between them and Bash
• We will only concern ourselves with Bash ...
• and to a lesser extent sh

Using a Different Shell

• Different version of Unix and Linux have different shells installed
• You can run a different version of the shell as a subshell of Bash
• You do this by simply typing the name of the shell

  $ sh
  
  $ ps
    PID TTY          TIME CMD
  13942 pts/5    00:00:00 bash
  13953 pts/5    00:00:00 sh
  13955 pts/5    00:00:00 ps
  

• To find out what shells are available on a machine ...
• look at the text file /etc/shells
• Here are the shell available on pe15

  $ cat /etc/shells
  # /etc/shells: valid login shells
  /bin/sh
  /bin/bash
  /usr/bin/bash
  /bin/rbash
  /usr/bin/rbash
  /bin/dash
  /usr/bin/dash
  /usr/bin/tmux
  /usr/bin/screen
  /bin/tcsh
  /usr/bin/tcsh
  /bin/zsh
  /usr/bin/zsh
  /usr/bin/fish

POSIX

• A standard exists for how different version of Unix should behave ...
• so they can be compatible with each other
• It was created by Portable Application Standards Committee
• A committee of the IEEE (Institute of Electrical and Electronics Engineers )
• It is called POSIX (Portable Operating System Interface) 1003.2
• sh and dash are fully POSIX compliant
• But Bash is not

Shells for Running Scripts

• Many scripts are needed to set up and maintain Linux and Unix
• Many of these scripts are quite old ...
• and were written before Bash was created
• They were written to run under sh
• There are subtle differences between different shell versions
• If a script was written to run in one version of the shell ...
• it is not a good idea to run it using another shell version
• Otherwise the script may not run properly
• Installer scripts for Unix packages are written for a POSIX compatible shell
• Like sh

Attendance

New Material

Running a Startup File after a Change has been Made

• Usually, when you change a startup file ...
• you want the changes to take effect immediately
• But unless you do something special ...
• the changes won't take effect until you log in again
• You can make the changes take effect immediately ...
• by running the source command like this

  source .bash_profile

• source is a builtin

  $ help source
  source: source filename [arguments]
      Execute commands from a file in the current shell.
  
      Read and execute commands from FILENAME in the current shell.  The
      entries in $PATH are used to find the directory containing FILENAME.
      If any ARGUMENTS are supplied, they become the positional parameters
      when FILENAME is executed.
  
      Exit Status:
      Returns the status of the last command executed in FILENAME; fails if
      FILENAME cannot be read.

• source runs a script inside your current shell
• When you run a script without using source ...
• a new process is created
• The program running in this process is a shell
• If you ran .bash_profile like this

  ./bash_profile

• Any changes it made would be made in the shell ...
• used to run the script
• But that shell would disappear as soon as the script ended
• So none of the changes would take effect in your current shell

Commands that are Symbols

• Unix has some commands that are symbols rather than words
• I'll just mention them now and go into greater detail in future classes

  ( )	Runs whatever commands are enclosed in the parentheses in a sub-shell
  $( )	Command substitution: runs the commands enclosed in the parentheses in a subshell and returns their value to the command line, replacing the dollar sign, the parentheses and everything in them with this value
  (( ))	Evaluates an arithmetic expression: by default, Unix treats everything as text, but this command evaluates whatever it encloses as a numerical, rather than a string, expression
  $(( ))	Arithmetic expansion: evaluates an arithmetic expression and returns its value at that place on the command line
  [ ]	The test command: used to evaluate a boolean expression in constructs like if clauses
  [[ ]]	The conditional expression: similar to [ ] but adds string comparisons

File Descriptors

• Resources are given to each process when it is created
• Every time a process is created it is given connections to three data streams
  • Standard input
  • Standard output
  • Standard error
• A program or script can also open files ...
• to create additional data streams
• So how does Unix keep track these multiple streans?
• It does so through file descriptors
• File descriptors are represented by numbers that denote data streams
• Each file descriptor is assigned a positive number starting with 0
• Think of a file descriptor as an integer that represents a file
• Standard input, standard output and standard error ...
• each have their own file descriptors

  Name	File Descriptor
  Standard input	0
  Standard output	1
  Standard error	2

• So while we think of standard input, standard output and standard error
• Unix thinks of the file descriptors 0, 1 and 2
• Most of the time, you do not have to worry about file descriptors
• But they can appear in complex scripts

Redirecting Standard Error

• Standard error is the data stream where error messages are usually sent
• Redirecting standard error allows a program to separate its output ...
• from its error messages
• But we redirect Standard error differenly than Standard input and output
• To redirect standard input we use the less than symbol, <
• Followed by a pathname

  $ ./repeat.sh < test.txt 
  Enter several lines
  Type X on a line by itself when done
  
  You entered
  -----------
  123456789
  abcdefg
  987654321
  hijklmnop
  foo
  bar
  bletch
  X

• < is really a shorthand for a notation using file descriptors
• When you type

  ./repeat.sh  <  test.txt

  Unix thinks of this as

  ./repeat.sh  0<  test.txt

• Where the 0 in front of the greater than sign ...
• is the file descriptor for standard input
• Similarly, when we use output redirection

  $ echo Hello there  >  hello.txt

  Unix thinks of this as meaning

  $ echo Hello there  1>  hello.txt

• Again the file descriptor precedes the redirection symbol, >
• So how do we redirect standard error?
• We place a 2 in front of the greater than symbol, >

  $ ls xxxx
  ls: cannot access xxxx: No such file or directory
  
  $ ls xxxx 2> error.txt
  
  $ cat error.txt 
  ls: cannot access xxxx: No such file or directory

• In this command line 2> ...
• sends the error messages to the file error.txt ...
• not to the screen
• You can redirect both standard output and standard error to the same file
• You do this with ampersand and greater than symbols together, &>
• You will not have to know about file descriptors for the final
• But you will have to know how to redirect standard error

Shell Scripts

• A shell script is a file that contains Unix commands
• You can think of a shell script as a collection of command line entries
• When the shell script is executed ...
• each line of the script is run in turn
• A shell script can use any shell feature that is available at the command line ...
• except those features which are provided by tty (the device driver for the keyboard and screen)
  • Command line editing(arrow keys, control key combinations)
  • Pathname completion (tab to get more of a filename)
  • The history mechanism (up arrow to recall previous command line)
• You can use ambiguous file references in a shell script
• That is, you have full use of the metacharacters ?, *; and [ ]
• You can use redirection in a shell script and pipes
• Unix also provides control structures
  • Conditionals
  • Loops
• When you run a script
• The shell executes each commmand it comes across
• Normally that means starting at the top ...
• and going straight to the bottom
• Control structures allow you to change the path taken through the script

Making a Shell Script Executable

• You can run a shell script without typing bash on the command line ...
• if you give the script both read and execute permissions
• You need read permission because the shell has to read the contents of the script ...
• to load it into the process memory space
• You need execute permission so the script can be run without calling bash directly
• If you try to run a script without both permissions you will get an error

  $ ls -l cheer.sh 
  -rw-rw-r-- 1 ghoffman grad 13 Oct 29 14:23 cheer.sh 
  
  $ cat cheer.sh 
  #! /bin/bash
  
  # this file roots for the home team
  
  echo "Let's go Red Sox!"
  
  $ ./cheer.sh
  -bash: ./cheer.sh: Permission denied

• You set these permissions using chmod
• Normally you would give a shell script file 755 permissions
• The owner can read, write and execute
• The group and everyone else can read and execute

  $ chmod 755 cheer.sh
  $ ls -l cheer.sh 
  -rwxr-xr-x 1 ghoffman grad 13 Oct 29 14:23 cheer.sh
  
  $ ./cheer.sh
  Go Sox!

• But they won't be able to change the script

How Unix Runs a Program

• When you run a program in Unix, the kernel creates a process ...
• and gives this process its own chunk of memory
• The most important thing stored in this memory is the code itself
• It consists of binary machine language instructions
• Each of these instructions is executed one at a time
• The CPU needs to know which instruction to execute next
• This is why the computer has a special piece of memory ...
• called the program counter
• The program counter contains the memory address of the next instruction
• After each instruction finishes ...
• the program counter advances to the next instruction address

Running a Shell Script

• The shell is just a special kind of program
• It reads what you enter at the command line ...
• and runs programs for you ...
• asking the kernel to create a process in which to run the program
• When a program is running ...
• the shell sleeps until the program is finished ...
• unless you tell the shell to run the command in the background
• What is the binary code that is loaded into memory when you run a shell script?
• The script itself is just text, which the CPU cannot understand
• To run the script we need to load a binary executable file
• This file must understand the lines of the script ...
• and execute each one in turn
• The only binary executable that can understand a shell script is a shell

Specifying Which Shell Will Run a Script

• The kernel needs to load the binary code for a shell ...
• into the process that runs the script
• The picture in memory looks like this

• But each Unix version has many shells that can be run
• Which shell should be loaded to run a script?
• Normally the shell chosen will be the one you are currently using
• So if you are running Bash to give you a command line ...
• the kernel will load bash executable file ...
• into the process running the script
• And if you are running sh, the binary for sh will be used
• What if you need to run the script in different shell?
• It makes little difference whether we use bash or sh to run the scripts for this course ...
• because our scripts are fairly simple
• But it would make a big difference if we were writing an install script ...
• for a Linux program
• Install scripts often use advanced features of a shell ...
• so they need to be run in a shell version that has these features
• Install scripts have to be able to run on any Unix machine
• So install scripts are written to run on a POSIX compatible shell
• But Bash is not fully POSIX compatible
• Unix provides us with a solution to this problem
• It allows us to specify which shell to run when a script is executed
• It does this using something called the hashbang line ...
• sometimes called the shebang line
• This line must be the first line of the script
• And the first two characters on the line must be a hash mark, #, ...
• followed by an exclamation mark, !
• The exclamation mark is sometimes called bang
• After these two characters comes the absolute pathname of the shell ...
• that will run with the script
• The pathname following #! must be an absolute pathname
• Because you don't know where the user will be when the script is run
• The hashbang line tells your current shell which shell to use to run your script
• The hashbang line must be the first line in the script
• Unix looks at the first few characters of a file before running a script
• If it sees #! ...
• it runs the script using the shell whose pathname follows #!
• For this course, the hashbang line should be

  #! /bin/bash 

• It is good form to always use a hashbang line
• You may follow hashbang with a couple of spaces before the pathname
• To show you that this works I'm going to run the script shell_test_1.sh ...
• which you can find in /home/ghoffman/course_files/it244_files

  $ cat shell_test_1.sh
  #! /bin/sh
  ps -f
  
  $ ./shell_test.sh 
  UID        PID  PPID  C STIME TTY          TIME CMD
  ghoffman    710   709  0 13:25 pts/1    00:00:00 -bash
  ghoffman   2741   710  0 15:35 pts/1    00:00:00 /bin/sh ./shell_test.sh
  ghoffman   2742  2741  0 15:35 pts/1    00:00:00 ps -f

• I specified that this script should be run with the sh shell
• I did this by specifying it in the hashbang line

  #! /bin/sh

• Now compare this with shell_test_2.sh

  $ cat shell_test_2.sh
  ps -f
  
  $ ./shell_test_2.sh
  UID        PID  PPID  C STIME TTY          TIME CMD
  ghoffman    710   709  0 13:25 pts/1    00:00:00 -bash
  ghoffman   2893   710  0 15:41 pts/1    00:00:00 -bash
  ghoffman   2894  2893  0 15:41 pts/1    00:00:00 ps -f

• The second script has no hashbang line ...
• so the script was run in a Bash shell ...
• because I am running Bash
• If I were running another shell ...
• that shell would be loaded in the process space
• The shell did this because I did not tell it otherwise
• You can leave out the hashbang line and still run a script without calling Bash
• But you must use a hashbang line for scripts written in scripting languages ..
• like Perl and Python
• Here is a simple Python script that does not have hashbang line

  $ cat hello_1.py
  print("Hello world!")

• It has read and execute permissions

  $ ls -l hello_1.py 
  -rwxr-xr-x 1 ghoffman it244-1 22 Mar 29 09:59 hello_1.py

• But when I try to run it, there is a problem

  $ ./hello_1.py 
  ./hello_1.py: line 1: syntax error near unexpected token `"Hello world!"'
  ./hello_1.py: line 1: `print("Hello world!")'

• When I tell Bash to run any script ...
• and I don't use a hashbang line ...
• it will run the script using Bash
• But Bash does not understand Python code ...
• so you get an error message
• I can only run this Python script by calling the Python interpreter

  $ python hello_1.py 
  Hello world!

• Here is the same script with a hashbang line that references the Python interpreter

  $ cat hello_2.py 
  #! /usr/bin/python
  
  print("Hello world!")

• I can run this script directly

  $ ./hello_2.py 
  Hello world!

• You will not have to use the hashbang line for homework assignments ...
• until you get to homework 11
• Homework 11 is the last homework assignment
• The scripts you will write for that assignment must all have the following hashbang line

  #! /bin/bash

Comments in Shell Scripts

• Programs are written by people for machines
• But programs also have to be read by the people
  • Who write the program
  • Who maintain the program
  • Who use the program
• To make clear what is happening inside a program, use comments
• Comments are text which is ignored by whatever program is running the script
• They are only meant for people to read
• Anything following a hash mark, #, is a comment ...
• except for the hashbang line
• Here is an example

  $ cat comment_test.sh 
  #! /bin/sh
  #  demonstrates that comments do not affect the
  #  way the script runs
  echo Hello there # This text is ignored
  
  $ ./comment_test.sh 
  Hello there

• Comments are a way to document a program ...
• within the text of the program itself
• This sort of documentation is extremely important
• You may create a script today and not use it for a couple of months
• When you need to change it, you may have forgotten how it works
• A few well placed comments can save you hours of work
• It is good practice to place a comment at the top of the shell script ...
• after the hashbang line
• This comment should say what the script does

  $ cat bother.sh
  #!/bin/bash
  
  # prints something to the terminal 1000 times
  
  count=0
  while [ $count -lt 1000 ]
  do
     sleep 5
     echo Excuse me
     (( count++ ))
  done

• This script uses features of bash that we will discuss in future classes
• You should also comment any part of a script that does something less than obvious
• You will not have to write comments, other than Step comments, until homework 11
• Homework 11 is the last homework assignment

Class Exercise

Class Quiz