IT 244: Introduction to Linux/Unix

IT 244: Introduction to Linux/Unix
Class 18

Review

Running a Startup File after a Change has been Made
Commands that are Symbols
File Descriptors
Redirecting Standard Error
Shell Scripts
Making a Shell Script Executable
Specifying Which Version of the Shell Will Run a Script
Comments in Shell Scripts

New Material

Microphone

Homework 8

I have posted Homework 8 here.

As usual, it will be due next Sunday at 11:59 PM.

Quiz 6

I have posted the answers to Quiz 6 here.

Hashbang Line and Comments in Homework Assignments

You will not have to include a hashbang line until homework 11.

Homework 11 is the last homework assignment.

You should use the following as your hashbang line in that assignment

#! /bin/bash

For homework assignments 8, 9 and 10 you will only need to write the usual Step comments.

For homework 11 you will have to write a comment at the top of each script.

This comment should describe what the program does.

Questions

Are there any questions before I begin?

Review

Running a Startup File after a Change has been Made

Usually, when you change a startup file you want the changes to take place immediately
But if you made a change to .bash_profile they don't take effect immediately
You have to wait until the next time you login
Unix gives you a way to make the changes take effect immediately
You do this by running the source command
```
source .bash_profile
```

Commands that are Symbols

Unix has some commands that are symbols rather than words
I'll just mention them now and go into greater detail in future classes

( )	Runs whatever commands are enclosed in the parentheses in a sub-shell
$( )	Command substitution: runs the commands enclosed in the parentheses in a subshell and returns their value to the command line, replacing the dollar sign, the parentheses and everything in them with this value
(( ))	Evaluates an arithmetic expression: by default, Unix treats everything as text, but this command evaluates whatever it encloses as a numerical, rather than a string, expression
$(( ))	Arithmetic expansion: evaluates an arithmetic expression and returns its value at that place on the command line
[ ]	The test command: used to evaluate a boolean expression in constructs like if clauses
[[ ]]	The conditional expression: similar to [ ] but adds string comparisons

File Descriptors

When a process is created, is connected to 3 data streams
- Standard input
- Standard output
- Standard error
A program can open other data streams besides these ...
such as connections to files
File descriptors represent data streams or files that your script can access
Each file descriptor is assigned a positive number, starting with 0
Think of a file descriptor as an integer that refers to a file
Standard input, Standard output and Standard error each have their own file descriptors

Name File Descriptor

Standard input 0

Standard output 1

Standard error 2
While we think of standard input, standard output and standard error
Unix thinks of the file descriptors 0, 1 and 2

Name	File Descriptor
Standard input	0
Standard output	1
Standard error	2

Redirecting Standard Error

Standard error is the data stream into which error messages are sent
Redirecting standard error allows a program to separate error messages ...
from its output stream
To redirect standard input we use the less than symbol, <

Followed by a pathname

$ ./repeat.sh < test.txt 
Enter several lines
Type X on a line by itself when done

You entered
-----------
123456789
abcdefg
987654321
hijklmnop
foo
bar
bletch
X

This construction is really a shorthand ...
f or a notation using file descriptors

When you type

./repeat.sh < test.txt

Unix thinks of this as

./repeat.sh 0< test.txt

Where 0 is the file descriptor for standard input

Similarly, when we use output redirection

$ echo "Hello there"  >  hello.txt

Unix thinks of this as meaning

$ echo "Hello there"  1>  hello.txt

Again the file descriptor precedes the redirection symbol, >
So how do we redirect standard error?

We place a 2 in front of the greater than symbol

$ ls xxxx
ls: cannot access xxxx: No such file or directory

$ ls xxxx 2> error.txt

$ cat error.txt 
ls: cannot access xxxx: No such file or directory

Remember, 2 is the file descriptor for standard error
Unix also gives you a way to redirect both standard output ...
and standard error ...
to the same file or device
You can do this using the ampersand and greater than symbols together, &>

Shell Scripts

A shell script is a file that contains Unix commands
When the shell script is run ...
each line of the script is executed in turn
A shell script can use any shell feature that is available at the command line
- Ambiguous file references using the metacharacters ?, * and [ ]
- Redirection
- Pipes
But not those features which are provided by tty
- Command line editing (arrow keys, control key combinations)
- Pathname completion (hit Tab to get more of a filename)
- The history mechanism (up arrow to recall previous command line)
Unix also provides control structures
- Conditionals
- Loops

Making a Shell Script Executable

You must have both read and execute permission to run a shell script ...
without explicitly calling bash
You need read permission because the kernel has to read the script into RAM
You need execute permission so the script can be run without calling bash
If you run a script without calling bash ...
and do no assign both read and execute permissions ...
you will get an error
Normally you would give a shell script file 755 permissions
The owner can read, write and execute
The group and everyone else can read and execute
All scripts for this course from now on must have 755 permissions set
Points will be deducted from homework 11 if you don't

Specifying Which Version of the Shell Will Run a Script

Every process has to have the binary code for some program
What binary code is loaded when you run a shell script?
The script is itself is text not binary code
And the CPU cannot understand text
Shells are the only binary code that can understand a shell script
To run a script the kernel must load both the script ...
and the code for a shell ...
into the process RAM
When you run a script picture in memory looks like this

But there are different versions of the shell
So what version of the shell is used when you run a script?
If you do not tell the kernel which shell version to load to run the script ...
it will load the code for the version of the shell you are currently running ...
into the process RAM
But Unix gives you an alternative
So you can tell the kernel to load the code for a different shell version ...
into process RAM
You do this with a hashbang line
The hashbang line must be the first line of the script
The first two characters on the line must be #!
After this comes absolute pathname ...
of the version of the shell which will run the script
! is sometimes called "bang" in Unix
The hashbang line we will use in scripts is
```
#! /bin/bash
```
The hashbang must be used with scripts written in scripting languages ...
like Python or Perl ...
if you want to make them executable

Comments in Shell Scripts

Programs are written by people for machines
But programs also have to be read by the people
- Who write the program
- Who maintain the program
- Who use the program
To make clear what is happening inside a program use comments
Anything following a hash mark, # , is a comment
Except for the hashbang line
Comments are a way to document a program inside the code itself
This sort of documentation is extremely important
It is good practice to place a comment at the top of the shell script ...
after the hashbang line
This comment should say what the script does
You should also comment any part of a script ...
that does something hard to understand

Attendance

New Material

Variables

A variable is a place in memory with a name ...
that holds a value
Variables are used to store values that will be needed later
Bash uses number of shell variables
They are very important in making it work properly
These variables are called keyword shell variables ...
or just keyword variables
A keyword is a word ...
with special meaning to the shell
You can change the values of some of these variables ...
to customize your Unix environment
For example, I have changed the value of PS1
It controls the prompt Bash prints when it is ready for the next command
You can create your own variables to hold values that you find useful
For example, I have defined the variable lncd
It holds the absolute address of our class directory
So if I need to go to the class directory, I can simply type
```
cd $lncd
```
To get the value of a variable you put a $ ...
in front of the name of the variable

Creating a Variable

You create a variable using the following format
```
VARIABLE_NAME=VALUE
```

Like this

$ username=ghoffman

$ echo $username
ghoffman

There must be no spaces on either side of =

If you include spaces you will get an error

$ username = ghoffman
username: command not found

The value you assign to a variable must not contain a space
```
$ name=Glenn Hoffman
Hoffman: command not found
```
There are two ways to deal with this
- Escape the whitespace characters with a \
- Quote the entire value
The backslash character turns off the special meaning ...

of the character that follows it

$ name=Glenn\ Hoffman

$ echo $name
Glenn Hoffman

Of course you must put the backslash in front of every whitespace character

If you don't, you will get an error

$ team=Boston\ Red Sox
No command 'Sox' found

For this reason it is usually better to quote the entire value

You can use either a single quote '

$ team='Boston Red Sox'

$ echo $team
Boston Red Sox

or a double quote "

$ team="Boston Red Sox"

$ echo $team
Boston Red Sox

There is a difference between the two quotes ...
but we will save that for another class
Be sure to use the same quote at the end of the value ...
that you use at the start

Scope

The place in which a variable can be used is called the scope
Variables have two scopes
- Local
- Global
Local variables only have meaning in the shell in which they are defined
They have no meaning in any other shell ...
or subshell
A local variable defined in your login shell will only work there
It will have no meaning inside the shell running a script
Global variables are variables defined in one shell
But they have meaning in all sub-shells created from that shell
That means you can define a global variable in your login shell ...
and that variable can be used in any script you run

Local Variables

By default every shell variable you create is a local variable
You must do something special to make it a variable global
If I create the variable name ...

I can use it in that shell

$ name='Glenn Hoffman'

$ echo $name
Glenn Hoffman

But not in a subshell

$ bash

ghoffman@itserver6:~$ ps
  PID TTY          TIME CMD
29566 pts/16   00:00:00 bash
29792 pts/16   00:00:00 bash
29796 pts/16   00:00:00 ps

ghoffman@itserver6:~$ echo $name

This means we cannot use a local variable defined in your shell ...
inside a script

Here is a script that simply prints the value of the variable name

$ cat print_name.sh 
#! /bin/bash

#prints the value of the variable called name

echo name = $name

When I run this script I get
```
$ ./print_name.sh 
name =
```
Why?
Because to run a script a sub-shell must be created
And that shell cannot see local variables ...
defined in another shell

Global Variables

To make a variable global use the export command
You can run export on a variable you have already created
```
$ export name

$ ./print_name.sh 
name = Glenn Hoffman
```
Notice that I did not put a $ in front of the name
That is because I am not changing its value
More often you use export when you are defining the variable
```
$ export name="Glenn Hoffman"
```
If you do not use export when defining a variable ...
it will be a local variable
All keyword shell variables like PATH ...
are global variables

Choosing the Scope of a Variable

In general, local variables are only used inside shell scripts
For example in all my test scripts I define the variable ex_no
```
ex_no=18
```
The value of this variable is different for each test script
But this variable has no use outside this script ...
so there is no need to make it global
But every test script for this course ...
needs to know the absolute pathname of the IT 244 course directory
Because that is where your Class Exercise scripts are located
I want to define this variable once so I can use it in every script ...
so I make it global
```
export lncd=/courses/it244/s19/ghoffman
```

Defining Global Variables

You can define local variables inside a script
And they will be available every time you run the script
What about global variables?
You could define them at the command line
But they will disappear as soon as you log out
To make them always available, define them in .bash_profile ...
the startup file in your home directory
It is automatically run after you log in ...
but before you get a prompt
This file contains commands you use to customize your Unix environment

Keyword Shell Variables

Keyword shell variables are needed to make the shell run properly
They are also called keyword variables
They have short, mnemonic names
By convention, the names of keyword variables are always CAPITALIZED
Most keyword variables can be changed by the user
This is normally done in the startup file .bash_profile
When you change your PATH variable ...
you a want to keep the current value of the variable ...
but add something to it
You can do this with a simple trick
Use the current value of the variable when defining its new value ...
but follow it with the address of the new directory
To add the directory that holds my shell scripts to PATH I would write
```
PATH=$PATH:/home/ghoffman/bin/shell
```
Notice the : between the list of directories in $PATH ...
and the directory I have added

Important Keyword Shell Variables

There are a number of keyword variables that affect your Unix session

Some of the more important are

Variable	Value
HOME	The absolute pathname of your home directory
PATH	The list of directories the shell will search when looking for the executable file associated with a command you entered at the command line
SHELL	The absolute pathname of your default shell
PWD	The absolute pathname off your current directory
PS1	Your command line prompt - what you see after entering each command
PS2	The secondary prompt - what you see if you continue a command to a second line

User-created Variables

User-created variables are any variables you create
By convention, the names of user-created variables are lower case
```
$ lncd=/courses/it244/s19/ghoffman
```
User-created variables can be either local or global in scope
The variables I create in my scripts are user-created variables
But I have many global variables

I define a global variable for the class directories of each class I teach

export lncd=/courses/it244/s19/ghoffman
export s1cd=/courses/it116/s19/ghoffman
export s2cd=/courses/it117/s19/ghoffman

I also define global variables for the pathnames ...

of the directories in which I keep my test scripts

export lntst=/home/ghoffman/code/it244_code/testing_scripts_it244
export s1tst=/home/ghoffman/code/it116_code/testing_scripts_it116
export s2tst=/home/ghoffman/code/it117_code/testing_scripts_it117

This means that to go to my test directory for this course I can type
```
cd $lntst
```

Instead of

cd /home/ghoffman/code/it244_code/testing_scripts_it244

The `read` command

Most scripts need input from the user
One way to get this input is with the read command
read is used to set the value of a local variable
It uses the following format
```
read VARIABLE_NAME
```
When bash gets to this line in a script it pauses ...
and waits for the user to type something
When the user hits Enter or Return ...
The value entered is assigned to the variable
Here is a simple script that uses read
```
#! /bin/bash

read value
echo $value
```
When I run this I get
```
$ ./read_1.sh 
hello
hello
```
What's wrong with this script?
It does not tell us that we need to enter a value ...
and then hit Enter or Return
We can fix this problem by using echo to print a prompt ...
which is text telling the user they need to enter a value
But there is an easier way to get a prompt
If you run read with the -p option it will print a prompt
```
read -p PROMPT VARIABLE_NAME
```

Here is an improved version of the script above

#! /bin/bash

read -p "Please enter a value: " value
echo $value

When I run this I get

$ ./read_2.sh
Please enter a value: hi there
hi there

`env` - Show All Global Variables

There is a Linux command that will show you every global variable
Running env without an argument ...

prints the values of all global variables

$ env
TERM=xterm-color
SHELL=/bin/bash
SSH_CLIENT=66.92.76.9 53785 22
OLDPWD=/home/it244gh
SSH_TTY=/dev/pts/8
USER=it244gh
...

Separating and Grouping Commands

Every time you type on the command line and hit Enter ...
the shell tries to execute your command line entry
So far, we have only run one command from the command line
Except for pipes
You can enter more than one command on the command line

But you must put a semicolon, ; between them

$ echo Here are the contents of my home directory ; ls -l ~ ; echo
Here are the contents of my home directory
total 40
drwxr-xr-x  4 ghoffman grad    4096 Oct 15  2016 bin
drwxr-xr-x  6 ghoffman faculty 4096 Oct 23  2016 code
drwxr-xr-x  6 ghoffman faculty 4096 Sep  9  2016 course_files
drwxr-xr-x  7 ghoffman grad    4096 May 25 10:53 html
lrwxrwxrwx  1 root     root      38 May 30 21:10 it116 -> /courses/it116/s19/ghoffman/ghoffman
lrwxrwxrwx  1 ghoffman faculty   52 Feb 20 09:43 it116_test -> /home/ghoffman/code/it116_code/testing_scripts_it116
lrwxrwxrwx  1 ghoffman faculty   38 May 30 16:59 it244 -> /courses/it244/s19/ghoffman/ghoffman
lrwxrwxrwx  1 ghoffman faculty   52 Feb  3 09:10 it244_test -> /home/ghoffman/code/it244_code/testing_scripts_it244
drwxr-----  2 ghoffman faculty 4096 Mar 27 09:43 mail
drwxrwxr-x 10 ghoffman faculty 4096 Jul  6 08:51 public_html
drwxrwxr-x  4 ghoffman faculty 4096 Jun 12 08:40 submitted
drwxr-----  4 ghoffman faculty 4096 Jun 23 11:38 tests_taken
drwxr--r--  2 ghoffman faculty 4096 Jul 11 17:55 tmp
drwxr-----  4 ghoffman faculty 4096 Sep 22  2016 xrchiv

When you hit Enter each command is executed
In the order it was typed at the command line

| (pipe) and & (ampersand) as Command Separators

The semicolon, ; , is a command separator
It separates multiple commands on a single command line
The pipe character, | , and the ampersand, & , are also command separators
A pipeline is a group of commands with at | in between
Each command takes its input from the previous command
Each command is a separate process ...
even though the pipeline is a single job
We use an ampersand, &, after a command ...
to make the command run in the background
```
./bother.sh  &
```
When we do this, two things happen
- The command is disconnected from the keyboard
- The shell gives you back a prompt
But the ampersand is also a command separator

So we can use it to run more than one command at the same time

$ ./bother.sh > /dev/null  & ./bother.sh > /dev/null & ./bother.sh > /dev/null & jobs
[1] 1794
[2] 1795
[3] 1796
[1]   Running                 ./bother.sh > /dev/null &
[2]-  Running                 ./bother.sh > /dev/null &
[3]+  Running                 ./bother.sh > /dev/null &

Here I created three jobs with one command line
Notice that I did not run the last command in the background

Continuing a Command onto the Next Line

Unix will let you type as long a command line as you like
If you reach the end of your session window your ...

text will wrap to the next line

$ echo asdfasdfasdfasdfasdfasdfasdf
asdfasdfasdfasdfasdfasdfasdfasdfasd
fasdfasdfasdfasdfasdfasdfasdfasdfas
dfasdfasdfasfads 
asdfasdfasdfasdfasdfasdfasdfasdfasd
fasdfasdfasdfasdfasdfasdfasdfasdfas
dfasdfasdfasdfasdfasdfasdfasdfasdfa
sdfasfads

If you then expand the window, you can see more text on each line

$ echo asdfasdfasdfasdfasdfasdfasdfasdfasdfasdfasdfas
dfasdfasdfasdfasdfasdfasdfasdfasdfasdfasdfasdfasdfasd
fasdfasdfasfads 
asdfasdfasdfasdfasdfasdfasdfasdfasdfasdfasdfasdfasdfa
sdfasdfasdfasdfasdfasdfasdfasdfasdfasdfasdfasdfasdfas
dfasfads

Unix thinks that the text above is only two lines
- Your command line entry
- The one line output
But sometimes it helps to break a long command into more than one line
You can do this by typing a backslash, \ ...

followed immediately by the Enter key

$ echo A man \
> A plan \
> A canal \
> Panama
A man A plan A canal Panama

Here we are escaping the the newline character at the end of the line
Escaping turns off the special meaning of a character
The backslash above turns off the special meaning of newline
Normally, the shell runs the command when it sees a newline
The newline character is sent when you hit Enter on a PC ...
or Return on a Mac
This trick won't work if you put a space before the newline ...
because backslash only escapes the character immediately following it
The > symbol you see on the next line is a secondary prompt
This symbol means the shell knows you are not done with you command ...
and is waiting for you to type more
The normal prompt is your primary prompt
You get the primary prompt when the shell is waiting for a new command
You get the secondary prompt ...
when the shell is waiting for more of your original command

Using Parentheses, ( ) , to Run a Group of Commands in a Subshell

Sometimes, you want to run a group of commands in a shell of it's own
You can do this by putting the commands within parentheses
```
( work1.sh ; work2.sh )
```
I learned this feature of Unix when I first taught this course
I thought I would never need to use it
But I was wrong
Sometimes I need to loop through several long files ...
running less on each
But I also need to know the filename ...
as each file appeared in the output
I realized I could do this if I combined the output of two commands
- echo to print the name of the file
- cat to print the contents of the file
Once I had this combined output I could pipe the results into less ...
like this
```
(echo $FILE; cat $FILE) | less
```

The Directory Stack

The cd command has no memory
Once you use cd ...
it forgets where it has been
In Unix you will often need to go back to a previous directory
Doing this can be a pain
You might have to type a long pathname
Even worse, you might forget where you were
Bash provides a mechanism ...
to make it easier to retrace your steps
It does this using something called a stack
A stack is an arrangement of data inside a program
In a stack, data is stored in the order in which it came in
Stack are used frequently in programming ...
when you want to return to a previous state
A stack is an example of a data structure
A data structure is an arrangement of data ...
inside the RAM of a running program
This structure makes certain operations work more efficiently
A stack is a LIFO data structure
Which stands for Last In First Out
The directory stack is part of Bash
But you must use special commands to make use of it
There are three of these commands
- pushd
- popd
- dirs
All three of these commands are built-ins

`pushd` - Pushes a Directory onto the Stack

In order to use the directory stack you must first use pushd
In programming, putting something onto a stack is called a push
pushd changes your current directory
Just like cd
But it also adds your new directory to the directory stack
When used with an argument, pushd
- Places the new directory on the stack
- Displays the current contents of the directory stack
- Moves to the new directory
You can also see what is in the directory stack
Using the dirs command

Let's look at an example

$ pwd
/home/ghoffman/tmp

$ dirs
~/tmp

$ pushd /home/ghoffman/course_files/it244_files
~/course_files/it244_files ~/tmp

$ dirs
~/course_files/it244_files ~/tmp

`popd` - Pops a Directory off the Stack

To return to a previous directory, you have to use popd
In programming, removing a value from a stack is called a pop
popd changes your current directory to the directory on top of the stack
It also removes that directory from the stack
When used without an argument popd
- Removes the top directory from the stack
- Prints the current stack
- Goes to the directory that is now on top of the stack

`dirs` - Displays the Directory Stack

dirs is a Unix command that prints the directory stack
The directory stack is used by pushd or popd
What happens to the directory stack if you never use these commands?
You might thinks that the stack would never change
But you would be wrong
If you only use cd the stack will only have one entry
That entry will be your current directory
But your current directory changes each time you use cd
You can see this by using dirs command
When I first log in, I am in my home directory
dirs shows this as the only entry in the stack
```
$ dirs
~
```
dirs always uses a tilde, ~ for your home directory
I can then use cd to go to a new directory
When I now run dirs it will only show the current directory
```
$ cd tmp

$ dirs
~/tmp
```

Review

New Material

Homework 8

Quiz 6

Hashbang Line and Comments in Homework Assignments

Questions

Review

Running a Startup File after a Change has been Made

Commands that are Symbols

File Descriptors

Redirecting Standard Error

Shell Scripts

Making a Shell Script Executable

Specifying Which Version of the Shell Will Run a Script

Comments in Shell Scripts

Attendance

New Material

Variables

Creating a Variable

Scope

Local Variables

Global Variables

Choosing the Scope of a Variable

Defining Global Variables

Keyword Shell Variables

Important Keyword Shell Variables

User-created Variables

The read command

env - Show All Global Variables

Separating and Grouping Commands

| (pipe) and & (ampersand) as Command Separators

Continuing a Command onto the Next Line

Using Parentheses, ( ) , to Run a Group of Commands in a Subshell

The Directory Stack

pushd - Pushes a Directory onto the Stack

popd - Pops a Directory off the Stack

dirs - Displays the Directory Stack

Class Exercise

Class Quiz

The `read` command

`env` - Show All Global Variables

`pushd` - Pushes a Directory onto the Stack

`popd` - Pops a Directory off the Stack

`dirs` - Displays the Directory Stack