IT 244: Introduction to Linux/Unix

IT 244: Introduction to Linux/Unix
Class 16

Microphone

Homework 7

I have posted homework 7 here.

It is due this Sunday at 11:59 PM.

Midterm

I have started scoring the Midterm.

I hope to be finished by Monday.

Graded Quiz Next Week

There will be a graded quiz next week.

It will cover the material we will discuss today, and the material from Class 13.

Questions

Are there any questions before I begin?

Tips and Examples

Killing All Running Jobs

A student once asked whether it is possible to kill all currently running jobs
I did some googling and the answer is yes
You can use command substitution
We will discuss this feature in an upcoming class
Command substitution has the format
```
$(COMMAND)
```
The shell creates a process to run the command inside the parentheses
Then it replaces the dollar sign and parenthetical expression ...
with the output of the command
When jobs is run with the -p option it returns the process numbers of the currently running jobs
```
    $ jobs -p
    15521
    15523
    15525
```
If you use command substitution with this command ...

you can kill all running jobs

    $ jobs
    [1]   Running                 ./bother.sh > /dev/null &
    [2]-  Running                 ./bother.sh > /dev/null &
    [3]+  Running                 ./bother.sh > /dev/null &

    $ jobs -p
    15579
    15582
    15585

    $ kill $(jobs -p)

    $ jobs
    [1]   Terminated              ./bother.sh > /dev/null
    [2]-  Terminated              ./bother.sh > /dev/null
    [3]+  Terminated              ./bother.sh > /dev/null

    $ jobs

    $

Seeing All Running Processes

If you run ps you will see all running processes

Even those running in the background

    $ jobs
    [1]+  Running                 ./bother.sh > /dev/null &

    $ ps
      PID TTY          TIME CMD
    18389 pts/2    00:00:00 bash
    19223 pts/2    00:00:00 bother.sh
    19229 pts/2    00:00:00 sleep
    19230 pts/2    00:00:00 ps

But what if you had more than one terminal session going?
Normally ps only shows you process running in your current session

To see processes running in all sessions use the -u option

    $ ps -u
    USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
    ghoffman  18389  0.0  0.1  23440  4768 pts/2    Ss   13:53   0:00 -bash
    ghoffman  19151  0.0  0.1  23312  4528 pts/3    Ss+  14:55   0:00 -bash
    ghoffman  19223  0.0  0.0  12424  1412 pts/2    S    14:59   0:00 /bin/bash ./bother.sh
    ghoffman  19306  0.0  0.0  12424  1396 pts/3    S    15:02   0:00 /bin/bash ./bother.sh
    ghoffman  19309  0.0  0.0   7196   620 pts/3    S    15:02   0:00 sleep 5
    ghoffman  19310  0.0  0.0   7196   620 pts/2    S    15:02   0:00 sleep 5
    ghoffman  19311  0.0  0.0  18448  1324 pts/2    R+   15:02   0:00 ps -u

Review

Running a Command in the Background

Normally, when you run a command you have to wait for it to finish
Such commands are said to be running in the foreground
Anything you run in the foreground can take input from the keyboard
But what if you ran a command that took a long time to finish?
Unix gives you a way to get the command prompt back ...
after running a command
You can run the command in the background
When you run a command in the background two things happen
The background job loses it's connection to the keyboard ...
and the shell will not go to sleep
The shell will give you a prompt immediately ...
so you can run another command
The shell will tell you when the background job has finished
Every time a program runs, a process is created
The process has access to system resources
Like memory (RAM) and a connection to data streams
Unix, like most operating systems, is multitasking
This means you can have more than one process running at a time
To run a command in the background type an ampersand, &, ...
at the end of the command line
```
    $ sleep 5 &
    [1] 17895
    $
```

Jobs

Every time you hit Enter on the command line you are creating a job
Usually a job consists of a single command
But not always
Every time a program runs ...
a process is created for that program
A pipeline is a collection of commands joined by pipes
Each command will have its own process
But the collection of all the separate processes is a single job
Each process in a pipeline will have its own process ID
As the pipeline progresses ...
the currently running process will change ...
but the job number does not change
The job is the collection of all processes created at the command line
You can have multiple jobs running at the same time
But only one job can be in the foreground at any one time
Every process has a unique process ID number
And every job has a job number
When you run a job in the background bash returns two numbers
```
    $ sleep 5 &
    [1] 7431
    $
```
The job number is enclosed in brackets and comes first
The second, larger, number is the process identification number ...
of the first process in the job
The process identification number is also known as the PID
When the job finishes, the shell prints out a message
```
[1]+  Done                  sleep 5
```
The message does not appear the moment the job finishes
The shell waits for the next time you hit Enter
Then it prints the message after the output from the command

Moving a Job from the Foreground into the Background

There can only be one foreground job
Though you can have many background jobs
Unix will let you move a job from the foreground to the background
To do this, you must first suspend the foreground job
A suspended job is not dead
It is in a state of suspended animation
You can reactivate it later
To suspend a foreground job you must use the suspend key sequence
On our systems you suspend a job by hitting Control Z
After you do this, the shell stops the current process
It also disconnects it from the keyboard
A suspended job can be run in the background ...
by using the bg command
bg stands for background
Once placed in the background, the job resumes running
If more than one job is suspended you must give bg the job number

Aborting a Background Job

There are two ways to abort a background job
You can bring a job from the background to the foreground ...
using the fg (foreground) command
When there is more than one job in the background ...
you must supply the job number
Once you have the job in the foreground you can abort it using Control C
You can also terminate any job using the kill command
But to use kill you must tell it what to kill
The usual way to do this is to give kill a process ID
If you don't remember the process ID run ps (process status)
You can also use the job number with kill
But you must precede a job number with a percent sign, %
You can get the job number using the jobs command

Attendance

New Material

Pathname Expansion

What if you wanted a long listing of files whose names started with "example"?
If would be painful to type all the names, one at a time
Fortunately, Unix provides a better way
This is a feature called pathname expansion
The it is also sometimes called globbing
Pathname expansion uses meta-characters
Meta-characters are sometimes called wildcards
They allow you to specify a pattern
When the shell sees such a pattern on the command line
It does something before executing the command
The shell replaces the pattern with a sorted list of all pathnames ...
that match the pattern
Then it runs this altered command line
The pattern is called an ambiguous file reference
What happens if the shell finds no matching pathnames?
It passes the ambiguous file reference to the program called
The shell lets the program try to make sense of the pattern
Pathname expansion is an operation performed by bash
It is done before bash asks the kernel to run the program
You can use as many meta-characters as you want to form a pattern
Pathname expansion lets you specify a set of files ...
with a minimum amount of typing
It also comes in handy when you can't remember the exact pathname
Pathname expansion is different from pathname completion ...
which you get by hitting Tab
Pathname completion only gives you one pathname
Pathname expansion can create several pathnames with one pattern
Pathname completion is an operation handled by the tty device driver
Pathname expansion is performed by the shell

The ? Meta-character

The question mark, ?, meta-character stands for any single character
If I wanted a long listing of everything whose names begin with "dir" ...

followed by any single characters I could use

    $ ls -ld dir?
    drwxrwxrwx 2 it244gh libuuid 512 2011-09-30 15:26 dir1
    drwxr--r-- 2 it244gh libuuid 512 2011-09-30 15:26 dir2
    drwxrw---- 2 it244gh libuuid 512 2011-09-30 15:29 dir3
    drwxrw---- 2 it244gh libuuid 512 2011-09-30 15:29 dir4

Meta-characters work with any command
```
    $ echo dir?
    dir1 dir2 dir3 dir4
```
The ? meta-character does not match a leading period in a filename
You must explicitly enter a leading period, . ...
when trying to match an "invisible" file

The * Meta-character

An asterisk, * , will match any number of characters in a pathname
It will even match no characters

To find all the entries with names beginning with "dir" we can use the * meta-character

    $ ls -ld dir*
    drwxr-xr-x 2 it244gh libuuid 512 2011-10-04 13:52 dir
    drwxrwxrwx 2 it244gh libuuid 512 2011-09-30 15:26 dir1
    drwxr-xr-x 2 it244gh libuuid 512 2011-10-04 13:53 dir10
    drwxr--r-- 2 it244gh libuuid 512 2011-09-30 15:26 dir2
    drwxrw---- 2 it244gh libuuid 512 2011-09-30 15:29 dir3
    drwxr-xr-x 2 it244gh libuuid 512 2011-10-02 17:07 dir4

Notice that * returns more names than ?
It returned dir
Which has no additional characters after the string "dir"
And it returned dir10
Because it will accept any number of characters
Note also that dir10 appears before dir2
This is because the list is sorted alphabetically
* can be used with any command

Even those that don't normally deal with files

    $ echo dir*
    dir dir1 dir10 dir2 dir3 dir4

* cannot be used to match the initial period, . ...
in a "hidden" filename

The [ and ] Meta-characters

The square brackets, [ and ], are also meta-characters
They work somewhat like the ?
They only match a single character in a pathname
But the pathname character must match one ...
of the characters within the brackets
If I wanted a long listing of directories named dir1, dir2 and dir3 ...
but wanted to omit dir4 ...

I could use

    ls -ld dir[123]
    drwxrwxrwx 2 it244gh libuuid 512 2011-09-30 15:26 dir1
    drwxr--r-- 2 it244gh libuuid 512 2011-09-30 15:26 dir2
    drwxrw---- 2 it244gh libuuid 512 2011-09-30 15:29 dir3

This meta-character only matches a single character
It does not matter how many characters are within the brackets
You can use the bracket meta-characters with any program
```
    $ echo dir[123]
    dir1 dir2 dir3
```
You can use a range to avoid explicitly listing all characters
A range is specified by listing the first and last characters of a sequence
The characters must be separated by a dash, -

The the upper and lower limits of the range must be in ascending order

    ls -ld dir[1-3]
    drwxrwxrwx 2 it244gh libuuid 512 2011-09-30 15:26 dir1
    drwxr--r-- 2 it244gh libuuid 512 2011-09-30 15:26 dir2
    drwxrw---- 2 it244gh libuuid 512 2011-09-30 15:29 dir3

The square brackets provide another shortcut
You can use [ ] to match any character not between the brackets
This is done by putting an exclamation mark, ! , or a caret, ^ ...
immediately after the opening bracket
The shell will match any single character NOT included within the brackets
```
    $ echo foo[!46]
    foo1 foo2 foo3 foo5 foo7 foo8 foo9
```

Builtins

Not all commands can be found on disk as executable files
Some are actually contained in the shell itself
Such commands are called builtins
When you run a builtin ...
the shell does not have to create a new process
Instead, the shell calls a procedure in its own code to perform the task
This means no process is created ...
which makes execution faster
What if there is an executable file with the same name as the builtin?
In that case the shell will run the builtin instead of the file
echo is a builtin
There is also an executable version of echo on the disk
```
    $ which echo
    /usr/bin/echo
```
If you want to run the disk version of echo you have to specify the pathname
```
    $ /usr/bin/echo foo
    foo
```
Most builtins have no executable counterpart on disk
Running which on such a command will find nothing
```
    $ which bg
    $
```
There is a Linux command that will tell you if a command is a builtin
The command is type ...
and you use it like this
```
type COMMAND_NANE
```

Here are some examples

$ type bg
bg is a shell builtin

$ type cat
cat is /bin/cat

$ type pwd
pwd is a shell builtin

$ type cp
cp is /bin/cp

$ type type
type is a shell builtin

Help for Builtins

If you run man on a builtin command you will get nothing
```
$ man bg
No manual entry for bg
```
info also has no information about builtins
But there is a command that you can use with builtins
To learn something about the commands
The command is help
And it is, itself, a builtin
```
$ type help
help is a shell builtin
```
If you follow help with the name of a builtin ...

it will give you information on that command

$ help bg
bg: bg [job_spec ...]
    Move jobs to the background.

    Place the jobs identified by each JOB_SPEC in the background, as if they
    had been started with `&".  If JOB_SPEC is not present, the shell"s notion
    of the current job is used.

    Exit Status:
    Returns success unless job control is not enabled or an error occurs.

help does not work for normal commands

$ help ls
-bash: help: no help topics match `ls".  Try `help help" or `man -k ls" or `info ls".

If you run help with no argument ...

it will show all the builtin commands

$ help 
GNU bash, version 4.3.11(1)-release (x86_64-pc-linux-gnu)
These shell commands are defined internally.  Type `help" to see this list.
Type `help name" to find out more about the function `name".
Use `info bash" to find out more about the shell in general.
Use `man -k" or `info" to find out more about commands not in this list.

A star (*) next to a name means that the command is disabled.

 job_spec [&]                      history [-c] [-d offset] [n] or history -anrw [filename>
 (( expression ))                  if COMMANDS; then COMMANDS; [ elif COMMANDS; then COMMA>
 . filename [arguments]            jobs [-lnprs] [jobspec ...] or jobs -x command [args]
 :                                 kill [-s sigspec | -n signum | -sigspec] pid | jobspec >
 [ arg... ]                        let arg [arg ...]
 [[ expression ]]                  local [option] name[=value] ...
 alias [-p] [name[=value] ... ]    logout [n]
 bg [job_spec ...]                 mapfile [-n count] [-O origin] [-s count] [-t] [-u fd] >
...

Different Shell Versions

The shell is a program that gives you the command line ...
and allows you to run programs
The shell we have been using is bash
Bash stands for Bourne again shell
The first Unix shell was written by Steve Bourne at AT&T's Bell Laboratories
This shell is sometimes called the Bourne shell
But the command used to run the shell was sh
System V Unix introduced the Korn shell, ksh
It was written by David Korn
It introduced aliases and command line editing
It also introduced other features that are now found in bash
Machines running Ubuntu have another shell called dash
dash is a stripped down shell with very few user features
dash is a POSIX-compliant version of sh that is as small as possible
I'll talk more about POSIX in a few minutes
dash was designed to run shell scripts ...
but not to be used at the command line
Another popular shell is the C shell
C programmers like it because it's scripting language looks like C
There are two versions of this shell
- csh
- tcsh
But the difference between them is slight
Both the Korn and C shells cannot run all bash scripts ...
because there slight differences between them and bash

Using a Different Shell

Different version of Unix and Linux have different shells installed
To find out what shells are available on a machine ...
look at the text file /etc/shells

Here are the shell available on pe15

$ cat /etc/shells
# /etc/shells: valid login shells
/bin/sh
/bin/bash
/usr/bin/bash
/bin/rbash
/usr/bin/rbash
/bin/dash
/usr/bin/dash
/usr/bin/tmux
/usr/bin/screen
/bin/tcsh
/usr/bin/tcsh

But if we look a little closer we will see that this list is misleading

$ ls -l /bin/*sh
-rwxr-xr-x 1 root root 1183448 Jun 18  2020 /bin/bash
...
-rwsr-xr-x 1 root root   53040 May 28  2020 /bin/chsh
...
lrwxrwxrwx 1 root root      21 Dec  2 13:30 /bin/csh -> /etc/alternatives/csh
-rwxr-xr-x 1 root root  129816 Jul 18  2019 /bin/dash
...
lrwxrwxrwx 1 root root       4 Jun 18  2020 /bin/rbash -> bash
...
lrwxrwxrwx 1 root root      21 Jul 31  2020 /bin/rsh -> /etc/alternatives/rsh
lrwxrwxrwx 1 root root       4 Jul 31  2020 /bin/sh -> dash
-rwxr-xr-x 1 root root  789448 Mar  9 09:17 /bin/ssh
...
-rwxr-xr-x 1 root root  447896 Jul 16  2019 /bin/tcsh
...

Because csh, rbash and sh are just links

Here are the shells installed on my Mac

$ cat /etc/shells
# List of acceptable shells for chpass(1).
# Ftpd will not allow users to connect who are not using
# one of these shells.

/bin/bash
/bin/csh
/bin/dash
/bin/ksh
/bin/sh
/bin/tcsh
/bin/zsh

Unlike pe15 none of these are links

$ ls -l /bin/*sh
-r-xr-xr-x  1 root  wheel   623472 Sep 21  2020 /bin/bash
-rwxr-xr-x  1 root  wheel   529424 Sep 21  2020 /bin/csh
-rwxr-xr-x  1 root  wheel   110848 Sep 21  2020 /bin/dash
-r-xr-xr-x  1 root  wheel  1300256 Sep 21  2020 /bin/ksh
-rwxr-xr-x  1 root  wheel    31440 Sep 21  2020 /bin/sh
-rwxr-xr-x  1 root  wheel   529424 Sep 21  2020 /bin/tcsh
-rwxr-xr-x  1 root  wheel   637840 Sep 21  2020 /bin/zsh

POSIX

Before Linux came along there were many different Unixes
They were each developed by different companies such as
- IBM
- HP
- Sun Microsystems
Each of these versions of Unix was slightly different ...
and the companies had no incentive to change this
Into this mess steped the IEEE (Institute of Electrical and Electronics Engineers)
They could not force the companies to play nice
So they developed a standard for how different version of Unix should work
This standard is called POSIX (Portable Operating System Interface) 1003.2
So anyone who uses a POSIX compatible shell can know what to expect
This is particularly important for people writing install scripts ...
that are used to install big software packages
sh is fully POSIX compliant but bash is not ...
due to certain design decisions made when bash was created
You can make bash somewhat compatible with the POSIX standard ...
if you run it with the --posix option

Shells For Running Scripts

Many scripts are needed to set up and maintain Linux and Unix
Many of these scripts are quite old ...
and were written before bash was created
They were written to run under sh
There are subtle differences between different shells
It is always best to run a script in the shell for which it was written
Otherwise the script may not run properly
Installer scripts for Unix packages are written for a POSIX compatible shell
Like sh
When you boot up Unix or Linux many scripts have to run
They are needed to configure different services
Scripts used to boot the original versions of Ubuntu ran under bash
But bash is very big ...
due to the many features that make it easier to use
It is also relatively slow
This meant it took Ubuntu a long time to boot
To improve the boot time Ubuntu switched its scripts to dash
dash was designed to be POSIX compatible
It only contains features needed to run scripts
Boot times improved dramatically after the switch

dash is much smaller than bash

$ ls -lh /bin/bash
-rwxr-xr-x 1 root root 1014K May 16  2017 /bin/bash

$ ls -lh /bin/dash
-rwxr-xr-x 1 root root 151K Feb 17  2016 /bin/dash

Its memory footprint is small ...
so it loads and executes scripts much faster than bash

Ways a Shell Can Be Created

There are three ways a user can run a shell
- Login shell
- Interactive non-login shell
- Non-interactive shell
There are subtle differences between these three types of shells
We'll concentrate on login shells in this course
But you should be aware of the other ways a shell can be run

Your Login Shell

When you first login to Unix, you are running a shell
This shell is your login shell
Which shell version you run is determined by the SHELL system variable
```
$ echo $SHELL
/bin/bash
```
In Ubuntu, the default shell version is Bash
When your login shell starts up it runs the commands found in /etc/profile
This is a file customized by the system administrator for all users
You can create your own customizations in a startup file in you home directory
That file must have one of these names
- .bash_profile
- .bash_login
- .profile
You can have each of these startup files in your home directory ...
and bash will run each one
They will be run in the order given above
We will only use .bash_profile in this course

Interactive Non-login Shells

The shell is a program, just like cat or ls
You can run another shell as a sub-shell of your current shell

You do this by typing the name of the shell at the command line

    $ ps
      PID TTY          TIME CMD
    12778 pts/1    00:00:00 bash
    12969 pts/1    00:00:00 ps

    $ bash

    $ ps
      PID TTY          TIME CMD
    12778 pts/1    00:00:00 bash
    12970 pts/1    00:00:00 bash
    12973 pts/1    00:00:00 ps
    $

Notice that there are two bash processes
Your login bash shell is still running
But you are now running a second bash
This new shell running as a sub-shell
When you exit this shell you will return to your login shell
This sub-shell is the second type of shell
It is not a login shell
It is an interactive non-login shell
You can only have one login shell for each ssh connection
It is interactive because you can type commands to it through the keyboard
You create non-login interactive shell without having to enter a password
You create a sub-shell of this type when you run script
script runs inside this sub-shell
The commands in the startup files named above
- .bash_profile
- .bash_login
- .profile
are NOT run before starting this kind of shell
Instead, the commands found in .bashrc are run
You are not limited to running Bash

You can also run sh in a sub-shell

    $ ps
      PID TTY          TIME CMD
    19874 pts/27   00:00:00 bash
    20500 pts/27   00:00:00 ps

    $ sh

    $ ps
      PID TTY          TIME CMD
    19874 pts/27   00:00:00 bash
    20510 pts/27   00:00:00 sh
    20526 pts/27   00:00:00 ps

You leave a interactive login shell by typing exit

    $ ps
      PID TTY          TIME CMD
    19874 pts/27   00:00:00 bash
    20737 pts/27   00:00:00 ps

    $ sh

    $ ps
      PID TTY          TIME CMD
    19874 pts/27   00:00:00 bash
    20743 pts/27   00:00:00 sh
    20751 pts/27   00:00:00 ps

    $ exit
    exit

    $ ps
      PID TTY          TIME CMD
    19874 pts/27   00:00:00 bash
    20771 pts/27   00:00:00 ps

Non-interactive Shells

A shell script is a text file with Unix commands
This is what you have been doing in the Class Exercises ...
and homework assignments
A shell script contains Unix commands which only a shell can understand
So you must run bash when running a script
This sub-shell will run Unix commands in the script
But it does not give you a command line with which to interact
Such a shell is called a non-interactive shell
There is no standard startup file for such a shell
You can create a startup file for non-interactive shells
But you have to set the shell variable BASH_ENV to the name of this file

Startup Files for This Class

A startup file contains Unix commands that are run just before you get a prompt
bash normally uses two startup files
- .bash_profile
- .bashrc
.bash_profile commands are run before you get a prompt in a login shell
.bashrc commands are run before you get a prompt in an interactive non-login shell
.bash_profile is where you define variables
We will not be talking much about .bashrc
Most Ubuntu installations only use .bashrc when running a GUI
You can be customized this new shell in .bashrc

Tips and Examples

Review

New Material

Homework 7

Midterm

Graded Quiz Next Week

Questions

Tips and Examples

Killing All Running Jobs

Seeing All Running Processes

Review

Running a Command in the Background

Jobs

Moving a Job from the Foreground into the Background

Aborting a Background Job

Attendance

New Material

Pathname Expansion

The ? Meta-character

The * Meta-character

The [ and ] Meta-characters

Builtins

Help for Builtins

Different Shell Versions

Using a Different Shell

POSIX

Shells For Running Scripts

Ways a Shell Can Be Created

Your Login Shell

Interactive Non-login Shells

Non-interactive Shells

Startup Files for This Class

Class Exercise

Class Quiz