IT 244: Introduction to Linux/Unix

IT 244: Introduction to Linux/Unix
Class 6

Microphone

Questions

Are there any questions before I begin?

Quiz 1

I have posted the answers to Quiz 1 here.

Homework 3

I have posted homework 3.

As usual, it will be due this coming Sunday at 11:59 PM.

Today's Class

Today I will wrap up the discussion of utiliities.

I will also discuss the Unix prompt and how you can use it to see where you are in the filesystem.

Finally I will talk about pipes, one of the most powerful features of the Unix command line.

General Advice

Attending Class

One of the simplest thing you can do to improve your grades ...
is to come to class regularly
This is why attendance is 5% of your grade
Studies have shown that you learn more by attending class ...
than by taking online courses
Teachers noticed a dramatic decline in student performance during the COVID lockdown ...
when schools switched from classrooms to online teaching
Why?
Humans are social animals
We must have contact with other people or we go nuts
There are large parts of our brains that help us deal with others
For example, there is a special part of our brains that recognizes faces
We pay more attention when we are with people ...
than when we only see them on a screen
Learning is a social experience
While being in a classroom is better than viewing on line ...
you learn even more when you interact with others in class
That is why it is so important for you to ask questions in class
If mine is the only voice you hear in class ...
it will not be as interesting as when you hear other students speak
Don't be afraid to ask questions in class
You don't even have to raise your hand
I am comfortable with interruptions
It makes the class more interesting for me too
If I don't know the answer I will tell you
I may look up the answer after class ...
and tell you what I have found in the next class
Let's keep things lively ...
so learning won't be a chore

Course Requirements

Shell Scripts for Homework Assignments

Starting with homework 3, you will need to write shell scripts for all homework assignments
This makes it easier for me to grade your homework
These script files must meet certain requirements or you will lose points
I've created a document which lists the requirements which you will find here

Scripts for Class Exercises

For now on, all Class Exercises will also ask you to write scripts
You do not have to follow all the requirements for homework scripts for Class Exercise scripts
For example there are no Step comments
But there are a few thing you should not do in these scripts
Do not do any of the following
- Run man or info
- Run more or less
- Run nano, vi, vim, emacs or any text editors
All these commands ask the user for input
I run scripts to check your Class Exercises
If you use any of the commands above the scripts will not run the way they should
You will get no credit for the assignment until you remove these commands

Tips and Examples

Copying and Pasting at the Command Line

You can have many ssh windows open at the same time
Each of these windows runs inside an ssh client
These programs allow you to copy from one window and paste into another
On my Mac I log into pe15 using the MacOS Terminal application
Once there, I can copy text by selecting it ...
and holding down the Apple Command key while pressing C
I can paste something I've copied onto the command line of another window with Command V
PuTTY also make it easy to copy from one ssh window to another
Simply hold down the left mouse button while selecting the text ...
then go to the other winsdow and click the left mouse button
You do not have to use Control C
Or use select Paste from the Edit menu

A Space in the Wrong Place

A single wrong character on the command line can cause a command to fail
I discovered a great example of this some semesters ago
I was helping a student with a ex5.sh script

The problem involved two lines of the script

...
cd   /home/ghoffman/course_files/it244_files
...
grep  -r  foo  .
...

The script was supposed to go to my it244_files directory ...
and search for lines that had "foo" in them in all files and subdirectories
The error was caused by putting a single space in the wrong place in the cd command
Instead of typing the cd command given above, the student typed
```
cd  / home/ghoffman/course_files/it244_files
```
cd only takes one argument
But here we have given it two
Because the is a space between the / and "home"
Because of this extra space, cd moved the student to the directory /
This directory is the top of the filesystem
The -r option to grep causes it to search files in all subdirectories
This command will print all lines containing "foo" in the entire filesystem
Needless to say this command keep going and going
The only way to stop it was to use Control C

Finding Errors in a Script

Shell scripts are small programs
If a script contains a command that makes no sense ...
running the script will give you an error

The following script contains an error

$ cat go_home_bad.sh
# this script contains an error
cd /home/ghofman
pwd

When I run it I get

$ bash go_home_bad.sh
go_home_bad.sh: line 2: cd: /home/ghofman: No such file or directory
/home/ghoffman/course_files/it244_file

Even though there was an error on the first line ...
the script continued to run
Here the error was a spelling mistake
The name of my home directory is ghoffman not ghofman
But what if the error message appeared after several lines of output?
You might not see the error as the output scrolled off the screen
The following script produces many lines of output ...

but it has the same error on the first line

$ cat go_home_bad_2.sh
# this script contains an error which is 
# hard to see in the output
cd /home/ghofman
ls -l

When I run this script it prints an error message
But that message is hard to see ...

because it scrolls by quickly off the top of the screen

$ bash go_home_bad_2.sh
go_home_bad_2.sh: line 3: cd: /home/ghofman: No such file or directory
-rw-rw-r-- 1 ghoffman faculty   51 Jun  8 11:54 numbers1.txt
-rw-rw-r-- 1 ghoffman faculty   51 Jun  8 10:55 numbers2.txt
-rw-rw-r-- 1 ghoffman faculty   50 Jun  8 10:55 numbers.txt
-rwxrwxr-x 1 ghoffman faculty   94 Jun  8 10:55 print_arg_numbers.sh
-rwxrwxr-x 1 ghoffman faculty  156 Jun  8 10:55 print_foo_bar.sh
-rwxrwxr-x 1 ghoffman faculty   71 Jun  8 10:55 print_foo.sh
-rwxrwxr-x 1 ghoffman faculty  132 Jun  8 10:55 print_positionals.sh
-rwxrwxr-x 1 ghoffman faculty  122 Jun  8 10:55 read_1.sh
-rwxrwxr-x 1 ghoffman faculty  141 Jun  8 10:55 read_2.sh
-rw-rw-r-- 1 ghoffman faculty 1214 Jun  9 10:35 red_sox.txt
-rwxrwxr-x 1 ghoffman faculty  260 Jun  8 10:55 repeat.sh
....

There is a trick that you can use to see only the error messages

$ bash go_home_bad_2.sh > /dev/null
go_home_bad_2.sh: line 3: cd: /home/ghofman: No such file or directory

When I follow the command with
```
> /dev/null
```
Unix throws away everything except the error messages
I used this trick in the "Error check" section of my Class Exercise test scripts
You'll see why this works when we discuss redirection in a future class

Review

`echo` - Print Text to the Terminal

echo prints text to the screen
```
$ echo Hello
Hello
```
You can use echo to print the value of a system variable ...
if you put $ before the variable name
```
$ echo $SHELL
/bin/bash
```

`hostname` - Print the Name of Your Host Machine

hostname prints the network name of the machine that you have logged on to
```
$ hostname
pe15
```
When used with the -i option hostname will print the IP address of the machine
```
hostname -i
158.121.104.215
```

Pagers - View a File One Screen at a Time

Pagers are programs that display the contents of a file one screenful at a time
The two pagers that Unix supplies are more and less
Hitting the Space bar advances to the next screen
Hitting the Enter or Return key takes you down one line
less, just to be confusing, has more features than more

Pathname Completion

When typing a long file name it is easy to make a mistake
To make life easier, Unix provides a feature called pathname completion
You type a few characters, then hit the Tab key ...
and Unix will try to supply the rest
If there is only one file or directory that matches ...
Unix will fill in the rest of the name
If there is more than one match ...
Unix will supply as much as it can and then beep
If you hit Tab again it will show you the possible matches
If there is no match, it will also beep

`grep` - Finding Strings inside Files

grep is used to find all lines in a file that contain a string
grep takes two arguments
- The string you are searching for
- The file or files in which to search
grep has the following format
```
grep STRING FILE [FILE ...]
```
grep -r will search recursively through a directory
That means it looks through the directory given as a second argument ..,
and all the subdirectories
grep, like Unix, is case sensitive
It thinks of "foo" and "FOO" as two different strings
To have grep ignore case, run it with the -i option
grep -v returns all lines that do not match the search string

`head` - View the Top of a File

head prints the first 10 lines of any file
For a different number of lines, follow head with a dash - ...
and a number
```
head -20 foo.txt
```

`tail` - View the Bottom of a File

tail prints the last 10 lines of any file
For a different number of lines, follow tail with a dash, - ...
and a number
```
tail -5 foo.txt
```

`sort` - Print a File in Sorted Order

sort prints all the lines in a file in sorted order
sort does not change the file
To sort in reverse order use the -r option
sort, by default, sorts in alphabetical order
This is a problem when the first characters on a line are numbers
That's because 11 will sort before 2
To sort in numerical order use sort -n
To sort in reverse numerical order use sort -nr

Attendance

New Material

The Shell Prompt

The Unix prompt is the shell saying it is ready for the next command
Before I changed it, my prompt looked like this
```
ghoffman@pe15:~/it244/work$
```
This prompt tells you three things
- Usename
- Hostname
- Current directory
The characters before the @ are my Unix username
The characters between @ and : are the hostname of the machine I am using
The characters from : to $ show my current directory
The last bit is the most important
You always have to know where you are in the filesystem ...
because some commands will not work if you are in the wrong directory
My current prompt looks like this
```
[498] (pe15) ghoffman - ~
```
This is one example of how you you can customize your Unix environment
It gives me the same information as the standard prompt ...
but in a a manner I find more useful
The ~ at the end means my home directory
I can change the prompt because what it looks like ...
is determined by the shell variables PS1
A variable is a space in RAM that has a name ...
and stores a single value
We'll talk a lot more about them in a few weeks
If you look carefully at your prompt
You can always see where you are in the Unix filesystem

Pipes - Stringing Programs Together

Pipes are one of the most powerful features of Unix
A pipe takes the output of one Unix command ...
and feeds it into the input of another
Using pipes you can string commands together ...
to get exactly the output you want
This way many commands can work together to do something ...
that none of them could do by itself
A pipe consists of two or more Unix commands ...
with each command separated from the one that came before by |
Let's say we want to find all Red Sox wins against the Rays
But sorted in reverse order by date

From the file red_sox.txt

$ cat red_sox.txt 
2011-07-02      Red Sox @  Astros       Win 7-5
2011-07-03      Red Sox @  Astros       Win 2-1
2011-07-04      Red Sox vs Blue Jays    Loss 7-9
2011-07-05      Red Sox vs Blue Jays    Win 3-2
2011-07-06      Red Sox vs Blue Jays    Win 6-4
2011-07-07      Red Sox vs Orioles      Win 10-4
2011-07-08      Red Sox vs Orioles      Win 10-3
2011-07-09      Red Sox vs Orioles      Win 4-0
2011-07-10      Red Sox vs Orioles      Win 8-6
2011-07-15      Red Sox @  Rays         Loss 6-9
2011-07-16      Red Sox @  Rays         Win 9-5
2011-07-17      Red Sox @  Rays         Win 1-0
2011-07-18      Red Sox @  Orioles      Win 15-10
2011-07-19      Red Sox @  Orioles      Loss 2-6
2011-07-20      Red Sox @  Orioles      Win 4-0
2011-07-22      Red Sox vs Mariners     Win 7-4
2011-07-23      Red Sox vs Mariners     Win 3-1
2011-07-24      Red Sox vs Mariners     Win 12-8
2011-07-25      Red Sox vs Royals       Loss 1-3
2011-07-26      Red Sox vs Royals       Win 13-9
2011-07-27      Red Sox vs Royals       Win 12-5
2011-07-28      Red Sox vs Royals       Loss 3-4
2011-07-29      Red Sox @  White Sox    Loss 1-3
2011-07-30      Red Sox @  White Sox    Win 10-2
2011-07-31      Red Sox @  White Sox    Win 5-3

To find all the games against the Rays ...
I use grep to find all lines with "Rays" ...

in the file red_sox.txt

$ grep Rays red_sox.txt
2011-07-15      Red Sox @  Rays         Loss 6-9
2011-07-16      Red Sox @  Rays         Win 9-5
2011-07-17      Red Sox @  Rays         Win 1-0

Now let's feed this into another command that selects the games the Sox won

$ grep Rays red_sox.txt | grep Win
2011-07-16      Red Sox @  Rays         Win 9-5
2011-07-17      Red Sox @  Rays         Win 1-0

Now we can use sort to get the results in the order we want

$ grep Rays red_sox.txt | grep Win | sort -r
2011-07-17      Red Sox @  Rays         Win 1-0
2011-07-16      Red Sox @  Rays         Win 9-5

Because Unix has pipes, utilities can be simple
They need do only one thing
Because they can be combined with other utilities using pipes ...
to get just the output we want
The Unix tool philosophy is to write simple programs ...
that do one thing weel
As we progress through this course ...
you will have many opportunities to use pipes

`uniq` - Eliminate Duplicate Lines

When you are combining data from different sources ...
it is easy to wind up with duplicate lines
You can use uniq to eliminate the duplicates
uniq prints the lines in a file, removing adjacent identical lines

Consider the following file

$ cat numbers2.txt 
4
5
6
7
8
9
10
11
11
11
12
13
14
15
16
17
18
19
20

If we run unique on this file we get

$ uniq numbers2.txt 
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

Note, the duplicate lines must be adjacent for uniq to work
They must appear one right after the other

So uniq will not work with this file

$ cat numbers3.txt
4
5
6
7
8
9
10
11
12
13
11
14
15
11
17
18
19
20

$ uniq numbers3.txt
4
5
6
7
8
9
10
11
12
13
11
14
15
11
17
18
19
20

This is where pipes prove their worth
We first run sort -n to sort the lines by number ...

and them pipe the output of sort into uniq

$ sort -n numbers3.txt | uniq
4
5
6
7
8
9
10
11
12
13
14
15
17
18
19
20

uniq -i ignores case when looking for duplicate lines

`diff` - Differences between Files

Sometimes you have two files that are almost identical ...
but not quite
You can use diff to see the differences
diff compares two files and displays the lines that are different
But if you run it without a special option ...
the output is almost useless

Consider the following 2 files

$ cat numbers1.txt 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

$ cat numbers2.txt 
4
5
6
7
8
9
10
11
11
11
12
13
14
15
16
17
18
19
20

The two files are almost identical
But if we run diff without a specific option ...

the output is useless

$ diff  numbers1.txt  numbers2.txt 
1,3d0
< 1
< 2
< 3
10a8,9
> 11
> 11

diff was created for use with the Unix patch utility
patch creates a new version of a file from the changes given by diff
This output was never meant to be read by people

The most readable output is obtained by using the -y option

$ diff  -y  numbers1.txt numbers2.txt 
1                                 <
2                                 <
3                                 <
4                               4
5                               5
6                               6
7                               7
8                               8
9                               9
10                              10
                                >  11
                                >  11
11                              11
12                              12
13                              13
14                              14
15                              15
16                              16
17                              17
18                              18
19                              19
20                              20

diff -i ignores case when looking for differences.

`file` - Show the File Type

Let's say you were working with someone ...
and they told you to look in a certain directory
There might be several different kinds of files there
And it would be nice to know what they are
The file utility can help you here
file takes an argument of one or more files ...

and shows the type of each

$ file *
bin:                 directory
cars.txt:            ASCII text
cmds:                directory
dead.letter:         ASCII news text
downloads:           directory
exercises_it244:     directory
homework_it244:      directory
it244:               symbolic link to `/courses/it244/s19/ghoffman/'
java:                directory
...

You can also run file on individual files

$ file break.sh
break.sh: Bourne-Again shell script, ASCII text executable

$ file make_upper.py 
make_upper.py: Python script, ASCII text executable

$ file index.html 
index.html: HTML document, ASCII text

`date` - Get the Date and Time

date displays the time and date
```
$ date
Tue Aug  7 20:02:48 EDT 2012
```
You can change the way the date is displayed ...

by following date with a + and a format string

$ date +"%Y-%m-%d %r"
2012-08-07 08:19:44 PM

The format string consists mostly of character pairs ...
the first character of which is a %
In the string above, %Y stands for the four digit year
To get more information on the various formatting options use
```
info date
```

Now move the cursor down to the line that reads

* Date format specifiers::         Used in ‘date '+...'’

Then hit the Enter key

Now move down to

* Date conversion specifiers::     %[aAbBcCdDeFgGhjmuUVwWxyY]

And hit Enter again
date is often used in scripts were you need to keep a record of what was done ...
in a text file called a log
Each entry in the log should be labeled with the time and date

`wc` - Counting characters

Most of the files we use in Unix are text files
Text files are a collection of lines
A line is a series of characters with the newline at the end

wc counts the lines, words and characters in a file

$ cat lines.txt 
Line 1
Line 2
Line 3
Line 4
Line 5

$ wc lines.txt 
 5 10 35 lines.txt

This output says there are 5 lines, 10 words and 35 characters in lines.txt

You can use options to tell wc what to count

c - bytes (characters)
w - words
l - lines

$ wc -c lines.txt 
35 lines.txt

$ wc -w lines.txt 
10 lines.txt

$ wc -l lines.txt 
5 lines.txt

`which` - Finding a Program File

Unix commands are programs which exist as binary files
These files contain the binary codes that the computer's CPU understands
These commands are located in different directories in the Unix file system
You can use the Unix utility which ...
to find the exact location of a binary program file
To find the location of the tar program file we would run
```
$ which tar
/bin/tar
```
which shows that the executable file for tar is located in the /bin directory
which uses the PATH system variable to find the location of the file
We'll discuss PATH in a future class

`whereis` - Finding Files Used by a Program

whereis is another program that can be used to locate program files
whereis takes an approach different from that of which
Every Unix or Linux system has certain standard places ...
where it stores commands and the files they use
whereis searches these locations
which only shows the location of a command
```
$ which tar
/usr/bin/tar
```

When we run whereis on tar we get more information than which returned

$ whereis tar
tar: /usr/bin/tar /usr/include/tar.h /usr/share/man/man1/tar.1.gz

The first entry is the executable file /usr/bin/tar
The second entry is a header file /usr/include/tar.h
The program needs the header file to get certain information
The third entry is the file that man uses to provide information about tar
A word of caution about using which and whereis
Some commands are actually built into the shell itself
These command are called builtins and we will talk about them in a future class
If you run which or whereis on these programs you will get nothing back
```
$ which cd

$
```

`find` - Search for Files Using Different Criteria

The most powerful Linux/Unix command for finding files is find
Unfortunately, it's power makes it hard to use
find can be used to search for a file based on many different things such as
- The name of the file
- The last time the file was used
- The last time the file was changed
- The access permissions on the file
An in-depth discussion of find is beyond the scope of this course
The simplest way to use find is searching for files by name
You do this using the following format
```
find DIRECTORY -name FILENAME
```
Where DIRECTORY it the directory where the search begins ...
and FILENAME is the name of the file

Here is an example

$ find  /home/ghoffman  -name red_sox.txt
/home/ghoffman/course_files/it244_files/red_sox.txt

find will not accept a partial file name
```
$ find  /home/ghoffman  -name red

$
```
You can get the effect of searching on a part of a filename by using the * character

But since * has special meaning on the command line you have to escape it

$ find /home/ghoffman -name memo.\*
/home/ghoffman/memo.bak
/home/ghoffman/memo.txt
/home/ghoffman/tmp/memo.bak
/home/ghoffman/tmp/memo.txt
/home/ghoffman/tmp/memo.backup

`who` - See Users Logged On

who prints a list of all users currently logged on to the machine

$ who
ghoffman  pts/0        2012-08-12 13:41 (dsl092-066-161.bos1.dsl.speakeasy.net)
rouilj    pts/1        2012-08-12 04:25 (pool-74-104-161-40.bstnma.fios.verizon.net)
eb        pts/2        2012-08-12 08:19 (pool-96-237-251-11.bstnma.fios.verizon.net)

who also provides information about each user's login session
It shows the time they logged in
It also shows the machine from which the user connected

who am i will show the user who is logged into a specific terminal

$ who am i
ghoffman  pts/0      2012-08-12 13:41 (dsl092-066-161.bos1.dsl.speakeasy.net)

This can be useful if you find an unattended terminal
You can run the same command without the spaces but it gives less information
```
$ whoami
ghoffman
```

`finger` - Get information on Users

finger, like mv, has two functions

When used without an argument finger shows every user currently logged in

$ finger
Login    Name                  Tty      Idle  Login Time   Office     Office Phone
admerrib Adil Merribi          pts/23     35  Sep  8 12:19 (75.69.126.23)
apera    Antonio Pera         *pts/2   23:47  Sep  7 04:43 (73.16.254.131)
bnsevak  Birva Sevak           pts/12     14  Sep  8 11:45 (65.96.118.54)
gabrgabr Gabriel Park          pts/13    45d  Jul 17 09:46 (tmux(29686).%0)
gabrgabr Gabriel Park          pts/14    52d  Jul 17 16:09 (tmux(29686).%5)
gabrgabr Gabriel Park          pts/17    52d  Jul 17 16:23 (tmux(29686).%6)
gabrgabr Gabriel Park          pts/20    45d  Jul 18 19:39 (tmux(29686).%8)
gghinita Gabriel Ghinita       pts/9      5d  Sep  2 16:00 (192.168.105.101)
ghoffman Glenn Hoffman         pts/21         Sep  8 12:51 (209.6.202.123)
grullnor Raffi T Sahagian      pts/11  20:42  Sep  5 15:27 (192.168.107.156)
grullnor Raffi T Sahagian      pts/26     29  Sep  8 12:28 (73.126.84.97)
ldavid   Leonard David         pts/1      3d  Sep  3 11:34 (158.121.23.66)
lilmnh   Michael Nguyen        pts/16         Sep  8 12:38 (71.174.229.240)
mkhr     Mikhail Khayretdinov  pts/8      19  Sep  8 10:54 (76.28.116.136)
offner   Carl D. Offner        pts/0      2d  Jul 30 07:38 (50.200.93.68)
offner   Carl D. Offner        pts/22     3d  Sep  4 15:20 (192.168.107.170)
rickm    Rick Martin           pts/5   20:43  Aug 17 09:42 (192.168.105.41)
vyiep    Vinson Yiep           pts/10     5d  Jul 24 18:30 (73.47.67.141:S.0)
xiaohui  Xiaohui Liang         pts/7      2d  Sep  6 10:55 (192.168.107.184)

When used with a username, finger provides information about the account

$ finger ghoffman
Login: ghoffman                         Name: Glenn Hoffman
Directory: /home/ghoffman               Shell: /bin/bash
On since Sun Sep  8 12:51 (EDT) on pts/21 from 209.6.202.123
Mail forwarded to glennhoffman@mac.com
Mail last read Fri Sep  6 12:04 2019 (EDT)
Plan:
Office:         McCormack 3-0201-22 (Pterosaur poster on door)
Office Hours:   Tuesday & Thursday, 11:OO AM - 12:00 PM and 5:30 to 6:30 PM
Classes:        IT 116-1   Introduction to Scripting         TuTh    12:30  - 1:45 PM     Wheatley 1-004
                IT 244-1   Introduction to Linux/Unix        TuTh     2:00  - 3:15 PM     McCormack M1-409
                IT 117-1   Intermediate Scripting            TuTh     4:00  - 5:15 PM     University Hall 2-2310
Home Address:   40 Central Street
                Somerville, MA 02143

General Advice

Course Requirements

Tips and Examples

Review

New Material

Questions

Quiz 1

Homework 3

Today's Class

General Advice

Attending Class

Course Requirements

Shell Scripts for Homework Assignments

Scripts for Class Exercises

Tips and Examples

Copying and Pasting at the Command Line

A Space in the Wrong Place

Finding Errors in a Script

Review

echo - Print Text to the Terminal

hostname - Print the Name of Your Host Machine

Pagers - View a File One Screen at a Time

Pathname Completion

grep - Finding Strings inside Files

head - View the Top of a File

tail - View the Bottom of a File

sort - Print a File in Sorted Order

Attendance

New Material

The Shell Prompt

Pipes - Stringing Programs Together

uniq - Eliminate Duplicate Lines

diff - Differences between Files

file - Show the File Type

date - Get the Date and Time

wc - Counting characters

which - Finding a Program File

whereis - Finding Files Used by a Program

find - Search for Files Using Different Criteria

who - See Users Logged On

finger - Get information on Users

Class Exercise

Class Quiz

`echo` - Print Text to the Terminal

`hostname` - Print the Name of Your Host Machine

`grep` - Finding Strings inside Files

`head` - View the Top of a File

`tail` - View the Bottom of a File

`sort` - Print a File in Sorted Order

`uniq` - Eliminate Duplicate Lines

`diff` - Differences between Files

`file` - Show the File Type

`date` - Get the Date and Time

`wc` - Counting characters

`which` - Finding a Program File

`whereis` - Finding Files Used by a Program

`find` - Search for Files Using Different Criteria

`who` - See Users Logged On

`finger` - Get information on Users