Introduction
Announcements

Schedule
Labs
Assignments
TA office hours

Tests, exam

Topic videos
Some course notes
Extra problems
Lecture recordings

Discussion board

Grades so far

Shell programming 6 of 6

Hover over the image to see the time range in the original video. Click to play that excerpt.

Or go back to the entire video.

This video is about a few systems programming facilities in sh. Understanding this video requires understanding the first four shell programming videos, plus some of the basics about how processes work in unix.
First, let's examine the "exec" keyword in sh.
In most cases when the shell invokes other programs, it does so in accordance with the standard fork/exec/wait idiom: The shell forks, the child process execs your command, and the parent process waits for the child process to exit before the parent proceeds.
The "exec" keyword means: don't fork(). So for commands which invoke a program, the shell program is replaced by that other program.

The 'exec' caused the echo command to replace the shell program, so the rest of that shell script was not executed — the "echo goodbye" was executed, and the echo program terminated, and that was it.

What happens if we do a redirection to the file 'output' without any command to execute?

The file is created, but that's it. The shell forks as usual; the child process opens the file for write... and that's it! Then we continue with the next command.

Now let's combine this with exec.

What happened here is that as I said in the beginning, the 'exec' keyword makes the shell refrain from the fork(). So rather than forking and having the child redirect the standard output, the main shell itself redirects the standard output. This means that all subsequent commands executed by this shell script will have that file as their standard output, until some other redirection occurs. This can be quite useful in a shell script; for example, a shell script which generates some file. You can tell people to run your shell script directed out to the appropriate file, and sometimes that works best; but you can alternatively do the redirection yourself using exec.

Here's another modification to the standard fork/exec/wait idiom.

First, for the basic example: In a command-line like "prog1; prog2" the prog2 command will not be started until after prog1 has completed.
However, if we replace the semicolon with an ampersand: "prog1 & prog2"
then this means to run both programs simultaneously. The shell will fork; the child process will exec prog1; but then the shell will not wait for this child process, but will proceed with forking again and execing prog2.

The syntax is a little bit out of sync with the concept behind the syntax. Interactively, we can write simply "prog1 &", and then we will get our shell prompt back immediately, while prog1 starts executing. More user-friendly modern versions of sh will asynchronously tell us when prog1 exits, too, but we're not going to get into that right now.

We tend to call this a "background process" — it runs in the background while you continue to execute further commands.

In a shell script, we can also start programs to run concurently in this way, either by replacing the separating semicolon with an ampersand, or simply by putting an ampersand at the end of a command-line.

We might then want to know the process ID of a so-called "background" process — we'll have a use for it in two commands I'll mention shortly. The special variable "$!" is the process ID number of the most-recently-started background process.

So we could write something like this...
and then pid1 would contain the process ID of the running prog1 and pid2 the process ID of prog2.

We have the expected difference between this as opposed to this.

Given the semicolon, in both cases prog1 and prog2 will run sequentially, not simultaneously; but in the case on the right prog3 will be executed immediately after prog1 is started, whereas in the case on the left, everything waits for prog1 to terminate, then we start prog2 and then immediately start prog3 without waiting for prog2.

Since an ampersand suppresses the normal 'wait', there is a way to cause the wait to happen later. The sh command for this is "wait". A wait command with no arguments waits for all outstanding background processes. Alternatively, you can say wait and a process ID number. For example, we can use that process ID number we saved earlier for prog2: "wait $pid2"

Something else you can do other than to wait for one of these processes to exit on their own is to send them a signal. Since the commonest use of this is to terminate the process, the command is named "kill".
If we write "kill 41" this sends signal 15 to process 41. Of course that process ID number is often from a variable. You might even be able to use $! here directly, although using $! any time more than a line or two after the ampersanded command can be error-prone since $! means the last background process ID number so it's easily overwritten by new events. So usually we assign $! to a normal variable immediately, if we are going to want to use it later.

If you want to send a signal number other than 15, you make it an option by prefacing it with a minus sign; For example, "kill -3 41" sends signal 3 to process 41, and "kill -3 41 48 59" sends signal 3 to each of these three processes.

This looks like a standard unix command-line option, but it's not quite. For example, "kill -12 41" might look like there are separate options -1 and -2, but that's not what it means — it just sends signal 12 to process ID 41. It's not a standard option format; it's a minus sign and then a number, of however many digits.

The last topic in this video is about making your shell scripts executable, so that you can execute them by typing their path name, without having to type "sh" as part of the command.
If we have a shell script: ...
so far we've been executing it by typing "sh s7". This makes sense, but it makes it seem like not a normal program. We want to be able to type something like "./s7", or maybe even simply "s7" depending on the value of your PATH variable.
But of course, you get an error from typing this, because the file is not executable:
Now, the naïve approach might be to say, hey, I know the chmod command!
...
Surely that can't work?
How can that possibly work?

What happens is, just to make things like this work, when the shell tries to exec a command, if it gets an error, as it will from trying to exec this file which is not a machine language program nor any other format of executable recognized by the operating system kernel, if the file is nevertheless marked as executable, the shell figures it must be a shell script so it goes ahead and executes it as one! It is the shell, after all; the shell is right here, it can go ahead and execute this shell script even without explicit operating system assistance.

Modern versions of sh are cleverer about this in that they won't attempt to execute certain files which are certainly not shell scripts but are, for example, machine language programs but for a different CPU architecture. But that's a detail. The basic idea is that a failing exec on an executable file means it's a shell script, so the shell goes ahead and executes it.

This doesn't give us everything we want, in two ways, which is why it's been improved upon. The above still works, but consider two things. First of all, the other program trying to exec our shell script might not be the shell. Suppose we write a C program which tries to exec another program and that other program is our shell script? Or for an example which comes up a lot these days, suppose we write our program to run under a web server based on certain web requests from the net? The web server isn't the shell; it will call exec on our program, but it won't have a fallback of executing it as a shell script when the exec fails.

Another problem is that these days there are at least two shells, sh and csh. Their programming languages are completely different. You might write a shell script for one of them, and then when it's executed you want it to be executed with the correct shell automatically. In fact, the mechanism I'm about to discuss works for executing any interpreted programming language, so it can also be used to mark a python program so that you can chmod +x it and make the python interpreter automatically be executed when someone tries to execute your program. Or any other interpreter.

The way it works is by putting this logic back into the operating system kernel. When you exec a file, the kernel is already examining the first few bytes of the file, which for most binary file formats contains what we call a "magic number". This tells all sorts of software what kind of file it is. For machine language files it says what platform it's for, so the operating system can gracefully refuse to exec a machine language program written for an incompatible CPU or operating system. There are standard magic numbers for various image file formats too, and so on, all sorts of binary files. And, there's a particular magic number, which corresponds to two bytes in the ASCII character range, which tells the kernel to read the rest of that line, up to a newline character, to tell it what interpreter to run!

The two characters are the number sign and an exclamation point.

If we preface our file with "#!/bin/sh", then the operating system knows to use /bin/sh as the interpreter.

This looks awfully similar. But in this case it's happening in the operating system kernel, it's not a feature of the shell. And it doesn't have to specify /bin/sh — I'll give examples of this in a moment.

These two characters are cleverly chosen... or at least the first one is. Because it does execute this file as a shell script. So why doesn't it give a syntax error for the #!/bin/sh line?
Because the number sign introduces a comment in sh.

The number sign also introduces a comment in Python, so you can do this with python just the same.
And we can use other programs similarly. Suppose we have a file containing some text. We can make it self-printing by using 'cat':
...

This is a bit weak, though, because we get to see the #! line. So instead, let's use sed to delete line 1 of the file.

Of course you would probably be using this for more text than just one line. This can be applied to any format of file and any program to process that file, so long as either that file-processing program treats '#' as introducing a comment to end of line, or there is some other way to make it ignore the first line as we just did by using sed.