The UNIX Shell and Computing Environment


References


Topics

A world of shell choices

There are a number of different shell programs that you can choose from -- all have their own special features and style.
Here is a side-by-side comparison of some of these.

In addition, there are many specialized shells such as python and perl whichare used for special languages and other things.

Getting Started...

Check to see if your Mac workstation is turned on. If it is not on, please turn it on.

Login with your PSU id and password.

The first thing we need to do is get out of this gui to a terminal that shows a command line. Using the magnifying-glass "Find" button in the menu bar at the top right, search for an application called "Terminal". Drag this to the Dock so it will be convenient.

Run "Terminal". This will open up a "terminal window", and the program running inside the window is called a "shell". A Shell is the generic term used in the biological sense - a simple and tough thing surrounding the important stuff. For example, your body is a shell for your mind. In this context, we are using a Unix shell. There are around 10 different unix shells that people use, and that you might want to explore once you become more familiar.

Shell tricks

The UNIX shell command line

Welcome to the command line. In the terminal window, there should be a line of text something like "$" This is called a command prompt. Different shells usually have different prompts. The Bourne again shell (bash) is the default shell and uses a "$" for its prompt usually. But what it is doesn't really matter. The important thing is that there is a cursor next to it. If you start typing, that's where things will appear.

The command-line is the oldest computer interface in use, and is incredibly powerful (Before 2001, macs did not have a command-line interface. Windows dropped their's in 1995, and brought it back in 2006), but also a little scary because of it's arcane and obscure syntax. Fortunately, only a few commands are important for us right now.

There are many features in the shell to make your life easier -- if you can think of it, somebody has probably made it possible. For example, you can use the UP ARROW to go back through your history of commands you've already run, so you don't have to re-type long commands. The TAB key can be used for autocompletion often. For more information about common commands and magical incantations, see this cheat-sheet or that cheat-sheet. And remember, google is your friend.

Input and output redirection

Basic File Permissions

Processes and job control

important commands for managing processes and jobs: kill, ps, jobs, nice, disown, fg, bg, top, xargs

Scripting

Conditionals and looping in scripts

i=0
for f in *; do
    echo "# file $i is $f"
    i=$(( $i + 1 ))
done

i=0
for f in *.py; do
    echo "# python file $i is $f"
    i=$(( $i + 1 ))
done

Exercises

Shell scripting examples

Here is a simple script that finds the current sales rank of three books on Amazon.

#!/bin/bash

nums="0356026256 B009PS06D4 B00DJFKT54"
ur="http://www.amazon.com/dp"

d=`date +%Y%m%d`
touch data
for h in $nums; do
    url="$ur/$h"
    echo $url
    x=`curl -s -S -L $url | grep -e '^$' -v  \
        | grep -A4 "Amazon Best Sellers Rank" \
        | iconv -f UTF-8 -t US-ASCII//TRANSLIT `
    y=`echo $x | cut -d ' ' -f 5`
    echo $d $h $y >> data
    sleep 3
done
exit

Here is a script for retrieving the bibtex entry for a DOI.

#!/bin/bash
# http://tex.stackexchange.com/questions/6848/automatically-dereference-doi-to-bib

if [ -z $1 ]; then
    echo Example usage:  doi_to_bib.sh  10.1098/rspb.2014.1901
    exit
fi

doi=$1

curl_opts="Accept: text/bibliography; style=bibtex"
doi=`echo $doi | perl -pe "s/^doi://; s/=/\//g;"`
value="$(perl -MURI::Escape -e 'print uri_escape($ARGV[0]);' "$doi")"
doiurl="http://dx.doi.org/$value"

B=` curl -c /tmp/cookiejar  -s -S -LH "$curl_opts" "$doiurl" `


echo $B | perl -pe "s/(\d\d\d\d,)/\$1\n/; s/(\w+={.*?}),? */\t\$1,\n/g; s/\@article./\@article\{bib:/; s/–/-/g;" 

An example shell-script I've used to organize numerical experiments.

#!/bin/bash

# if run with no command line arguments, start the experiment
#       This is supposed to be done on a remote machine.
# Optional arguments:
#       clean - remove emphemoral files
#       upload - send this director to the math home directory
#       download - periodically rsync from the math directory
#       download tmp - same as download, but does not go quiet in daemon mode

app=$0

time=50000000 # long test
#time=50000  # good test-time
processors=4

cleaner() {
    rm -rf config_output.txt
    rm -rf data*.txt
    rm -rf *.eps
    exit
}
if [ "$1" = "clean" ]; then
    cleaner
    exit
fi


uploader() {
    d=`readlink -f $1 | xargs dirname  | xargs basename`
    remote="math:$d/"
    rsync --delete -auvz ./ $remote
    exit
}

if [ "$1" = "upload" ]; then
    uploader $0 $1 $2
    exit
fi


downloader() {
    d=`readlink -f $1 | xargs dirname  | xargs basename`
    if [ -z $3 ]; then
        exec $app $2 $d 2> /dev/null 1> /dev/null
        exit
    fi
    remote="math:$d/"
    i=0
    while [ $i -le 40 ]; do
        rsync $remote ./ -auvz
        ./plot.gplt
        i=$(($i+1))
        sleep 600
    done
    exit
}
if [ "$1" = "download" ]; then
    downloader $0 $1 $2 $3 $4
    exit
fi


runner() {
    pycode="for i in range(1,100,3): print i "

    datafile="$1"
    cmd="$2"

    touch config_output.txt
    $cmd -v 2>> config_output.txt
    touch $datafile
    echo "# $cmd" >> $datafile
    echo "# python code: $pycode" >> $datafile

    echo $pycode | python - | xargs -P $processors -n 1 -I{} $cmd -a{} >> $datafile
    echo "" >> $datafile
}


########### main seqence #############
touch config_output.txt
hostname >> config_output.txt
cmdbase="./R_emerge -n4 -g4 -x30 -y30 -z10 -b2.0 -f0.001 -t$time"
ivals=` echo "for i in range(0,10): print '%1.6f'%(0.5**i) " | python - `
for i in $ivals; do
    cmd="$cmdbase -i$i "
    datafile="data_i$i.txt"
    runner "$datafile" "$cmd"
done
exit