sciware

Sciware

Command line and Shell interaction

https://sciware.flatironinstitute.org/23_CommandLine

https://github.com/flatironinstitute/learn-sciware-dev/tree/main/23_CommandLine

Rules of Engagement

Goal:

Activities where participants all actively work to foster an environment which encourages participation across experience levels, coding language fluency, technology choices*, and scientific disciplines.

*though sometimes we try to expand your options

Rules of Engagement

(These will always be a work in progress and will be updated, clarified, or expanded as needed.)

Zoom Specific

Future Sessions

Today’s Agenda

Introduction

Jeff Soules (CCM)

Definitions

You may hear people use these terms to mean the same thing:

But there are some subtle differences. Let’s do a quick review.

Operating System

We mention this mainly to explain the…

Shell

The shell interface surrounds the kernel just as a nutshell surrounds a nut 🥜.

Terminal

Terminal

Command Line

All Together

A shell is a program that uses a terminal to offer a command line interface, through which a user can direct the operating system to run other programs.

Shell Command Example

Here’s a typical shell command, with three parts:

Shell Command Example

Shell Command Example

What happens when you run a command?

Why not just use a GUI desktop?

Shells are:

Common Shells

Changing your shell

Dotfiles and shell configuration

Shell config files list

In practice, you probably just want to use .bashrc (if you run bash) or .zshrc (if you run zsh)

shelllogin (ssh)interactiveneither
bash.bash_profile | .bash_login | .profile.bashrc-
.bash_logout--
zsh.zshenv
.zprofile--
.zshrc-
.zlogin, .zlogout--

Interacting

Robert Blackwell (SCC)

Nomenclature/shorthand/tips

Directories

Tab key completion

Editing

Shells have a builtin clipboard ‘kill ring’ where ‘cuts’ add new entries to the ring.

History

Processes - Stopping

Controlling background processes (1)

%<id>: Identifier for job, where <id> is some number

Controlling background processes (2)

Evaluation

How does the shell decide what to do when you enter a command?

Environment Variables

Nils Wentzell (CCQ)

Environment Variables

Env. Variable Description
PATH List of directories searched or executables
USER The current username
HOME Home directory of the current user
PWD Path to the current working directory
VISUAL Text editor

Enviroment Variables - Live Demo

FOO="Hello there" # Define a shell variable
echo $FOO         # Use $ to obtain the value

# Shell variables are not seen by child process
bash -c 'echo $FOO'

# Convert into an environment variable using export
export FOO

# Environment variables are seen by child processes
bash -c 'echo $FOO'

# Variable expansion 
echo "$FOO fellow coders"

# Delete existing variables
unset FOO

# Definition and export in one line
export BAR="Hello again"

# Inspect your environment
env
env | grep BAR

Enviroment Variables - Live Demo

# PATH is a colon separated list of directories searched for programs
echo $PATH

# Create directory ~/bin for personal programs
mkdir ~/bin

# Create empty file ~/bin/prog
touch ~/bin/prog

# Make it executable
chmod u+x ~/bin/prog

# See if prog is found
which prog

# Prepending ~/bin to the PATH
export PATH=~/bin:$PATH

# Check that now we can find prog
which prog

Permanent Environment Variables

# -- ~/.bashrc

...

# Make sure programs in ~/bin are found
export PATH=~/bin:$PATH

# Set my Editor
export VISUAL=nano

Executing shell scripts: source


# -- myenv.sh
export PATH=~/bin:$PATH
export VISUAL=nano
...


✅ Execute script in current shell ➜ Script variables will persist

source myenv.sh


⛔ Run script as child process ➜ Script variables will not persist

bash myenv.sh

Environment Modules - Lmod


FI Cluster: Use the module command to find and load software

$ module load git/2.35.1

$ which git
/mnt/sw/nix/store/wix1v2lrfbwrh4mar9ry07zzvsix82i5-git-2.35.1/bin/git

$ module show git/2.35.1
...
prepend_path("PATH","/mnt/sw/nix/store/wix1v2lrfbwrh4mar9ry07zzvsix82i5-git-2.35.1/bin")

Instructions in our FI Wiki!

Managing Python Environments


* `virtualenv` ([user guide](https://virtualenv.pypa.io/en/latest/user_guide.html)) ```sh python -m venv --system-site-packages /path/to/myenv source /path/to/myenv/bin/activate ```
* `conda` ([user guide](https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html)) ```sh conda create --name myenv conda activate myenv ```

Environment variables affecting OpenMP

Control the number of threads for parallel programs.

Env. Variable Description
OMP_NUM_THREADS Number of OpenMP threads to use
MKL_NUM_THREADS Number of OpenMP threads to use for Intel MKL
OPENBLAS_NUM_THREADS Number of OpenMP threads to use for OpenBlas

Environment variables for software developement


Env. Variable Description
PYTHONPATH Searched for Python modules
CC, CXX C/C++ compiler
CFLAGS,CXXFLAGS Compiler flags for C/C++
LDFLAGS Flags for the linker
LIBRARY_PATH Searched for libraries at linktime
LD_LIBRARY_PATH Searched for dyn. libraries at runtime
CPATH Searched for C/C++ header files

Shell Customization

Dylan Simon (SCC)

Prompt ideas

Basic prompts

bash

https://www.gnu.org/software/bash/manual/bash.html#Controlling-the-Prompt
PS1='\u@\h:\w\$'
dylan@rusty:~$

zsh

https://zsh.sourceforge.io/Doc/Release/Prompt-Expansion.html
PS1='%n@%m:%~$'
dylan@rusty:~$

Color codes

PS1='       %n       @       %m       :       %~      $  '
PS1='%F{213}%n%F{177}@%F{141}%m%F{147}:%F{111}%~%F{75}$%f'

Including command output

The simplest way to include extra information in your prompt

git branch --show-current
main
PS1='\u@\h:\w [$(git branch --show-current 2> /dev/null || echo NONE)]$'

For zsh you may need to add
setopt prompt_subst

Unicode!

PS1='%F{green}%c $(git branch --show-current 2> /dev/null) %F{blue} // %F{red} ♥ %f'

Conditionals

Display number of background jobs (if any)
PS1='%n@%m:%~%1(j. [%j].)$'
user@host:~ [1]$
Use different prompts in different places
case $HOST in 
	rusty*) color=blue ;;
	cc?lin*) color=green ;;
	laptop) color=red ;;
esac
PS1="%F{$color}..."

zsh extras

Right-aligned prompt

RPS1='%F{blue}%T%f'

Options

https://zsh.sourceforge.io/Doc/Release/Options.html
setopt autocd   # type directory names without cd
setopt autolist # show options on single tab
setopt nobeep   # stop making noise on every tab

File colors (Linux/GNU coreutils only)

Set the colors you see in ls output

dircolors -p > .dircolors
edit .dircolors
dircolors .dircolors >> ~/.bashrc

dotfiles shared on github

Shell commands and scripting

Géraud Krawezik (SCC)

Most commands presented here take from 0 to N arguments. The command is usually just repeated on the N arguments

Getting help

$ man ls
LS(1)                              User Commands                         LS(1)

NAME
       ls - list directory contents

SYNOPSIS
       ls [OPTION]... [FILE]...

DESCRIPTION
       List information about the FILEs (the current directory by default).  Sort entries alphabetically if none of -cftuvSUX nor --sort is specified.

       Mandatory arguments to long options are mandatory for short options too.

       -a, --all

Working with files

The local directory is assumed when no argument is provided

Text handling

Outputs and errors

Reusing commands outputs

$ KERNEL_VERSION=`uname -r`
$ echo $KERNEL_VERSION
4.18.0-372.13.1.el8_6.x86_64

Control flow: loops

for nprocs in 1 2 4 8 16 32
do
    mpirun -np $nprocs ./program
done

shopt -s globstar     # Enable globs on bash
for f in **/*.md
do  # Iterate over the .md files in the directory and its subdirectories
    echo $f
done

Control flow: tests

if [[ $i -eq 0 ]]
then
    echo "Null value"
elif [[ $i -lt 5 ]]
then
    echo "Small value"
else
    echo "Large value"
fi

Scripts

Create complete scripts to perform common operations

#!/bin/bash
# The line above is the shebang, to tell Linux what interpreter to use
# Comments start with hash/pound

today=`date +%Y%m%d`
backup_folder=/tmp/save$today
list_files=`ls $HOME/ceph/data/`

mkdir $backup_folder

for file in $list_files
do
    mv $file $backup_folder
done

tar czvf $backup_folder.tar.gz $backup_folder

Please give us feedback

https://bit.ly/sciware-shells-2022

Other resources

Environment variables

EnvironmentVariables.md

Shell Comparison

History

Evolution

most modern shells copied, adopted similar, popular features

Preferences

if [[ $- == *i* && -x /bin/zsh ]] ; then
	SHELL=/bin/zsh exec /bin/zsh -l
fi

Shell variables

Modules

Python (conda and venv/virtualenv)

Conda

    conda create -n myenvname # places inside conda
    conda activate myenvname
    conda install numpy
    conda deactivate

venv (virtualenv)

    python3 -m venv myenvname # places in cwd
    source myenvname/bin/activate
    pip install numpy
    deactivate

Manual sourcing

$ cat setenv.sh
module purge
module load gcc python3 intel/compiler/2020 --force
module load pvfmm/1e4bef5 home/stkfmm/59951f4 openmpi2/2.1.6-intel-hfi intel/mkl/2020
source $HOME/projects/codes/fiber-private/env/bin/activate
$ source setenv.sh

Shell features

Builtins

More About Builtins
Today’s Builtins

Quoting and Word Splitting

I/O Redirection

stdin, stdout, stderr

Command and Process Substitution

Globs

Globs match things (but are less awesome than regular expressions)

Not Builtins But Still Great

Configuring your prompt 🎨

⚠️ Warning ⚠️

Prompt design

Off the shelf options

Off the shelf options

Prompt variables (bash)

Accessing git branch (bash)

  function parse_git_branch {
    git symbolic-ref --short HEAD 2> /dev/null
  }

Example prompt (bash)

  # Define the prompt character
  char="♥"

  # Define some local colors
  red="\[\e[0;31m\]"
  blue="\[\e[0;34m\]"
  green="\[\e[0;32m\]"
  gray_text_blue_background="\[\e[37;44;1m\]"

  # Define a variable to reset the text color
  reset="\[\e[0m\]"

  # Export PS1:  default interactive prompt
  PS1="\[\e]2;\u@\h\a[$gray_text_blue_background\t$reset]$red\$(parse_git_branch) $green\W\n$blue//$red $char $reset"

Prompt variables (zsh)

Accessing git branch (zsh)

  autoload -Uz vcs_info
  precmd() {vcs_info}
  zstyle ':vcs_info:git:*' formats '%F{yellow}%B% (%b)'

Example prompt (zsh)

  setopt PROMPT_SUBST
  PROMPT='%F{green}%B% %c ${vcs_info_msg_0_} %F{blue}%B% // %F{red}%B% ♥ %F{black}%B%'