Bash argument

For example submit.sh

POSITIONAL=()
while [[ $# -gt 0 ]]; do
  key="$1"

  case $key in
    -c|--config)
      CONFIG="$2"
      shift # past argument
      shift # past value
      ;;
    -n|--notes)
      NOTES="$2"
      shift # past argument
      shift # past value
      ;;
    -m|--mode)
      MODE="$2"
      shift # past argument
      shift # past value
      ;;
    *)    # unknown option
      POSITIONAL+=("$1") # save it in an array for later
      shift # past argument
      ;;
  esac
done

set -- "${POSITIONAL[@]}" # restore positional parameters

# set default value
if [ -z ${MODE+x} ]; then echo "MODE is unset, set to default as test"; MODE=test; fi
if [ -z ${NOTES+x} ]; then echo "NOTES is unset, set to default as none"; NOTES=ani; fi
echo

if [ -z "${CONFIG-}" ]; then
  # unset
  echo "Must provide a config file"
  exit 1
else
  # set
  echo "CONFIG  = ${CONFIG}"
  echo "MODE    = ${MODE}"
  echo "NOTES   = ${NOTES}"
  echo
fi

Usage:

bash submit.sh -c config.yaml -n "I am a note" -m run

output

CONFIG  = config.yaml
MODE    = run
NOTES   = I am a note

Redirect Output to File

redirect stdout

command > file

redirect stderr

command 2> file

redirect stdout and stderr to different files

command > file.out 2> file.err

redirect stdout and stderr to the same file

command > file 2>&1

find

count lines recursively

find . -name '*.py' | xargs wc -l

Find all the xyz files in dir1, copy the corresponding png files in dir2 to dir1.

richard@mordor~/test> tree
.
├── dir1
│   ├── 1.xyz
│   ├── 2.xyz
│   └── 3.xyz
└── dir2
    ├── 1.png
    ├── 2.png
    └── 3.png
2 directories, 6 files

richard@mordor~/test> find ./dir1 -type f -name '*.xyz' -exec bash -c 'file="$1"; filename="$(basename ${file} .xyz)"; command="cp dir2/${filename}.png dir1/"; echo "${command}"; bash -c "${command}"' bash {} \;
cp dir2/3.png dir1/
cp dir2/2.png dir1/
cp dir2/1.png dir1/

richard@mordor~/test> tree
.
├── dir1
│   ├── 1.png
│   ├── 1.xyz
│   ├── 2.png
│   ├── 2.xyz
│   ├── 3.png
│   └── 3.xyz
└── dir2
    ├── 1.png
    ├── 2.png
    └── 3.png
2 directories, 9 files

ffmpeg

mp4 -> mp3

mkdir outputs
for f in *.mp4; do ffmpeg -i "$f" -c:a libmp3lame "outputs/${f%.mp4}.mp3"; done

cut 视频

ffmpeg -ss 00:02:46 -i 勇气.mov -to 00:06:47 -c copy 勇气o.mov
ffmpeg -ss 00:00:00 -i 21.mp4 -to 00:00:10 -c copy o21.mp4

cut mp3

ffmpeg -ss 00:00:00 -to 00:10:00 -i inputfile.mp3 -c copy outputfile.mp3

调高音量

ffmpeg -i input.mp3 -af volume=23dB output.mp3

使输出文件能够快速加载并播放（通过将index等数据放在mp4文件起始位置）

ffmpeg -i input.mp4 -movflags +faststart output.mp4

Use python in bash script

Simple case:

for i in {0..5}
do
    file=$(python -c "a='wavefunction{:02d}'.format($i); print(a)")
    echo $file
    sleep 0.5
done
#wavefunction00
#...
#wavefunction05

python -c "print('some')"
some  #output

a=$(python -c "a='I am {}'.format('richard'); print(a)")
echo $a
I am richard   #output

a=$(python -c "print('some')" 2>&1)  # 2>&1的意思是python的stderr输出重定向到stdout
echo $a
some   #output

for i in {0..29}
do
    echo $i
    python -c "import os; os.remove('input'); a = 'blabla'; print(a); file=open('input', 'w+'); file.write(a); file.close()"
    cpptraj -p ../template.pdb -i input
done

Slurm

Enforce memory limit for jobs

related settings: /etc/slurm-llnl/slurm.conf

TaskPlugin = task/affinity,task/cgroup
MaxMemPerNode=100000
DefMemPerNode=100000
DefMemPerCPU=5000
SelectTypeParameters=CR_CPU_Memory

/etc/slurm-llnl/cgroup.conf

ConstrainRAMSpace=yes

The default mem-per-cpu is 5gb, you can specify more using #SBATCH --mem-per-cpu=10gb for example in your script. If the job exceed the requested mem, it will get killed by slurm to avoid OOM system error.

Ref: hpc - Cannot enforce memory limits in SLURM - Stack Overflow

Update Slurm after Replace GPU

If chang gpu on mordor or isengard, both of them needs to be updated

sudo vim /etc/slurm-llnl/slurm.conf
sudo vim /etc/slurm-llnl/gres.conf

Then restart

sudo /etc/init.d/slurmd restart
sudo /etc/init.d/slurmctld restart