notes blog about

runp: run shell commands in parallel

I’m using shell (bash specifically) on daily basis. From time to time a need arises to run multiple commands in parallel. For example my .bashrc runs commands like these to download or clone vim plugins I use:

curl -L -o $HOME/.git-completion.bash
rm -rf $HOME/.vim/pack/plugins/start/grep.vim && git clone $HOME/.vim/pack/plugins/start/grep.vim
rm -rf $HOME/.vim/pack/plugins/start/vim-go && git clone $HOME/.vim/pack/plugins/start/vim-go

The problem is that these commmands run sequentially and it takes a while until they are done. I was thinking of a way how to speed them up. So to scratch my itch I came up with runp.

Why and how to use it

Now I can run those commands (I stored them in install-my-stuff.txt) in parallel:


Let’s see how much time did I save:

$ time bash install-my-stuff.txt
real	0m15.690s
user	0m3.440s
sys	0m0.902s

$ time runp install-my-stuff.txt
real	0m3.678s
user	0m3.904s
sys	0m0.880s

Hmm, around 12 seconds. Not bad I think :-).

If you don’t want runp to add any output of its own (that is sent to stderr by the way) use the -q flag to quiet it:

$ runp -q install-my-stuff.txt 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 70071  100 70071    0     0   145k      0 --:--:-- --:--:-- --:--:--  145k
Cloning into '/home/reisinge/.vim/pack/plugins/start/vim-go'...
Cloning into '/home/reisinge/.vim/pack/plugins/start/grep.vim'...

How to install it

runp is easy to install it. It’s a single binary that you download and make it executable:

# choose your system and architecture
export SYS=linux  # or darwin
export ARCH=amd64 # or arm

# download it an make it executable
curl -L$SYS-$ARCH -o ~/bin/runp
chmod u+x ~/bin/runp

More examples

The commands to execute can be supplied also via stdin. It means that runp can be used within pipelines like this one:

$ for dir in $HOME /etc /tmp; do echo sudo "du -sh $dir"; done | runp -q | sort -h
13M	/tmp
17M	/etc
370G	/home/reisinge

Here we generate the commands to run in a bash for loop. Then we pipe the commands into runp. Finally the runp’s output (stdout) is sorted.

We can simplify by using the -p flag which adds a prefix string to the final command that will be run:

$ echo -e "$HOME\n/etc\n/tmp" | runp -q -p 'sudo du -sh' | sort -h
13M	/tmp
17M	/etc
370G	/home/reisinge

The final example shows how to find open ports from a list of hosts and ports:

cat << EOF > /tmp/host-port.txt
localhost 22
localhost 80
localhost 81 443 444 22 23 443

cat /tmp/host-port.txt | runp -q -p 'netcat -v -w2 -z' 2>&1 | egrep '(succeeded!|open)$'

You can find the source code and more examples here.