Bash function that provides the ability to run commands in parallel, with a predefined number of threads. People tend to use GNU parallel for this, however, implementing this as a shell function has the following advantages: It is portable (doesn't require parallel to be installed) - and you can run shell functions as commands, whereas external programs require the commands to be standalone binaries.
#!/usr/bin/env bash
#
# Run commands in parallel
#
# This provides the ability to run commands in parallel with a predefined number
# of threads.
#
# People tend to use GNU parallel for this, however, implementing this as a
# shell function has the following advantages:
#
# - It is portable (doesn't require parallel to be installed).
# - You can run shell functions as commands, whereas external programs require
# the commands to be standalone binaries.
#
# Notes:
#
# - Provide the commands as an input to this function
#
# Wait for background processes to exit
#
# $ wait [PID [...]]
#
# Notes:
#
# - If you use wait without specifying a PID, bash will wait for all background processes.
# - If you want to catch the return code of the background process, you must specify the PID.
# - If you specify multiple PIDs, the return code of wait will be the return code of the last PID specified.
#
# https://github.com/dansimau/bashlib/blob/master/bash.md#run-commands-in-parallel
parallel() {
local max_threads=$1
local -a pids=()
local ret=0
# Set up named pipe to communicate between procs
fifo="$(mktemp)" && rm -f "$fifo"
mkfifo -m 0700 "$fifo"
# Open pipe as fd 3
exec 3<> $fifo
rm -f $fifo # Clean up pipe from filesystem; it stays open, however
local running=0
while read cmd; do
# Block, when at max_threads
while ((running >= max_threads)); do
if read -u 3 cpid; then
wait $cpid || true
((--running))
fi
done
# Spawn child proc
(
$cmd
sh -c 'echo $PPID 1>&3' && :
) &
pids+=($!)
((++running))
done
# Return 1 if one or more pids returned a nonzero code
for pid in "${pids[@]}"; do
wait "$pid" || ret=1
done
return $ret
}