Skip to main content

Bash function that provides the ability to run commands in parallel, with a predefined number of threads. People tend to use GNU parallel for this, however, implementing this as a shell function has the following advantages: It is portable (doesn't require parallel to be installed) - and you can run shell functions as commands, whereas external programs require the commands to be standalone binaries.

#!/usr/bin/env bash

#
# Run commands in parallel
#
# This provides the ability to run commands in parallel with a predefined number
# of threads.
#
# People tend to use GNU parallel for this, however, implementing this as a
# shell function has the following advantages:
#
# - It is portable (doesn't require parallel to be installed).
# - You can run shell functions as commands, whereas external programs require
#   the commands to be standalone binaries.
#
# Notes:
#
# - Provide the commands as an input to this function
#
# Wait for background processes to exit
#
#   $ wait [PID [...]]
#
# Notes:
#
# - If you use wait without specifying a PID, bash will wait for all background processes.
# - If you want to catch the return code of the background process, you must specify the PID.
# - If you specify multiple PIDs, the return code of wait will be the return code of the last PID specified.
#
# https://github.com/dansimau/bashlib/blob/master/bash.md#run-commands-in-parallel
parallel() {
    local max_threads=$1
    local -a pids=()
    local ret=0

    # Set up named pipe to communicate between procs
    fifo="$(mktemp)" && rm -f "$fifo"
    mkfifo -m 0700 "$fifo"

    # Open pipe as fd 3
    exec 3<> $fifo
    rm -f $fifo # Clean up pipe from filesystem; it stays open, however

    local running=0
    while read cmd; do
        # Block, when at max_threads
        while ((running >= max_threads)); do
            if read -u 3 cpid; then
                wait $cpid || true
                ((--running))
            fi
        done

        # Spawn child proc
        (
            $cmd
            sh -c 'echo $PPID 1>&3' && :
        ) &
        pids+=($!)

        ((++running))
    done

    # Return 1 if one or more pids returned a nonzero code
    for pid in "${pids[@]}"; do
        wait "$pid" || ret=1
    done

    return $ret
}