Open-sourced tool speeds up Linux scripts via parallelization

MIT has open-sourced pa.sh (also called pash), a tool that can dramatically speed up Linux scripts by using parallelization, saving time and without risk of introducing errors.

The process of parallelization first examines a script for code that can be run separately and independently, so not all scripts can benefit from the tool. But when pa.sh does find portions that can run independently, it runs them in parallel on separate CPUs. It also uses other techniques to get the code to run faster.

Below is a demonstration I ran on my home Fedora box, first running a script on its own and then again using pa.sh. Note that this script was provided with the pa.sh tool and lends itself to parallelization. It’s not nearly as demanding as scripts that might process gigabytes of data in a scientific or artificial-intelligence lab, so the results are not dramatic.

Running the script on the command line

I used the time command to gauge the performance of the hello-world.sh script.

$ time ./evaluation/intro/hello-world.sh
2176

real    0m55.077s
user    0m54.815s
sys     0m0.062s

NOTE: The “2176” on the second line is the script’s output.

Running the script using pa.sh

In the next command, I ran the same script through pa.sh.

$ time ./pa.sh ./evaluation/intro/hello-world.sh
2176

real    0m19.216s
user    0m37.509s
sys     0m0.255s

Notice that when run using pa.sh, the script used little more than a third of the time (real time) that it used when run directly. If I run a script that simply loops from 1 to 10,000 and display the count every 100th step, it takes significantly longer to run using pa.sh. That’s because with pa.sh, the script doesn’t benefit from parallelization but still requires an analysis:

$ time ./count_to_10000         $ time pa.sh ./count_to_10000
100                             100
200                             200
300                             300
400                             400
500                             500
600                             600
700                             700
800                             800
900                             900
1000                            1000
real    0m0.010s                real    0m59.121s
user    0m0.007s                user    0m41.386s
sys     0m0.003s                sys     0m19.263s

The script runs a single loop and looks like this and provides no opportunity for parallelization:

for num in {1..1000}
do
  if [[ "$num" == *"00" ]]; then
    echo $num
  fi
done

For complex scripts that can benefit from parallelization, however, pash can make a tremendous difference in how long they take to run. All you have to do is invoke your scripts using pa.sh. And, as already noted, pa.sh does this without introducing errors, so you can be confident that you will get the results expected, just a whole lot faster. If you are using scripts that need to process a large amount of data, this can save a lot of time.

Installing and using pa.sh

You will need to have tools like sudo, wget, and curl, but these tools are likely already available on your Linux system.

Once pa.sh is installed, you will need to export $PASH_TOP that will point to the top of the directory where it is installed. For example:

$ export PASH_TOP=/opt/pash
$ echo $PASH_TOP
/opt/pash

Wrap-Up

From everything I’ve seen and read, pa.sh can provide a dramatic performance improvement to complex and data-hungry scripts. If you or your organization might benefit from this kind of tool, it is well worth looking into.

The tool, as well as the example code, are open source, and pa.sh is available at github. There is no man page, but help is available when you use the pa.sh –help command. A technical paper explaining pa.sh has been posted by Nikos Vasilakis, a research scientist at MIT’s Computer Science & Artificial Intelligence Laboratory (CSAIL) who chairs the international committee of researchers who have worked on the tool for nearly two years. MIT announced pa.sh earlier this month.