A few words from Agical

Changing my mind: Converting a script from bash to Babashka

Peter Strömberg

Here, a description of one of my many rewrites of shell scripts.

Clojure reaches wherever I want to code something. For shell scripting it is a great choice because Babashka. By old habit I still often start out with bash. I paste a piece of my command line history in a file and go from there. Then when my script stops behaving like I intend it to, and resists my attempts to make it: I regret not doing it with Babashka.

What’s the big difference? With Babashka you can connect your editor of choice (VS Code for me of course, dog food is my favorite food!) to the script. With the REPL I can explore my way to understanding. And I get to use Clojure, which is a delightful programming language.

The bash

This time i had created a bash script looking like so:

#!/bin/bash

clojure -P -A:dev

original_classpath=$(clojure -Spath -A:dev)
new_classpath=""
IFS=":"

for path in $original_classpath; do
   if [[ $path == /root/.m2/repository/org/clojure/clojure* ]] ||
      [[ $path == /root/.m2/repository/org/clojure/core.specs.alpha* ]] ||
      [[ $path == /root/.m2/repository/org/clojure/spec.alpha* ]]; then
       continue
   fi
   if [[ $path == *.jar ]]; then
       jar_basename=$(basename "$path" .jar)
       target_dir="/app/dependencies/$jar_basename"
       mkdir -p "$target_dir"
       unzip -q -o "$path" -d "$target_dir"
       new_classpath="$new_classpath:$target_dir"
   else
       new_classpath="$new_classpath:$path"
   fi
done

echo "${new_classpath#:}"

It’s for the clojure-clr-starter project. The function of the script is described in How to create a really simple ClojureCLR dependency tool. TL;DR:

  1. Download any dependencies used in my deps.edn file under the :dev alias, missing in my local Maven repository

  2. Build the classpath for the :dev alias

  3. Traverse the classpath and for each jar dependency:

    • Unzip it to a local directory

    • Change the classpath from .m2/.../ .jar to the local directory

    • Skipping Clojure Core(-ish) things

  4. Print the new classpath to stdout

The bash-in-Babashka

The script is straightforward enough and worked fine. But now I wanted to get rid of the hard coded :dev aliases there… It’s not all that hard to do with bash, but enough for me to realize I shouldn’t have been doing this in bash to begin with!

Drake Disapproves of bash. Drake Approves of Babashka

So, I rewrote it for Babashka. A straight port:

#!/usr/bin/env bb

(require '[clojure.string :as string])
(require '[babashka.process :refer [sh]])
(require '[babashka.fs :as fs])

(sh "clojure" "-P" "-A:dev")

(def original-classpath (string/trim (:out (sh "clojure" "-Spath" "-A:dev"))))

(def skip-deps ["org/clojure/clojure"
               "org/clojure/core.specs.alpha"
               "org/clojure/spec.alpha"])
(def skipped-deps-re (re-pattern (str ".*/\\.m2/repository/("
                                     (string/join "|" skip-deps)
                                     ")/.*")))

(defn skip-path? [path]
 (re-matches skipped-deps-re path))

(defn process-classpath [classpath]
 (let [paths (string/split classpath #":")]
   (->> paths
        (remove skip-path?)
        (map (fn [path]
               (if (.endsWith path ".jar")
                 (let [deps-subdir (string/replace (fs/file-name path) #".jar$" "")
                       deps-dir (str "/app/dependencies/" deps-subdir)]
                   (sh "mkdir" "-p" deps-dir)
                   (sh "unzip" "-q" "-o" path "-d" deps-dir)
                   deps-dir)
                 path)))
        (string/join ":"))))

(when (= *file* (System/getProperty "babashka.file"))
 (-> original-classpath
     process-classpath
     println))

Which worked, and I had a great deal of benefit from that guard (when (= *file* (System/getProperty "babashka.file")) ...) at the bottom of the script. I add it to all my Babashka scripts. The test returns false when the file is loaded in the REPL, which gives me the REPLs help to inspect things in the script and to get a few things right that I don’t get right immediately. (See Clojure Workflow in VS Code, coding FizzBuzz for a demo.)

However, the script was a bit verbose, especially around the blacklisting of dependencies. Instead of trying to fix that, I wanted to see if I could somehow get rid of the requirement to do the skipping altogether. I asked about it in the #tools.deps channel over at the Clojurians Slack and got some pointers from Alex Miller about using tools.deps as a library instead of shelling out to the clojure executable using tools.deps by that proxy. He also told me that tools.deps is assuming that all projects need the Clojure Core(-ish) things. Which meant that there wasn’t really a way to get rid of that part of the script.

That last part was a bit boring. But I still wanted to avoid shelling out, now that I knew I didn’t need to. I scratched my head about how to do it.

The Babashka

Then, magically, Michiel Borkent, the creator of Babashka, popped up in the conversation and told me:

He then proceeded by identifying a few more things I could take advantage of in Babashka to avoid shelling out and make things more tidy and portable. I can use:

  1. babashka.classpath/split-classpath instead of string/split

  2. babashka.classpath/path-separator instead of ":"

  3. babashka.fs/create-dirs instead of shelling out to mkdir

  4. babashka.fs/unzip instead of shelling out to unzip

Together it all gave me this script:

#!/usr/bin/env bb

(require '[clojure.string :as string])
(require '[clojure.edn :as edn])
(require '[babashka.deps :as deps])
(require '[babashka.classpath :as classpath])
(require '[babashka.fs :as fs])

(defn process-jar-deps! [classpath]
 (->> classpath
      classpath/split-classpath
      (map (fn [path]
             (if (clojure.string/ends-with? path ".jar")
               (let [deps-subdir (string/replace (fs/file-name path) #".jar$" "")
                     deps-dir (str "/app/dependencies/" deps-subdir)]
                 (fs/create-dirs deps-dir)
                 (fs/unzip path deps-dir {:replace-existing true})
                 deps-dir)
               path)))
      (string/join fs/path-separator)))

(when (= *file* (System/getProperty "babashka.file"))
 (deps/add-deps (edn/read-string (slurp "deps.edn")) {:aliases [:dev]})
 (-> (classpath/get-classpath)
     process-jar-deps!
     println))

Which is small and nice. And it runs much, much faster than both the bash version and the bash-in-Clojure version. Because, there is now zero shelling outs there! babashka.fs is very rich. With the knowledge gained from this exercise I will go visit a lot of my Babashka scripts and make them less bash-y.

Those command line options

Sweet. I was ready for adding those requirements. The script should accept the aliases as command line arguments. E.g:

$ ./docker/cheap-deps.clj -a foo -a bar

I want it to keep defaulting to :dev as the alias.

babashka.cli makes both these requirements super easy to implement:

...

 (let [aliases (:alias (cli/parse-opts *command-line-args*
                                       {:spec {:alias {:alias :a
                                                       :default [:dev]
                                                       :coerce [:keyword]}}}))]
   (deps/add-deps (edn/read-string (slurp "deps.edn")) {:aliases aliases})
...

I’ll add some error handling later. Scout’s honor!

Peter Strömberg

Peter Strömberg is an autodidact programming nerd living in Stockholm, Sweden. He has been all over the (coder) place for a long time. After some 20 years in the profession he made an attempt to leave it for another passion of his: Product. That attempt ended when Peter was introduced to Clojure and he felt how it gave him back all the joy of programming, and then some.

You can contract Peter for remote Clojure/ClojureScript gigs up to 80% of an FTE. The other 20% Agical pays him to spend doing open source work. Most often this is about maintaining Calva, a popular Visual Studio Code Clojure IDE extension which Peter created 2018.