shadow-cljs + Clojure with Calva: The basics

Peter Strömberg 2022-09-13

This How to use shadow-cljs with Calva in a fullstack project guide will not take the short path of presenting some steps and then ”BOOM there it is!” Instead we are going to look at the pieces involved and how they are composed to create Calva’s connection with the ClojureScript app being developed. The point with this is to give you the knowledge to get yourself past obstacles that a short 1-2-3 step guide might land you in.

How to start and connect Calva to a fullstack shadow-cljs and deps.edn project?

Just kidding. All the above said, here are the 1-2-3 steps. 😄

Configure shadow-cljs to use deps.edn for source-paths and dependencies. In it’s simplest form that is :deps true at the to level of shadow-cljs.edn. For some more granular control, see the shadow-cljs User Guide.
Include thheller/shadow-cljs {:mvn/version "..."} as dependencies in deps.edn. Throw in org.clojure/clojurescript {:mvn/version "..."} and binaryage/devtools {:mvn/version "..."} as well, for good measure.
Run the VS Code command: Calva: Start a Project REPL and Connect (aka Jack-in)
1. Select the shadow-cljs project type
2. Select to start whatever shadow-cljs builds you are working with (:app is a common one, you might have a :test build also want to start).
  - … the shadow-cljs server and the watcher will start …
3. Select which build’s REPL to connect the ClojureScript session to (:app for a typical project)

Now the Clojure REPL will be automatically used for *.clj files. And the ClojureScript REPL will be used for *.cljs files. For *.cljc files you will see in the status bar which REPL is used, and you can also use the status bar to toggle this between the Clojure and ClojureScript REPLs.

Note that you will need to load the app in the web browser (assuming you are building a browser app) before you can use the CLojureScript REPL session for evaluating anything. If this is in anyway not obvious to you, you really should read on.

In more depth/Rationale

I will assume you know almost nothing about Clojure editor tooling. Skim and skip anything you think is too basic for you. If you learn best by watching a video (or by combining reading and watching video), there is this ”version” of this guide:

https://www.youtube.com/watch?v=islMjv55cN8

REPL <> nREPL

A common confusion is about not realizing that the Clojure REPL and nREPL are two very different things. nREPL is not a kind of REPL. nREPL basically is two things:

It is a protocol for connecting tooling to the Clojure REPL
It is a server running in process with the Clojure REPL, implementing the server part of the protocol.

But I’m getting ahead of myself here. Let’s start with something more fundamental:

What is the Clojure REPL?

The REPL is a way to interact with the application, to modify it, to inspect it, and to test it.

Picture what is going on when a Clojure application is run. It is a fascinating story, which I am not going to detail here. Suffice it to say that there is a program, running on the JVM, that can read Clojure code, compile it, and run your application. This program stays there when your application has started. Additionally, your application has the potential of being partly or fully recompiled and redefined while it is running. What if this program could be asked to read some more Clojure code, compile it, and run it and redefine your application?

It can! This is the role of the REPL, an optional ”part” of the application you are developing. It provides the means by which a developer can reach the program mentioned above, and get the recompilation and reevaluation to happen interactively. We just need to start it.

How is the REPL started?

The REPL is started by that program running on the JVM. During development, we normally start this program with the clojure (small c), or lein, or shadow-cljs command line executables. Since the REPL is an optional part, we need to ask for it to be started, providing arguments to clojure, lein, or shadow-cljs.

How does Calva connect to the REPL?

For Calva to connect to the REPL, a special server, the nREPL server, needs to run in the same process. It receives messages from the nREPL client (Calva) and evaluates code towards the REPL, then the nREPL server packages the results as messages which it sends back to Calva.

NB: Calva has a feature for logging these messages, which you can enable using a VS Code command: Calva Diagnostics: Toggle nREPL Logging Enabled

In the Clojure REPL workflow, Clojure code ”lands” in three places:

The editor
nREPL server
Clojure REPL (reads and evaluates)

Then the evaluation results travel back to Calva in the reverse order.

This communication takes place in an nREPL session. Any number of nREPL sessions can be started. Calva creates and uses one (for Clojure development) or two (with ClojureScript) nREPL sessions. The first session is created when Calva connects to the nREPL server. This session is used until the REPL connection is closed. The second is created when Calva connects to the ClojureScript nREPL ”relay”. It is also used until the REPL connection is closed.

What’s needed by Calva to connect the REPL?

To support Clojure development Calva needs the app/project to include two dependencies: nrepl, and cider-nrepl.

nrepl is the nREPL server and the infrastructure for talking to the REPL and sending and receiving messages to the nREPL client. nrepl comes with some default commands/ops. A very important such op is eval.
cider-nrepl is an nrepl extension, providing a set of extra ops. (Currently Calva is a bit too stupid about this dependency and blindly assumes cider-nrepl is available. It can result in strange behaviour if cider-nrepl is missing.)

Calva can start the Clojure program, the REPL and the nREPL server, and inject these dependencies. This is what we call Jack-in. Calva can also just provide you with the command it would use to start these things: Calva: Copy Jack-in Command to the Clipboard. Use this if you prefer a workflow where you connect to a running REPL over one where Calva starts the REPL for you. It will ensure that your REPL is started with the dependencies and middleware configuration that Calva needs.

What’s the relation between the ClojureScript and the Clojure REPL?

The ClojureScript REPL is a quite more involved thing. Especially when nREPL is part of the picture. This REPL is also running on the JVM where reading and compilation of the code happens.

However, the result of the compilation is JavaScript code. In order to evaluate things, this code needs to be evaluated on the same host (browser, node, whatever), and in the same context, as the application beeing developed. A part of the program under development is responsible for this.

The ClojureScript REPL needs to be able to send JavaScript code to this evaluator and receive the results back. For shadow-cljs and Figwheel this is done by the respective tool running a server in the same process as the Clojure REPL, and the ClojureScript compiler, is running, exposing a WebSocket which the evaluator connects to.

Using nREPL, a ClojureScript REPL evaluation has the following ”stations”:

Code in the editor
nREPL server
”Something” that relays to the ClojureScript Compiler
ClojureScript compiler (compiles to JavaScript)
shadow-cljs or Figwheel server
evaluator running in the JavaScript environment

The results of the evaluation travel back to Calva in the reverse order.

Thanks to Bruce Hauman for writing this great article on the topic: Figwheel editor integration

What are Calva’s dependencies for ClojureScript development?

Calva needs that ”Something” glue between the nREPL server and the ClojureScript compiler. For vanilla ClojureScript, for Figwheel, and for most other scenarios, this ”something” is Piggieback. For shadow-cljs piggieback is not needed or used. It has another solution that will get included automatically if shadow-cljs sees that cider-nrepl is a dependency.

Tip: To see a full ClojureScript REPL connection happening quickly and easily: Open a new VS Code window and issue one of the the commands: Calva: Fire up a ClojureScript QuickStart <Node or Browser> REPL. Check the Jack-in command Calva is using and you’ll see the dependency injection and nrepl middleware configuration.

How does Calva connect to the ClojureScript REPL?

From Calva’s perspective the ClojureScript REPL is reached via a second nREPL session. Things happen in this order:

Calva connects to the nREPL server
Calva creates a new nREPL session. This session is connected to the Clojure REPL
Calva clones the first nREPL session
Calva promotes the second nREPL session clone to a ClojureScript nREPL session, by starting the ClojureScript REPL from the Clojure REPL. And gluing it to the nREPL session. Piggieback middleware is used for this, (unless you are using shadow-cljs, which solves this in some other way).

NB: Calva encodes these steps in something we call REPL Connect Sequences. There are some sequences built in, and you can also define your own. See below for a tad more on this.

Do all ClojureScript environments support nREPL?

No. There is something called self-hosted ClojureScript, where there is no JVM involved and all compilation happen on the JS host. Conceptually very similar to how Clojure works. It needs an nREPL server to work with Calva, and I don’t think there is one. (I also don’t think there is ton of work involved to create this, so please consider if the idea of self hosted ClojureScript appeals to you and you want Calva support for it.)

Is nbb and JoyRide self-hosted?

I guess you could say they are, but we don’t use that term about it. There is no compiler. Joyride, like nbb, is interpreted. They have nREPL servers built in. In Calva it is treaded much more like with the Clojure REPL/nREPL scenario. Calva even sort of ”pretends” they are Clojure REPLs.

So: What about Calva and shadow-cljs in fullstack development, then?

Assuming we are talking about a Clojure backend here, we have two options:

We can set up the backend as one project and the frontend as another project. Each will use their own JVM instance running totally separate REPLs and nREPL servers.
We can set it up as one project, where the Clojure program takes care of both the backend development and the ClojureScripts compilation, etc. Then you have only one nREPL server.

Option 2 lets you work with the full project in one VS Code Window. With option 1, you’ll need two windows. Because, within one VS Code window, Calva can only connect to one nREPL server. And also only supports two nREPL sessions per server, one for Clojure and one for ClojureScript, as mentioned previously.

Let’s focus on setup option 2. We have one project and one nREPL server:

Remember: shadow-cljs, Figwheel servers (and vanilla ClojureScript), are started by the Clojure program.

Assuming deps.edn and shadow-cljs we would let deps.edn specify the source paths and the dependencies. shadow-cljs will take care of the ClojureScript part of things. However, we now again have two options:

Calva can use clojure to start the app. This is the deps.edn + shadow-cljs project type.
Calva can use shadow-cljs to start the app. This is the shadow-cljs project type.

With option 2, the shadow-cljs.edn configuration needs to let deps.edn handle the source paths and the dependencies. This will also cause shadow-cljs to in turn use clojure.

Thus, in both cases clojure will be used to start the JVM program that starts the compiler, the Clojure REPL and all that. In both cases it is from the Clojure REPL that the shadow-cljs server, watcher, and nREPL middleware will be running. In both cases the deps.edn file will need to specify a dependency on thheller/shadow-cljs.

With option 1 the shadow-cljs nrepl middleware needs to be injected. With option 2, it doesn’t. In either case, Calva’s Jack-in command takes care of this.

Option 2, using shadow-cljs to start things, is my recommendation. It is the easiest one to get right, and it also keeps shadow-cljs output separated in the command line process, and not intermixed with the evaluation results and printouts from your application, like it is with option 1.

What if I want it to be separate projects?

You can choose to let shadow-cljs handle the frontend dependencies and tools-deps handle the backend dependencies. If you do this, you will need to start two nREPL servers to work with the project, one for shadow-cljs and one for the Clojure backend. Given the Calva’s limitations mentioned above, this means that you need to work with the project in two VS Code windows. VS Code supports this with so called Workspaces. See One Folder, Two Windows for a bit more on this.

Appendix 1: Manually create and connect a ClojureScript REPL

Recently I made an attempt to demystify the process of creating a ClojureScript REPL and connect with an nREPL client, by publishing a project together with a README that takes you through doing this without Calva, and purely from the command line. That README might be a bit hard to follow, since it tries to leave several options open. Here’s a version of the guide using the assumptions of the recommendations in this article.

Prerequisites:

You have Java working
You have clojure working
You have nodejs working
You have cloned a copy of the shadow-w-backend to your machine.
You have initialized the npm dependencies in the project (npm i)

Steps

In a terminal change directory to the root of the project:

Start the Clojure REPL and the nREPL server:

$ clojure -Sdeps '{:deps {nrepl/nrepl {:mvn/version,"1.0.0"},cider/cider-nrepl {:mvn/version,"0.28.5"}}}' -M -m nrepl.cmdline --middleware "[cider.nrepl/cider-middleware,shadow.cljs.devtools.server.nrepl/middleware]"
nREPL server started on port ... on host localhost - nrepl://localhost:...

In another terminal in the same directory:

Connect an nREPL client to the nREPL server you started:

$ clojure -Sdeps '{:deps {reply/reply {:mvn/version "0.5.1"}}}' -M -m reply.main --attach `< .nrepl-port`
...
user=>

Start the shadow-cljs server:

user=> (do (require 'shadow.cljs.devtools.server) (shadow.cljs.devtools.server/start!))
...
shadow-cljs - HTTP server available at http://localhost:8700
shadow-cljs - server version: 2.19.9 running at http://localhost:9630
shadow-cljs - nREPL server started on port ...
:shadow.cljs.devtools.server/started
user=>

Start the shadow-cljs watcher on the :app build:
```
user=> (do (require 'shadow.cljs.devtools.api) (shadow.cljs.devtools.api/watch :app))
[:app] Configuring build.
[:app] Compiling ...
[:app] Build completed. (170 files, 0 compiled, 0 warnings, 1.35s)
:watching
user=> 
      
```
You can now access your freshly compiled app on http://localhost:8700. And you can edit some hiccup in src/main/core.cljs and see the app update. This means that shadow-cljs is re-compiling the code, is sending the compiled JS to the browser, and is calling the :after-load function of the app, making it re-render. (See src/main/core.cljs for the function marked with metadata: ^:dev/after-load.)

But you still don’t have access to the ClojureScript REPL…

Promote the Clojure nREPL session to a ClojureScript session:

user=> (do (require 'shadow.cljs.devtools.api) (shadow.cljs.devtools.api/nrepl-select :app))
To quit, type: :cljs/quit
[:selected :app]
cljs.user=>

There it is. The nREPL session is now connected to the ClojureScript REPL that is in turn connected to your app. You can confirm by evaluating somehing like:

cljs.user=> (load-file "src/main/core.cljs")
[]
cljs.user=> (in-ns 'main.core)
nil
main.core=> (swap! app-state update :text str " UPDATED")
{:text "Hello world! UPDATED"}

Or just do (js/alert).

Appendix 2: Connect Sequences

When you select a Jack-in (or Connect) ”Project type” like deps.edn + shadow-cljs or shadow-cljs, under the hood, Calva will execute what we call a REPL Connect Sequences-. Such a sequence is sort of a script for what and when to execute and evaluate to get the REPLs connected. The sequences mentioned here are built-in to Calva, but they are essentially not very different from what you can create yourself.

Connect Sequences have a Clojure and an (optional) ClojureScript, part. For the purpose of this article, the ClojureScript part is the most relevant. Some projects, or frameworks, have their own functions for starting and connecting the ClojureScript REPL. Custom connect sequences let you provide the code that should be used. The approach in Appendix 1 is good for experimenting and figuring out the exact code that should be used.

A few words from Agical