dissoc

Clojure’s jar is not reproducible. In other words, if you clone Clojure’s git repo and call mvn clean install multiple times, you will get subtly different (but semantically equivalent) release artifacts.

There are three jars associated with each Clojure release:

• a directly-linked jar (clojure-1.x.x.jar), which includes AOT compiled sources, compiled Java classes, and Clojure sources
• a slim jar (clojure-1.x.x-slim.jar), which includes compiled Java classes, and Clojure sources
• a sources jar (clojure-1.x.x-sources.jar), which includes all source code

There are two main unpredictabilities in Clojure’s release artifacts:

1. jar metadata (file order, timestamps)
2. AOT compilation

Stabilizing the jar metadata is the easiest to tackle, thanks in part to Maven having already solved this problem for us. We simply add a property to Clojure’s pom.xml to tell Maven we’d like a reproducible build:

<properties>
  <project.build.outputTimestamp>
    2023-01-01T00:00:00Z
  </project.build.outputTimestamp>
</properties>

That takes care of the slim and sources jar. In fact, this source of unpredictability can be automatically accounted for as seen in Clojure’s Reproducible Central report. That project tests whether Maven Central artifacts can be rebuilt deterministically. For Clojure 1.12.5, one of four artifacts can be rebuilt deterministically (the reproducible fourth not a jar, but the pom.xml) and two of three remaining artifacts can be “stabilized” such that only the directly-linked jar cannot be reproduced.

The remaining issue is due to nondeterminism when Clojure compiles itself. I believe I have narrowed the issue down to the order in which closed-over locals are emitted, reported as CLJ-2959.

Briefly, a closure like (let [a 1 b 2] #(+ a b)) compiles to a class that has two fields and a constructor for closing over the values of a and b. Based on the nondeterministic ordering of hash maps within Clojure’s compiler, the signature for its constructor is either new closure(Object a, Object b) or new closure(Object b, Object a), which will also change callsites.

Stabilizing closure constructors alone seems to be sufficient for Clojure to compile itself deterministically, even though there are other possible sources of nondeterminism in Clojure’s compiler.

I have a proof-of-concept for a reproducible Clojure jar in frenchy64/clojure#45, from which I identified the problem of closed-over locals ordering. You can see in the first commit of that PR that the initial (admittedly MVP) patch changed just two lines, one for each kind of unpredictability. You’ve already seen the first, here’s the second:

- IPersistentMap closes = PersistentHashMap.EMPTY;
+ IPersistentMap closes = PersistentArrayMap.EMPTY;