A program evolves to be more like itself.
At Papers We Love this week, I was struck by Zach’s perspective on software growth. One of the joys I find in project maintainership is discovering the nature of the program itself. Sometimes the results are surprisingly beautiful. Zach puts into words the gut feelings maintainers often consult when guiding the evolution of their programs.
Perhaps Zach’s talk was so refreshing in part because most of the advice I’ve read on software evolution is more about how to go about change once you’ve decided to do it. I’m not as insightful as Zach, so this post is no different!
We know that accretion, relaxation, and fixation and are the main ways we should redefine things. I think recipes to responsibly grow data shapes and function behavior are well disseminated by the Clojure community, but less so about growing macros.
There is an extra dimension to be aware of when growing macros: the classpath in which a macro expands can differ from the one its expansion runs in. This usually happens in conjunction with AOT-compiled sources.
A recent Clojure enhancement
demonstrates this subtle point, which attempts to improve
case’s error messages.
It seems like a harmless choice to factor out part of the body of a macro from
(defmacro case-v1 [...] `(throw (IllegalArgumentException. (str "No matching clause: " ~ge))))
(defn case-fallthrough-err-impl [val] (IllegalArgumentException. (str "No matching clause: " (pr-str val)))) (defmacro case-v2 [...] `(throw (case-fallthrough-err-impl ~ge)))
By introducing the additional var, this patch caused a compatibility issue. Compiled Clojure code with the change emits a call to a var that does not exist prior to Clojure 1.10.2 so that would fail if used with an older runtime, something like:
Execution error (IllegalStateException) at cf/foo (cf.clj:5). Attempting to call unbound fn: #'clojure.core/case-fallthrough-err-impl
Alex has also hinted
definline (similar to
defmacro) has fallen out of favor with the Clojure core team.
Perhaps preserving binary compatibility is a factor in that trend:
the growth of functions is easier to reason about.
We can use this insight to design growable macros by
encapsulating their runtime component in a function,
(defn with-bindings* [binding-map f & args] (push-thread-bindings binding-map) (try (apply f args) (finally (pop-thread-bindings)))) (defmacro with-bindings [binding-map & body] `(with-bindings* ~binding-map (fn  ~@body)))
All accretion, relaxation, and fixation relating to the
can be localized to the
with-bindings* function. The macro only does
what the function cannot: change evaluation order.
Isolating growth to
with-bindings* increases the likelihood that
we can compile
with-bindings calls with a future Clojure
compiler and run the compiled expansion in a past Clojure runtime.
Another macro whose growth is difficult to reason about is
which is my motivation
for thinking about this topic. It’s defined like so as of Clojure 1.11:
(defmacro testing-v1 [string & body] `(binding [*testing-contexts* (conj *testing-contexts* ~string)] ~@body))
Similar to the
case example, I would like to enhance its error messages
but a naive implementation might unnecessarily break some programs.
(defn record-uncaught-exception-contexts [e] ...) (defmacro testing-v2 [string & body] `(binding [*testing-contexts* (conj *testing-contexts* ~string)] (try (do ~@body) (catch Throwable e# (record-uncaught-exception-contexts e#) (throw e#)))))
If this change was merged in Clojure 1.12, and an AOT-compiled library/app
was created using Clojure 1.12, then anyone running that artifact
in Clojure 1.11 would error since
record-uncaught-exception-contexts does not exist.
testing would have been designed with a companion function
that receives a thunk, like the previous
It might look like this:
(defn testing* [string f] (binding [*testing-contexts* (conj *testing-contexts* string)] (try (f) (catch Throwable e (record-uncaught-exception-contexts e) (throw e))))) (defmacro testing-v3 [string & body] `(testing* ~string #(do ~@body)))
However, the ship has sailed on this approach, since old Clojure runtimes
do not define
The compromise I am proposing resolves the helper function at runtime and ignores
it on older runtimes:
(defmacro testing-v4 [string & body] `(binding [*testing-contexts* (conj *testing-contexts* ~string)] (try (do ~@body) (catch Throwable e# (when-some [f# (resolve 'record-uncaught-exception-contexts)] (f# e#)) (throw e#)))))
is another good example of a difficult macro to grow:
- It predates
with-bindings, so uses
push-thread-bindings. Can we update it to
with-bindings? It will break anyone running Clojure 1.2 with AOT-compiled sources.
- It uses private vars in its expansion. We cannot break the interface for these private vars for forwards compatibility reasons: AOT-compiled sources in past Clojure versions will break if we run it with a future Clojure.
On the plus side, we could fix its use of
~@body to not leak implementation details
because all versions of Clojure understand
It’s possible that in practice this dimension of macro evolvability is not particularly important beyond contributing to Clojure, ClojureScript, or commerical libraries like Datomic, due to the rarity of distributing AOT compiled sources. However, if you find yourself bumping up against this awkward problem, it can be difficult to test for and remedy. It may be worth paying for an extra few indirections in your next macro.