dissoc

Copilot coding agent is a service provided by GitHub that supervises Large Language Models within Actions runners to complete prompted coding tasks. Users can interact with the LLM using standard GitHub features like issue assignments, comments and pull request reviews, but also handy buttons littered across the UI (look for the cloud-surrounded >_ icon).

While these familiar user controls make coding agents immediately accessible, what really sets this apart from other approaches is security. Agents are sandboxed and firewalled within their runners, with write access only to their copilot/* branch. Even once they push, CI triggers still require explicit approval from an authorized user. Furthermore, repo secrets are sequestered from the coding agent, with only explicitly enabled secrets accessible to the coding agent (via the copilot environment at github.com/<org>/<repo>/settings/environments). GitHub have done a great job applying the fork+pr security model for coding agents—maintainers can treat coding agents as any other untrusted collaborator.

To get started, first check your Copilot plan at github.com/settings/copilot. You may have complimentary access as an open source developer (I did). Copilot coding agents use “premium requests”, as of writing one per prompt. Here you can see your quota, if any.

Go to github.com/copilot/agents to write a prompt. Pick a Clojure repo and write a detailed prompt. Easier said than done, so let’s walk through an example.

In previous explorations, I found LLM’s did not know how to use clojure.java.process in Clojure scripts, so I created a repo of examples for training data in frenchy64/clojure.java.process-examples. The problem was, I didn’t know how to use it either, so I was hoping a coding agent could figure it out by itself. Here’s my initial prompt:

This repository is meant for supplementary documentation and tests for the new Clojure API clojure.java.process. Part of the problem is that the documentation is sparse and AI agents have very little example code to train on. So this task will be challenging to complete for an AI agent: I want you to fill out the test suite covering examples of a wide range of use-cases, especially common, for clojure.java.process. I have set the repo up to help you. You must install Java and Clojure. Then you should use clojure -M:test to run the tests. It uses cognitect’s test-runner so filter the tests as needed. Please run the test suite as you work. Please cover examples of every function in clojure.java.process, and every option it supports. As well as common use-cases, I’m especially interested in ones that coordinate concurrent processes. For example, a process that is running a Clojure REPL accepting input from and piping out to from another process that is reading from stdin (make sure you use clojure and not clj to avoid issues with readline). Look at these places for inspiration: Java documentation for Process, babashka’s process library. Add at least 100-150 tests. The tests should pass relatively quickly, aim for 10-20 seconds. Faster is better. Finally, enhance the README.md with all the examples you use, ensuring they match exactly to the example in the test suite (but strip away the testing scaffolding if it’s clearer to use in the documentation).

I find higher success when giving coding agents ways to check their own work, in this case running tests. This prompt was only the beginning though—you can find subsequent prompts in Copilot’s pull request. The PR evolved into machine-checked documentation suitable for both human consumption and training data. Notice you should mention @copilot only once per prompt to get its attention, especially relevant for requesting changes over multiple files.

You should set up Copilot’s environment via .github/workflows/copilot-setup-steps.yml to save having to include it in the prompt and help the LLM get right on task. The setup steps execute just before the firewalled coding agent gains control. Typed Clojure’s Copilot setup steps is a good starting point. It installs several distributions of Clojure and downloads deps from clojars. fully-satisfies’ Copilot setup steps additionally shows how to set up Leiningen. With this setup, you can keep firewall exceptions minimal. If an agent coding session requires a new clojars dep, it should add them to your deps and the next session will download them.

Here are some tips. Setup steps must be on the default branch. To use on forks of open source projects, change the default branch to one that includes the setup steps. The timeout-minutes of the setup steps dictates coding agent’s session duration. I found 30 minutes to be a good starting point, modifying your prompts to direct the agent to break up tasks into chunks if needed. Rate limits are problematic, especially on non-Enterprise plans. In practice, I found 2-3 concurrent coding agents stay within these limits, bumping up to 4-5 on Enterprise plans.