Building and testing sbt plugins

If you use sbt, there is a chance that eventually you will create a plugin for it. Whether it is a new member of the community plugins (opens new window) family, or an internal helper used by your organization only, there is one thing you will most likely need — tests. As a maintainer of the sbt-updates (opens new window) plugin, I have spent some time writing tests for it, and I would like to share my learnings and things to pay attention to.

Unit tests

Before moving on to heavy artillery, if you can test anything in isolation with unit tests, do it. You can use your favorite testing framework, they run much faster, and usually have better support from IDEs and sbt itself.

Scripted tests

The next step is writing scripted tests, which can verify how your plugin behaves inside sbt. The official documentation has a fairly detailed guide (opens new window) on writing scripted tests that I will not repeat here.

Probably the only noteworthy thing is that it makes sense to keep a balance between the number of scripted tests and the number of steps in each test. If your tests are very granular, you pay a noticeable penalty of sbt initialization for every test. If you have fewer tests with more steps inside, they are faster altogether, but you cannot easily run individual subtests.

Building for multiple sbt versions

If you haven't done anything special, your plugin is built and tested against the same sbt version that is used by your build. Usually, you have this version set in project/build.properties, and it is quite tempting to use the latest sbt version to benefit from the improvements that land in sbt every now and then. This, however, is not a safe choice for building sbt plugins because of sbt/sbt#5049 (opens new window), at least at the time of writing. If your plugin is built against sbt 1.3.x, it will not work with sbt 1.2.8 and older. You can still use the latest sbt for development while building a plugin against an older version by setting pluginCrossBuild / sbtVersion := "1.2.8".

A similar question arises if you want to support sbt 0.13. Cross-building of sbt plugins is also well documented (opens new window) on the sbt website. You define the crossSbtVersions := Vector("1.2.8", "0.13.18") setting and prefix your commands with ^:

$ sbt ^scripted

Under the hood, it will modify pluginCrossBuild / sbtVersion and run the command for both sbt versions.

Testing with multiple sbt versions

Now the tricky part starts. How could we find out that our plugin is broken because of sbt/sbt#5049 (opens new window)? Until now the scripted tests were always executed with the same version of sbt as used during the build. Ideally, we should build the plugin once (or twice, if we support sbt 0.13), and then run scripted tests for multiple sbt versions. For example, we may want to run them for the minimal supported sbt version and for the latest one.

By default, when you run scripted tests two things happen — first, your plugin is built and published to your local Ivy repository, and then the tests are executed with this published version. We need to break this dependency to be able to use different sbt versions for these stages:

scriptedDependencies := {}

Now we can test plugin compatibility with different sbt versions:

$ sbt ^publishLocal ^^1.0.0 scripted ^^1.3.10 scripted

The same command can of course be executed in your CI pipeline. There is a variation of this approach, when you let your CI test sbt versions compatibility, but locally you prefer the convenience of a single scripted command. The can be achieved with

scriptedDependencies := Def.taskDyn {
  if (insideCI.value) Def.task(())
  else Def.task(()).dependsOn(publishLocal)
}.value

Testing with multiple JDK versions

We tested how our plugin works with different sbt versions, but there is at least one other important component that may be different for your plugin users — a JDK. You really need to test your plugin with different JDK versions, otherwise, it is pretty easy to start using some Java 8 APIs that are not available in Java 11.

You need to have a way to switch from one JDK version to another. I personally use Nix (opens new window) but there are other popular tools like SDKMAN (opens new window) or Jabba (opens new window):

$ jabba use zulu@1.8
$ sbt ^publishLocal ^^1.0.0 scripted ^^1.3.10 scripted
$ jabba use zulu@1.11
$ sbt ^^1.0.0 scripted ^^1.3.10 scripted

Same as before, you can do it in your CI pipeline. It is even possible to run scripted tests for different sbt and JDK versions in parallel — just take the result of sbt publishLocal from ~/.ivy2/local and use it for as many test combinations as you wish. The exact solution will, of course, depend on the CI system you use.

Project matrix

There is one thing that, as I have always thought, contradicts the elegant concept of sbt — cross-building. You have a declarative and parallelizable graph of tasks and their dependencies, but to cross-build, you need to imperatively mutate the state of your build with a ^^ command.

There are a couple of sbt plugins that improve the cross-building support in sbt. The most flexible one, in my opinion, is sbt-projectmatrix (opens new window) created by Eugene Yokota (opens new window). The typical use-cases for this plugin are cross-building for different scala versions, platforms, or incompatible library versions. We can, however, apply it to cross-building and cross-testing of sbt plugins.

First, we need to define a cross-building "axis" in project/SbtAxis.scala:

case class SbtAxis(version: String, idSuffix: String, directorySuffix: String)
    extends VirtualAxis.WeakAxis

object SbtAxis {
  def apply(version: String): SbtAxis =
    SbtAxis(version, version.replace('.', '_'), version)
}

With this case class, we will define the sbt version we want to use for cross-building:

lazy val `sbt-1.3.10` = SbtAxis("1.3.10")
lazy val `sbt-1.2.8`  = SbtAxis("1.2.8")
lazy val `sbt-1.0.0`  = SbtAxis("1.0.0")

We will also need some extension methods for the ProjectMatrix class:

implicit class RichProjectMatrix(val matrix: ProjectMatrix) extends AnyVal {
  def sbtPluginRow(axis: SbtAxis,
                   process: Project => Project = identity
  ): ProjectMatrix =
    matrix.customRow(
      autoScalaLibrary = false,
      axisValues = Seq(axis, VirtualAxis.jvm),
      project =>
        process(
          project.settings(
            sbtPlugin := true,
            scalaVersion := axisScalaVersion(axis),
            pluginCrossBuild / sbtVersion := axis.version
          )
        )
    )
  def sbtScriptedRow(axis: SbtAxis,
                     buildAxis: SbtAxis,
                     process: Project => Project = identity
  ): ProjectMatrix =
    sbtPluginRow(
      axis,
      project =>
        process(
          project
            .enablePlugins(ScriptedPlugin)
            .settings(
              publish / skip := true,
              compile / skip := true,
              scriptedDependencies := Def
                .task(())
                .dependsOn(matrix.finder(buildAxis)(false) / publishLocal)
                .value
            )
        )
    )
}

The first extension method sbtPluginRow adds a subproject that will build your scala plugin for a particular sbt version. Due to some details of sbt-projectmatrix implementation, we have to specify the scala version used by sbt explicitly. Also, we don't enable the SbtPlugin as we would do in a normal case — it would also bring the ScriptedPlugin, but now we want to have scripted tests separated from the plugin build.

The second extension method sbtScriptedRow adds a subproject that will only run scripted tests and it takes two arguments — an sbt version for running the tests, and an sbt version of the sbtPluginRow.

With these two methods, we can finally define our main project matrix:

lazy val `my-plugin` = (projectMatrix in file("."))
  .sbtPluginRow(`sbt-1.2.8`)
  .sbtScriptedRow(`sbt-1.0.0`, `sbt-1.2.8`)
  .sbtScriptedRow(`sbt-1.3.10`, `sbt-1.2.8`)

Now we can run scripted tests for both sbt versions with just one command:

$ sbt scripted

It will build the plugin against sbt 1.2.8, publish it to the local Ivy repository, and run scripted tests both for sbt 1.0.0 and 1.3.10.

Of course, we can still run tests for a single sbt version as well:

$ sbt my-plugin1_3_10JVM/scripted

The generated name of the project is not very friendly. This can be improved, but I left out those details for simplicity.

The advantages of this approach:

• You don't need to remember to run sbt commands in some particular order,
• Incremental compilation is no longer affected by constant switching from one version to another,
• Compilation and tests run in parallel as much as possible.

If you want to see a real-life example, you can check the sbt-updates plugin source code (opens new window). There I use GitHub Actions to cross-build the plugin for sbt 0.13.x and sbt 1.x, and to test it against four different sbt and three different JDK versions.