[SBT] How to use provided libraries in run configurations

"Libraries in provided scope" is a popular concept used in variety of SBT projects. Some notable examples are: projects using sbt-assembly plugin to produce fat jars; projects using libraries like spark, predictionIO and others to perform data manipulation; etc. This tutorial will show you how to setup a run configuration in order to run or debug this kind of projects in IDEA.

Project configuration

Before start working in IDEA I'd like to show you example project for this tutorial. It is a simple Scala application which depends on Google's Guava library. It is being said that Guava is present in the environment where this application will be executed, but Scala is not. Therefore I need to make a fat jar that contains Scala library, but not Guava. I will use sbt-assembly plugin in order to do this.

The project consists of 3 files: build.sbt, project/plugins.sbt and src/main/scala/Main.scala. The contents of these files are presented below.

build.sbt

name := "provided-libs-example"

lazy val guavaAndDependencies = Seq(
  "com.google.guava" % "guava" % "18.0",
  "com.google.code.findbugs" % "jsr305" % "1.3.9"
)

libraryDependencies ++= guavaAndDependencies.map(_ % "provided")

run in Compile <<= Defaults.runTask(fullClasspath in Compile, mainClass in (Compile, run), runner in (Compile, run))

project/plugins.sbt

addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.13.0")

src/main/scala/Main.scala

object Main {
  def main(args: Array[String]): Unit = {
    import com.google.common.base.Strings
    assert(Strings.emptyToNull("") != null)
    println("YAY!")
  }
}

Now I can run this application in SBT repl using run command and produce a fat jar using assembly command. Sadly, but the application is not working: when I run it I expect "YAY!" to be printed, but instead I got an error :(

Developing in IDEA: naive approach

OK, my colleagues said that IDEA has a debugger, so I imported this project in IDEA in order to debug this sophisticated application. To do this I created a simple run configuration:

But when I hit "Debug" it fails with the following error:

Exception in thread "main" java.lang.NoClassDefFoundError: com/google/common/base/Strings
	at Main$.main(Main.scala:5)
	at Main.main(Main.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at com.intellij.rt.execution.application.AppMain.main(AppMain.java:140)
Caused by: java.lang.ClassNotFoundException: com.google.common.base.Strings
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 7 more

Seems like Guava is not loaded. Sadly, but IDEA's SBT support is not smart enough to automatically detect my shaman's dances around run SBT task, so I need to tell it where Guava is. I open "Project Settings", but then I remember that...

IDEA synchronize SBT project one-way only

Yes, that's it. No matter how hard I will tweak my project in IDEA all these changes will be overwritten on next refresh. The only way I can save my settings is to store them in build.sbt.

Developing in IDEA: the right way

The main trick here is to create another subproject that will depend on the main subproject and will have all its provided libraries in compile scope. To do this I add the following lines to build.sbt:

lazy val mainRunner = project.in(file("mainRunner")).dependsOn(RootProject(file("."))).settings(
  libraryDependencies ++= guavaAndDependencies.map(_ % "compile")
)

Now I refresh project in IDEA and slightly change previous run configuration so it will use new mainRunner module's classpath:

Now the project runs and I am able to debug!

Q: But what about other guys in my team?

It is very likely that you're not the only one working on your project. In a situation when modification of build.sbt is not acceptable you can do the following:

• Create idea.sbt file
• Put lazy val mainRunner = ... lines into it
• Add idea.sbt to .gitignore or .hgignore

Thanks to SBT multiple .sbt files are concatenated into one and interpreted. Now you won't spam build.sbt, but still be able to develop in IDEA.