How to publish to Amazon S3 with sbt

Question

As explained here, one can deploy to Amazon S3 with maven.

How can we do the same with sbt, that is, to publish to S3 with sbt?

Rhinarium

Answer 1

Here is an SBT Plugin I wrote for publishing to Amazon S3: https://github.com/frugalmechanic/fm-sbt-s3-resolver

It's similar to the already mentioned sbt-s3-resolver but is Apache 2.0 Licensed (instead of AGPL) and is available on Maven Central. It's also a little easier to use and configure.

Publishing to S3

publishTo := Some("S3" at "s3://s3-us-west-2.amazonaws.com/YOUR_BUCKET/repo")

Resolving from S3

resolvers += "S3" at "s3://s3-us-west-2.amazonaws.com/YOUR_BUCKET/repo"

Enable the Plugin

Just add this to your project/plugins.sbt file:

addSbtPlugin("com.frugalmechanic" % "fm-sbt-s3-resolver" % "0.19.0")

Configure AWS Credentials

There are multiple ways to configure the AWS Credentials which are documented on the GitHub Page.

One method is to create an ~/.sbt/.s3credentials that looks like:

accessKey = XXXXXXXXXX
secretKey = XXXXXXXXXX

The credentials file will be automatically picked up by the plugin and you will be able to resolve and publish.

Burden

I love your solution, tpunder; it required the least configuration of all the solutions listed here to get going. – Smocking

The only solution that worked for me! (resolving maven repo on s3) thank you! – Washwoman

there's no scala 2.11 release :/ – Monometallic

@Monometallic What are you trying to do? AFAIK SBT still uses Scala 2.10 for plugins. – Burden

@Burden I was trying to use it in my project and we use scala 2.11. I'm relatively new to SBT - so you're saying I can use this plugin in my project? – Monometallic

@Monometallic Yes. I've used the plugin on Scala 2.10, 2.11, and 2.12 projects without problems. – Burden

Thanks, sbt confuses me. I need more practice with it. – Monometallic

@Burden sbt 1.5.0 can't resolve resolvers += "FrugalMechanic Snapshots" at "s3://maven.frugalmechanic.com/snapshots" Is it still alive ? – Sprage

@Sprage That was just an example and not something you should actually try pointing to. – Burden

Answer 2

The question is pretty old, so may be you already found some workaround, but may be this answer will be useful for somebody else.

We also had such problem in our team and we just created an sbt-plugin for that: sbt-s3-resolver. We were using it for a while and it seems to do it's work fine. It can

• publish ivy/maven artifacts to S3 (private/public) buckets
• resolve ivy artifacts from private buckets (because from public buckets you can resolver with standard sbt resolvers)

Take a look at the usage instructions in readme and open an issue if something is missing.

Immix

While this plugin seems exactly what I need, unfortunately its license is a bit problematic for commercial use... My company is saying "AGPL 3.0 will never be approved even as a tool."... – Maclean

This saved me a bunch of time -- thanks @Immix – Fenwick

@connor-doyle, I'm glad that it's useful for you! – Immix

@michael-yakobi, I'm not sure if I understood you right, but as far as I know, APGLv3 doesn't restrict anyhow usage of software, it restricts you if you distribute it, change the code, etc. – Immix

Answer 3

I was able to get this to work using the sbt-s3 plugin

https://github.com/sbt/sbt-s3

Here's an example:

import sbt._
import Keys._
import play.Project._

import com.typesafe.sbt.S3Plugin._
import S3._
import sbtassembly.Plugin.AssemblyKeys
import sbtassembly.Plugin.AssemblyKeys._
import sbtassembly.Plugin.assemblySettings
import sbtassembly.Plugin.MergeStrategy
import sbtbuildinfo.Plugin._

object ApplicationBuild extends Build {

  val appName = "og-ws"
  val appVersion = "1.0-SNAPSHOT"

  val appDependencies = Seq(
    // Add your project dependencies here,
    jdbc,
    anorm,
    "com.netflix.astyanax" % "astyanax-cassandra" % "1.56.28",
    "com.netflix.astyanax" % "astyanax-thrift" % "1.56.28",
    "com.netflix.astyanax" % "astyanax-entity-mapper" % "1.56.28")

  val main = (
    play.Project(appName, appVersion, appDependencies)
    settings (s3Settings: _*)
    settings (assemblySettings: _*)
    settings (
      // call its setters to configure it, see api docs above etc.
      publish := (),
      publishLocal := (),
      mappings in upload <<= dist map { distFile =>
        Seq(distFile -> "snapshot/%s/%s/%s-%s.zip".format(appName, appVersion, appName, appVersion))
      },
      host in upload := "plaor.maven.s3.amazonaws.com",
      credentials += Credentials(Path.userHome / ".s3credentials")))
}

Stepparent

Could you add content of s3Settings ? – Sprage

sorry this was 8 years ago. It seems like I used and implicit for that object but I expect it just used the s3 credentials file. – Stepparent

Answer 4

I haven't tried this, but from looking at the api docs, this might work:

1. Get http://www.jarvana.com/jarvana/view/org/springframework/aws/spring-aws-ivy/1.0/spring-aws-ivy-1.0-javadoc.jar!/org/springframework/aws/ivy/S3Resolver.html (see http://repo1.maven.org/maven2/org/springframework/aws/spring-aws-ivy/1.0/ --- you should be able to download the jar in project/lib/, or put "org.springframework.aws" % "spring-aws-ivy" % "1.0" in project/build.sbt).

2. Add this to your build.sbt:

   {
     val s3r = new org.springframework.aws.ivy.S3Resolver
     // call its setters to configure it, see api docs above etc.
     publishTo := Some(new sbt.RawRepository(s3r)
   }
   

Again, I haven't tried this, but since publishTo is a SettingKey[Option[Resolver]], and RawRepository extends Resolver and takes an org.apache.ivy.plugins.resolver.DependencyResolver, and S3Resolver implements DependencyResolver, I'm guessing that it would work.

After all, sbt is really just a wrapper around Apache Ivy.

Bearberry

Answer 5

I too had to do upload an assembly jar to s3 bucket from our build tool. @dres answer helped me to go in the right direction.

I got it working with the following in build.sbt file.

As you can see, here I have made use of SettingKey[String] values available in sbt-assembly

// s3Upload
import S3._

s3Settings
mappings in upload := Seq(
  (assemblyOutputPath in assembly).value -> 
       s"${name.value}/${version.value}/${(assemblyJarName in assembly).value}"
)

host in upload := "my-bucket.s3.amazonaws.com"

S3.progress in S3.upload := true

• assemblyOutputPath in assembly gives you the full output path of your assembly jar file
• assemblyJarName in assembly gives you the jar name produced by assembly.
• you need to invoke .value to get the actual value from SBT keys

This uploads assembly jar to my-bucket under name/version/name-assembly-version.jar, with the command sbt s3Upload.

Note that, I also have set my AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in the environment. The order in which credentials are looked up can be found here.

Kerekes