Deploying, starting and stopping Scala applications on a remote server

Question

What the preferred way is to deploy Scala applications on a remote Linux server.

This is a fairly simple, but limited, way of deploying a Scala app on a remote server (nice for quick testing of not-so-sensitive projects):

1. From the remote server I pull my source from git
2. Using the sbt-assembly plug-in I build a jar on the server

3. I then run the Scala application using nohup, which allows you to exit the remote session without terminating the process:

   nohup java -jar myapp.jar > myapp.log 2> myapp.err < /dev/null &

Firstly, what is the best way to stop the process once it is running, considering it's using resources such as databases etc. Do I just lookup the java process id and nuke it?

Secondly, what is the best way to start a java application automatically on restart. I recall using init.d in the past, but remember getting some uphill since it was a java application.

Update:

I missed the elephant in the room here. I'm using the Spray library, which in turns uses Akka, so that provides a number of interesting options.

Glutenous

How about installing app as service or sending shutdown message through TCP socket? – Campobello

Regarding your first requirement of stopping a process at any time, I recommend using screen and run your jar in the screen session. Use ctrl+a k to kill a window and the processes running in it. – Inflexed

Thanks @Kane. I've forgotten about that one. That's a cool idea, especially since you can share screens with other developers (if memory serves me right) – Glutenous

Answer 1

There is a number of ways to skin a cat...

You could use the sbt-start-script https://github.com/sbt/sbt-start-script or even sbt-native-packager https://github.com/sbt/sbt-native-packager

You could wrap Spray's Boot example script in a simple init.d script that calls sbt as detailed in this answer https://mcmap.net/q/564095/-how-to-deploy-my-spray-api-into-production, or just use the plain nohup java command.

You could create larger daemon aware classes and scripts, or extend those with init.d scripts that use Jsvc http://commons.apache.org/proper/commons-daemon/jsvc.html or Java Service Wrapper. http://wrapper.tanukisoftware.com/

An example of a daemon and application class:

package com.example.myapplication.server

import akka.actor.{Props, ActorSystem}
import spray.can.Http
import akka.io.IO
import com.example.myapplication.api.MyServiceActor
import org.apache.commons.daemon._

trait ApplicationLifecycle {
  def start(): Unit
  def stop(): Unit
}

abstract class AbstractApplicationDaemon extends Daemon {
  def application: ApplicationLifecycle

  def init(daemonContext: DaemonContext) {}

  def start() = application.start()

  def stop() = application.stop()

  def destroy() = application.stop()
}

class ApplicationDaemon() extends AbstractApplicationDaemon {
  def application = new Application
}

object ServiceApplication extends App {

  val application = createApplication()

  def createApplication() = new ApplicationDaemon

  private[this] var cleanupAlreadyRun: Boolean = false

  def cleanup(){
    val previouslyRun = cleanupAlreadyRun
    cleanupAlreadyRun = true
    if (!previouslyRun) application.stop()
  }

  Runtime.getRuntime.addShutdownHook(new Thread(new Runnable {
    def run() {
      cleanup()
    }
  }))

  application.start()
}


class Application() extends ApplicationLifecycle with Logging {

  private[this] var started: Boolean = false

  private val applicationName = "MyApplication"

  implicit val actorSystem = ActorSystem(s"$applicationName-system")

  def start() {
    logger.info(s"Starting $applicationName Service")

    if (!started) {
      started = true

      val myService = actorSystem.actorOf(Props[MyServiceActor], "my-service")

      IO(Http) ! Http.Bind(myService, interface = "0.0.0.0", port = 8280)
    }
  }

  def stop() {
    logger.info(s"Stopping $applicationName Service")

    if (started) {
      started = false
      actorSystem.shutdown()
    }
  }

}

If you deploy the jar (use sbt-assembly for a fat jar) in /opt/myapplication/myapplication.jar, add some external configurations in the /etc/mycompany folder, then you can wrap that in an /etc/init.d/myapplication script, for example using Jsvc:

#!/bin/sh
### BEGIN INIT INFO
# Provides:          myapplication
# Required-Start:    $local_fs $remote_fs $network
# Required-Stop:     $local_fs $remote_fs $network
# Should-Start:      $named
# Should-Stop:       $named
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Control myapplication
# Description:       Control the myapplication daemon.
### END INIT INFO

set -e

if [ -z "${JAVA_HOME}" ]; then
        JAVA_HOME=$(readlink -f /usr/bin/java | sed "s:/bin/java::")
fi
JAVA_OPTS="-Xms512m -Xmx1024m"

APP=myapplication

PID=/var/run/${APP}.pid
OUT_LOG=/var/log/myapplication/${APP}_out.log
ERR_LOG=/var/log/myapplication/${APP}_err.log

DAEMON_USER=yourserviceuser

APP_LOG_CONFIG=/etc/mycompany/${APP}_logback.xml
APP_CONFIG=/etc/mycompany/${APP}.conf
APP_HOME=/opt/${APP}
APP_CLASSPATH=$APP_HOME/${APP}.jar
APP_CLASS=com.example.myapplication.server.ApplicationDaemon

if [ -n "$APP_LOG_CONFIG}" ]; then
        JAVA_OPTS="-Dlogback.configurationFile=${APP_LOG_CONFIG} ${JAVA_OPTS}"
fi

DAEMON_ARGS="-home ${JAVA_HOME} -Dconfig.file=${APP_CONFIG} ${JAVA_OPTS} -pidfile ${PID} -user ${DAEMON_USER} -outfile ${OUT_LOG} -errfile ${ERR_LOG} -cp ${APP_CLASSPATH} ${APP_CLASS}"

. /lib/lsb/init-functions

case "$1" in
        start)
                log_daemon_msg "Starting ${APP}"
                cd ${APP_HOME} && jsvc ${DAEMON_ARGS}
                log_end_msg 0
                ;;
        stop)
                log_daemon_msg "Stopping ${APP}"
                cd ${APP_HOME} && jsvc -stop ${DAEMON_ARGS}
                log_end_msg 0
                ;;
        *)
                log_success_msg "Usage:  {start|stop}"
                echo "Usage:  {start|stop}"
                exit 1
                ;;
esac

exit 0

With this you can now sudo service myapplication start|stop

And if as mentioned that you want it to automatically start on boot then run this command

sudo update-rc.d myapplication defaults

This daemon approach works with the Spray apps I have.

Assembled

Excellent answer! Unfortunately I don't quite get the "add some external configurations in /etc/mycompany folder" part. Are you referring to jar's etc? Still great answer. – Glutenous

The external configurations are optional and referenced by APP_LOG_CONFIG and APP_CONFIG ie a logback and spray's application.conf configuration that may override the one inside your jar's configuration. But you need your build.sbt and Build.scala to listen to those javaoptions as well. If you do not need them you can simplify your DAEMON_ARGS and JAVA_OPTS by removing their reference. – Assembled

Answer 2

If maven is suitable then following plugin can be used: http://evgeny-goldin.com/wiki/Sshexec-maven-plugin

Possible it can be easily ported to sbt

Campobello