Is there a way to setup a remote quarkus postgresql dev-service?

Question

Due to security concerns, my company doesn't allows to use containers on our laptops.

So we can't use the normal quarkus:dev to run our test that connects to Postgresql.

But they provides us a remote machine where we can use Podman to run some containers.

What I'm doing now is to manually ssh to that machine and starting a Postgresql container before running local tests.

What I would like is to do this automatically, and also find a way to do the same on Jenkins when when we need to run a pipeline to release a new version.

Rundle

Automatically when? And surely it's just a matter of writing a short script and getting Jenkins to run that? Can you explain where you have gotten to and which bit is causing you problems? – Siobhan

Well, when I said automatically, it is the same way that is done when we have a docker locally, after we start the server with quarkus:test or quarkus:dev. Yep, for Jenkins, running on linux, I can use a bash script and use maven exec to call it before the tests and using a service user account. But I can't think a way to do the same locally on our windows since we need to interactively pass the password to the ssh. – Rundle

Sorry, you're still not making any sense. "the same way" without saying what that way is currently means nothing to me. And I'm pretty sure that Windows does let you run an ssh-agent. I certainly did so with putty some 10 years ago (which was the last time I used Windows extensively). – Siobhan

Richard, what happens when we type quarkus dev or quarkus test? It will enter in Dev mode and then start all dev-service, right ? This is what I mean with "the same way". So, how can we hook up bat or cmd scripts to these two commands ? I managed a way to copy each dev ssh pub key to our podman server, so the ssh is not an issue anymore. – Rundle

I have no idea what a "quarkus" is - I'm reading this based on the podman tag. You need to assume that most readers of multiply tagged questions won't be familiar with most of them. You can run podman commands remotely ("podman-remote") or if quarkus doesn't let you customise what it is doing you could wrap the call to it with a script that tunnels the docker-compatible socket back from your container server, or run quarkus on that server via a shell script. All of those are fairly obvious though and you've not mentioned which you've tried or why they didn't work for you. – Siobhan

Answer 1

This is a common scenario, and the intention of Quarkus's developer joy features is to allow it to work in a frictionless way, without requiring scripts or manual tunneling.

There are two options, although which one works best for you will depend a bit on how your company's remote podman is set up.

1. Remote dev services. (When you run Quarkus in dev mode, the automatic provisioning of unconfigured services is called 'dev services'.) The idea here is that you use dev services normally, but under the covers, the container client is connecting to the remote instances. For Dev services, Testcontainers provides container connectivity under the covers. This should work transparently as long as podman run works. You'd set it up using something like

podman system connection add remote --identity ~/.ssh/my-key ssh://my-host/podman/podman.sock
podman system connection default remote

If you don't have a local podman client, or if the podman connection settings don't sort it out, setting DOCKER_HOST to the right remote socket will also tell Testcontainers where to look.

1. Remote dev mode. Here, the whole application is running in a container on the remote server. Changes to your local files are reflected in the remote instance.

To use remote dev mode, you build a special jar and then launch it in the remote environment. Add the following to your application.properties:

%dev.quarkus.package.type=mutable-jar 

Then build the jar (they could be in the application.properties, but then you couldn't commit it to source control):

QUARKUS_LIVE-RELOAD_PASSWORD=<arbitrary password> ./mvnw install

The install will build you a normal fast-jar dockerfile. Run it in your remote environment with QUARKUS_LAUNCH_DEVMODE=true added to the podman run command.

Then, locally, instead of mvn quarkus:dev, you'd run ./mvnw quarkus:remote-dev -Dquarkus.live-reload.url=http://my-remote-host:8080

https://quarkus.io/guides/maven-tooling#remote-development-mode has a more complete set of instructions. This option does have more moving parts and more latency, since you're transferring your whole application to the remote server every time code changes. So if you can, just configuring podman and using remote dev services is probably better.

A third option, which probably isn't relevant for you, is to use Testcontainers Cloud. Quarkus dev services use Testcontainers under the covers, and Testcontainers Cloud is a convenient way of running Testcontainers remotely.

Rahr

I was able to setup podman-remote on my windows laptop and I'm connecting to our podman server through ssh perfectly now, with this address and my id_ed25519 key: ssh://myuser@myhost:22/run/user/35268/podman/podman.sock. I have set DOCKER_HOST=ssh://myuser@myhost:22. But I'm getting this error: EnvironmentAndSystemPropertyClientProviderStrategy: failed with exception IllegalArgumentException (Unsupported protocol scheme: ssh://myuser@myhost:22)As no valid configuration was found, execution cannot continue. what am I missing ? – Rundle

Hi @Cristiano, it seems the docker java client doesn't support ssh.. It's a bit ambiguous about whether testcontainers does. One suggestion is to set up ssh forwarding. On Linux it would be ``` ssh -nNT -L $(pwd)/docker.sock:/var/run/docker.sock user@someremote export DOCKER_HOST=$(pwd)/docker.sock ``` (Be sure to manually delete the local socket file.) I'm not sure if there is a windows equivalent that could help? Full details of that are here: medium.com/@dperny/… – Rahr

Hello @Holly Cummins. Finally I got it working on my windows. :D. I needed to use this: ssh -N -L 127.0.0.1:3307:/run/user/11111/podman/podman.sock cristianoga@mymachine. Then set DOCKER_HOST=tcp://127.0.0.1:3307. and TESTCONTAINERS_HOST_OVERRIDE=mymachine. thank you – Rundle

So glad you got it working! I'll look to add those instructions to our docs. – Rahr