Port forwarding in CICD (Github Actions)

Question

I want to run db migrations in Github Actions. The DB is behind a bastion.

My solution was to forward Postgres port 5432 to the db host through the bastion.

I tried below script but does not seem to work.

mkdir ~/.ssh
ssh-keyscan -H <bastion_ip>  >> ~/.ssh/known_hosts
echo "${{secrets.BASTION_SSH_KEY}}" >> key
chmod 600 ./key
ssh -T -i ./key -L 5432:<db_host_url>:5432 user@<bastion_ip> &
make migrate
rm ./key

make migrate runs migration against localhost:5432.

When I run the pipeline I get following error

Error:  AssertionError [ERR_ASSERTION]: ifError got unwanted exception: connect ECONNREFUSED 127.0.0.1:5432

Anyway to fix it? I am open to other ways of doing it.

Ludwigg

What does "does not seem to work" mean? Did it fail with an error? Did it successfully connect to the bastion host? – Myrtice

@Myrtice I have updated the post with the error I got. Looking at it, it seems the port forwarding didn't work. – Ludwigg

Answer 1

Thanks @larsks, I got it working. There were a couple of things I had to change to get it working.

1. added -fN as suggested by @larsks
2. used ssh-agent to handle the key

Below is the working code snippet:

mkdir ~/.ssh
ssh-keyscan -H <bastion_ip> >> ~/.ssh/known_hosts
eval `ssh-agent -s`
ssh-add - <<< "${{secrets.BASTION_SSH_KEY}}"
ssh -fN -v -L 5432:<db-host>:5432 user@<bastion_ip>
make migrate

Ludwigg

Answer 2

I think your ssh command is incorrect, try this:

ssh -fN -i ./key -L 5432:<db_host>:5432 user@<bastion_ip>

From the man page:

-f    Requests ssh to go to background just before command execution.
      This is useful if ssh is going to ask for passwords or passphrases,
      but the user wants it in the background.  This implies -n.  The
      recommended way to start X11 programs at a remote site is with
      something like ssh -f host xterm.

And:

-N    Do not execute a remote command.  This is useful for just
      forwarding ports.

Myrtice

Thanks it was helpful. Also you rsuggestion to use verbose mode helped me realize that the way I was passing ssh key didn't work. – Ludwigg

Answer 3

Let me add several recommendations to the answers above, which seem quite reasonable. I had a similar case and needed to run one shell command on the server and transfer files to it during GitHub Actions workflow. The server was also behind the gateway (bastion).

• carefully troubleshoot the Hostkey verification failed error if you encounter any. If you use custom port for SSH connections for the intermediate server (bastion), then don't forget to add it to the ssh-keyscan command options.
• after establishing SSH tunnel it may be crucial to pause the workflow a bit, especially when the next command uses this tunnel right away! It is enough to use sleep command. It was decisive for me as without sleep I got connection refused.
• do not expect that in the next step in GitHub Actions job the tunnel will be alive, even if you put it to the background (by means either ssh -f or & options; screen also didn't work for me). It appears that new step is running in another shell, while the processes invoked in previous steps are terminated. So, it is better to run shell commands (like scp) in the same step where the tunnel was created, instead of using custom actions from GitHub Marketplace (since they require a new step). Perhaps there is yet a custom action which allows to create SSH tunnel with port forwarding, but I didn't find one.

With that being said, this is the piece of code that finally worked for me :

- name: Creating SSH key and establishing SSH tunnel
       run: |
         mkdir -p ~/.ssh/
         echo "${SSH_PRIVATE_KEY}" > ~/.ssh/id_rsa
         chmod 600 ~/.ssh/id_rsa
         echo "$(ssh-keyscan -p ${BASTION_PORT} ${BASTION_IP})" >> ~/.ssh/known_hosts
         ssh -fNL 2222:${SERVER_IP}:22 <user>@${BASTION_IP} -p ${BASTION_PORT} 
         sleep 5
         echo "$(ssh-keyscan -p 2222 localhost)" >> ~/.ssh/known_hosts
         ssh -p 2222 ${SERVER_USER}@localhost "<shell command>"
         scp -P 2222 -r <source_folder> ${SERVER_USER}@localhost:<target_folder>
       shell: bash
       env:
         SSH_PRIVATE_KEY: ${{ secrets.SSH_PRIVATE_KEY }}
         BASTION_IP: ${{ secrets.BASTION_IP }}
         BASTION_PORT: ${{ secrets.BASTION_PORT }}
         BASTION_USER: ${{ secrets.BASTION_USER }}
         SERVER_IP: ${{ secrets.SERVER_IP }}
         SERVRE_USER: ${{ secrets.SERVER_USER }}

Incurrent