Setting up private Github access with AWS Elastic Beanstalk and Ruby container

Question

Going by a recent tutorial on setting up AWS Elastic Beanstalk for Ruby deployment using Git, I just set up a Elastic Beanstalk environment from my CI server. However, the application failed to start. I went through the logs to find that bundle install was failing with an error message.

Fetching [email protected]:example/private-repository.git Host key verification failed. fatal: The remote end hung up unexpectedly [31mGit error: command git clone '[email protected]:example/private-repository.git' "/var/app/ondeck/vendor/cache/ruby/1.9.1/cache/bundler/git/private-repository-e4bbe6c2b13bb62664e39e345c1b01d80017934c" --bare --no-hardlinks in directory /var/app/ondeck has failed.[0m

Gemfile of my Rails application contains references to gemified plugins hosted on a couple of my owned private repositories on Github. Something like

gem 'somegemname', :git => '[email protected]:example/private-repository.git'

I had faced similar issues with Capistrano deployments which were resolved by setting up ssh_options[:forward_agent] = true.

AWS Elastic Beanstalk Ruby container supports custom configuration through custom .config files placed under .ebextensions. Would setting up an SSH forward agent help in this case? Are there any other alternatives to reach a private Github repository while starting an Elastic Beanstalk environment?

Update 1: I just checked for the user with which a bundle install is initiated. Found out that a script /opt/elasticbeanstalk/hooks/appdeploy/pre/10_bundle_install.sh starts bundle install as root user. I tried creating an SSH Key under /root/.ssh and added it's pub-key to Github Deploy keys for that repository. No luck so far. Will now try to add an SSH pub-key to my user account on Github so that it applies to all private repositories accessible through my Github account.

Westbound

Answer 1

After a good day of effort, I finally enabled use of my organization's private GitHub repos with Elastic Beanstalk by just using a .config file. I am using Python and pip, but it should also work for other package installers on EB.

rhetonik's ssh-agent+ssh-add approach did not work for me at all, so I elected to set up an ssh configuration file instead.

Here is my .ebextensions/3-pip-install-from-github.config file:

files:
    "/root/.ssh/config":
        owner: root
        group: root
        mode: "000600"
        content: |
            Host github.com
                User git
                Hostname github.com
                IdentityFile /root/.ssh/github

commands:
    01-command:
        command: sudo ssh-keyscan -H github.com >> /root/.ssh/known_hosts
    02-command:
        command: sudo chmod 644 /root/.ssh/known_hosts
    03-command:
        command: sudo aws s3 cp s3://bucket-with-your-github-ssh-key/github /root/.ssh
    04-command:
        command: sudo chmod 600 /root/.ssh/github

Rough instructions:

• Set up an S3 bucket accessible by your EB instance. Inside of that bucket, store the SSH key allowing access to the GitHub repository you want to access via pip, npm, bundle, etc. Use sudo aws s3 cp to copy that key onto your EB instance on deploy. sudo is necessary because EB scripts use root and not ec2-user.

• This ebextensions config file also creates 2 files on your EB instance. /root/.ssh/config tells ssh (invoked by pip and git) to use the key you copied from S3. Storing the output of ssh-keyscan -H github.com into /root/.ssh/known_hosts will pre-verify that ssh on your EB instance is actually communicating with GitHub to avoid MITM attacks. This is better than disabling StrictHostKeyChecking in /root/.ssh/config.

Here is my requirements.txt file for pip:

Beaker==1.7.0
Flask==0.10.1
Jinja2==2.7.3
MarkupSafe==0.23
# [...]
git+ssh://[email protected]/myorganization/[email protected]

While running eb-deploy, you can tail -f /var/log/eb-activity.log to make sure everything runs smoothly.

Carreon

that's a great answer, thanks for sharing it. For some reasons this solution does not work for me: ssh is keeping asking me for a password of the key file, but it has no password. that's my output of ssh -v github.com: debug1: Trying private key: /root/.ssh/github debug1: key_parse_private2: missing begin marker debug1: key_parse_private_pem: PEM_read_PrivateKey failed debug1: read PEM private key done: type <unknown> Enter passphrase for key '/root/.ssh/github': – Hass

The output of ´ssh-keyscan -H github.com´, could you maybe add an example of that? Not sure if my syntax is correct when adding my own output. – Athens

This saved my life. Thanks so much. I struggled for a full day following advice found elsewhere which suggested that the .ssh folded that git will use on EB should be located under /tmp or under /home/ec2-user. Putting it under /root solved all my issues. My related question was #43498163 THANK YOU! – Brendabrendan

Thank you for this great answer, saved me A LOT of time :). Here's an improvement that you could add, thanks to some ElasticBeanstalk new feature that came after your answer: the download from the SSH key file can now be done through a files directive instead of a command : see the last example on AWS-EB customization docs: docs.aws.amazon.com/elasticbeanstalk/latest/dg/… – Kedah

"Set up an S3 bucket accessible by your EB instance." Can you explain how to do so? – Louque

@Erez.S #21653676 – Carreon

Answer 2

Here's how I finally did it. It's all about setting up an SSH Key for the user which is responsible for bundle install phase.

1. Start an environment for an application in AWS Elastic Beanstalk
2. Optional - Login to Amazon EC2 console and change instance type to a desired value
3. Update SSH Key pair name to enable remote SSH login. (I'm sure there must be a way to specify instance type and SSH key pair name while starting an environment)
4. Look for the newly launched instance either in EC2 console or through CLI, note the Fully Qualified Domain Name (FQDN) for this instance. EB instances are like any other instance you would create with Amazon EC2. Login via SSH to this instance.
5. Execute the following commands to create an SSH key for root user

   $ sudo su - root

   $ ssh-keygen -t rsa -C "[email protected]"

6. Edit .bash_profile to explicitly start ssh-agent and add the newly generated SSH Key. Add the following lines (This might seem unnecessary, I did it just to be sure)

   eval `ssh-agent

   eval ssh-add ~/.ssh/id_rsa

7. Note the SSH public key E.g.: ~/.ssh/id_rsa.pub and add it to the set of SSH Keys for Github account which has access to private repositories

8. At this point, your instance has access to your private Github repositories. You could test this by issuing a git clone on those repositories by logging in as root user.

9. Create an AMI out of this instance using standard methods

10. Come back to your AWS Elastic Beanstalk Dashboard and look for Edit Configuration option in your application's environment. In the Server tab, look for an option which lets you specify a Custom AMI. Update this field with the newly created AMI ID E.g.: ami-4324fd4.

11. Save configuration by hitting Apply Changes. AWS Elastic Beanstalk would start deploying new instances across your environment and terminating the old ones. This is to ensure all your auto-scaled instances have the whitelisted SSH Key required for private Github access.

After the above steps are done, you could go ahead and deploy your Rails application with git aws.push

Hope this helps others who are stuck. I'd be glad to see a more graceful solution than this one though.

Westbound

Hey, thank you for sharing this. I have one question regarding using custom AMI with Elastic Beanstalk, do you still get security updates on this AMI? – Cacilie

@YannMilin: Apologies for a delayed response. I'm afraid, I have not tried this yet. But I think as long as you are running the base OS, you should continue to receive security updates. – Westbound

It'd be great to see an example of a functional .ebextensions/[name].conf file that accomplishes this. – Moorings

As of '16 this solution is obsolate and never was optimal, since it required manual changes on instance. Every time amazon release new version instance, procedure have to be repeated to be able to use it. In addition proper solution should not require logging in to the instance. Ideally, possibility of SSH login on Elastic Beanstalk instances should not exist. – Kwangju

Answer 3

If you're in a hurry, and your application repo is also private, you can create an additional Github user account and assign it read-only privileges to the repo containing the gem.

Then give bundler the https url with the new account's credentials:

gem 'somegemname', git: "https://username:[email protected]/example/privaterepository"

Radmilla

Answer 4

There are two approaches to authenticating with GitHub. I recommend associating your personal GitHub account with the private GitHub repo in either case.

The first approach passes the same ssh credentials you use locally to push, pull, and so on from the remote GitHub repo -- you uploaded your public key for your personal account, and that's what GitHub uses. To make this work when running on another server, you need to have ssh-agent running and use ssh-add to add your key to the agent -- then your personal GitHub credentials can be used to perform git commands.

The second approach is to allow the remote server(s) you're deploying to to have access to GitHub -- this may be elastic beanstalk or your actual server(s). Create a passwordless ssh key on the server (ssh-keygen -t rsa, accept defaults, or perhaps EB has a way of its own) then copy the generated public key contents and create a new "deploy key" containing that key in your GitHub repo -- you'll need to be admin, which I assume you are. An installed deploy key will allow the EB users to log into the remote server and do git pull and related commands (read-only) from the server.

I think the first method is more elegant and more easily managed as the number of servers you're deploying to grows, but which method you use may depend on EB's options.

Rieger

As mentioned, Capistrano is something I used earlier and those issues were resolved with ssh_options. Here, I am trying to set the same Rails project up with AWS Elastic Beanstalk. Unfortunately, deployment does not seem to involve Capistrano anywhere. – Westbound

Ah, sorry, I misread. I'll edit the answer, although it will say pretty much the same thing without the capistrano stuff – Rieger

Yes, I have been considering deploy keys allowed by Github. However, I will then have to create a custom AMI or at least a custom .ebextensions configuration to accomodate this change. I'm trying to figure out the best way to achieve it. – Westbound

Answer 5

Remove the private github repos from your requirements.txt file and create a script to install them using environmental variables for usernames and passwords.

file: project-root/install-extra-requirements.sh

#!/bin/sh

source /opt/python/run/venv/bin/activate
python ".extra-requirements.py"

file: project-root/.extra-requirements.py

import os

def main():
    github_username = os.environ['GITHUB_USERNAME']
    github_password = os.environ['GITHUB_PASSWORD']
    repository = "git+https://%s:%[email protected]/yourgithubrepo" % (github_username, github_password)
    os.system("pip install %s" % repository)

if __name__ == '__main__':
    main()

file: project-root/.ebextensions/002_container.config

container_commands:
  01_install_extra_requirements:
    command: './install-extra-requirements.sh'

Now you can just set GITHUB_USERNAME and GITHUB_PASSWORD as environmental variables in your elastic beanstalk environment.

Aphelion

It's bad practice to expose a password via env var. AWS Secrets Manager should be used for that. – Garett