Connect to Neptune on AWS from local machine

Question

I am trying to connect to Neptune DB in AWS Instance from my local machine in office, like connecting to RDS from office. Is it possible to connect Neptune db from local machine? Is Neptune db publicly available? Is there any way a developer can connect Neptune db from office?

Stockpile

Answer 1

Neptune does not support public endpoints (endpoints that are accessible from outside the VPC). However, there are few architectural options using which you can access your Neptune instance outside your VPC. All of them have the same theme: setup a proxy (EC2 machine, or ALB, or something similar, or a combination of these) that resides inside your VPC, and make that proxy accessible from outside your VPC.

It seems like you want to talk to your instance purely for development purposes. The easiest option for that would be to spin up an ALB, and create a target group that points to your instance's IP.

Brief Steps (These are intentionally not in detail, please refer to AWS Docs for detailed instructions):

1. dig +short <your cluster endpoint> This would give you the current master's IP address.

2. Create an ALB (See AWS Docs on how to do this).

3. Make your ALB's target group point to the IP Address obtained for step #1. By the end of this step, you should have an ALB listening on PORT-A, that would forward requests to IP:PORT, where IP is your database IP (from Step 1) and PORT is your database port (default is 8182).
4. Create a security group that allows inbound traffic from everywhere. i.e. Inbound TCP rule for 0.0.0.0 on PORT-A.
5. Attach the security group to your ALB

Now from your developer boxes, you can connect to your ALB endpoint at PORT-A, which would internally forward the request to your Neptune instance.

Do checkout ALB docs for details around how you can create it and the concepts around it. If you need me to elaborate any of the steps, feel free to ask.

NOTE: This is not a recommended solution for a production setup. IP's used by Neptune instances are bound to change with failovers and host replacements. Use this solution only for testing purposes. If you want a similar setup for production, feel free to ask a question and we can discuss options.

Astragal

I would be interested in the recommended production setup @big-K. – Lighthouse

I tried this and worked fine for few days. But suddenly it stopped working. As I noticed target health checks are failing. Finally I managed to work it back by entering /?gremlin=g.V%28%29.count%28%29, the URL encoded version of /?gremlin=g.V().count() to health check path – Biserrate

SSH tunnel on a t3.nano bastion host will be cheaper than an ALB – Logy

For the ALB mother, do you just forward all TCP traffic, or have to deal with setting up TLS and a certificate? If Neptune only accepts TLS connections, how should a local client (say a Gremlin web app such as GraphExp) connect to the ALB for development purposes? – Charest

@PrestonLee Thats an interesting question. Would suggest posting it a question with a lot more details on your setup, ALB listener/forwards and the details around how you test things. The use case is pretty standard, so my hope is that ALB is able to make TLS requests to the backing endpoint, and then terminate that session on the ALB. Between ALB and the client, you might just end up using ALB's cert. Best if you share more details and get one of the experts to take a look. – Astragal

I ended up configuring my own reverse proxy and it worked fine, but took a lot of trial and error. – Charest

Glad to know it finally worked - it should not be too hard of a setup, so if you ever feel like you need more specifics on how it works, feel free to post a Q. – Astragal

We have created a AWS VPN to each VPC. This, we found, is the best solution for us, as we can have developers from all different locations without a worry of 8182 exposed to the public. – Nefertiti

Answer 2

Reference: https://github.com/M-Thirumal/aws-cloud-tutorial/blob/main/neptune/connect_from_local.md

Connect to AWS Neptune from the local system

There are many ways to connect to Amazon Neptune from outside of the VPC, such as setting up a load balancer or VPC peering.

Amazon Neptune DB clusters can only be created in an Amazon Virtual Private Cloud (VPC). One way to connect to Amazon Neptune from outside of the VPC is to set up an Amazon EC2 instance as a proxy server within the same VPC. With this approach, you will also want to set up an SSH tunnel to securely forward traffic to the VPC.

Part 1: Set up a EC2 proxy server.

Launch an Amazon EC2 instance located in the same region as your Neptune cluster. In terms of configuration, Ubuntu can be used. Since this is a proxy server, you can choose the lowest resource settings.

Make sure the EC2 instance is in the same VPC group as your Neptune cluster. To find the VPC group for your Neptune cluster, check the console under Neptune > Subnet groups. The instance's security group needs to be able to send and receive on port 22 for SSH and port 8182 for Neptune. See below for an example security group setup.

Lastly, make sure you save the key-pair file (.pem) and note the directory for use in the next step.

Part 2: Set up an SSH tunnel.

This step can vary depending on if you are running Windows or MacOS.

1. Modify your hosts file to map localhost to your Neptune endpoint.

   Windows: Open the hosts file as an Administrator (C:\Windows\System32\drivers\etc\hosts)

   MacOS: Open Terminal and type in the command: sudo nano /etc/hosts

   Add the following line to the hosts file, replacing the text with your Neptune endpoint address.

   127.0.0.1 localhost YourNeptuneEndpoint

   Open Command Prompt as an Administrator for Windows or Terminal for MacOS and run the following command. For Windows, you may need to run SSH from C:\Users\YourUsername\

   ssh -i path/to/keypairfilename.pem ec2-user@yourec2instanceendpoint -N -L 8182:YourNeptuneEndpoint:8182

   The -N flag is set to prevent an interactive bash session with EC2 and to forward ports only. An initial successful connection will ask you if you want to continue connecting? Type yes and enter.

   To test the success of your local graph-notebook connection to Amazon Neptune, open a browser and navigate to:

   https://YourNeptuneEndpoint:8182/status

   You should see a report, similar to the one below, indicating the status and details of your specific cluster:

    {
      "status": "healthy",
      "startTime": "Wed Nov 04 23:24:44 UTC 2020",
      "dbEngineVersion": "1.0.3.0.R1",
      "role": "writer",
      "gremlin": {
        "version": "tinkerpop-3.4.3"
      },
      "sparql": {
        "version": "sparql-1.1"
      },
      "labMode": {
        "ObjectIndex": "disabled",
        "DFEQueryEngine": "disabled",
        "ReadWriteConflictDetection": "enabled"
      }
    }
   

Close Connection

When you're ready to close the connection, use Ctrl+D to exit.

Wamble

Thanks for the details. Instead of changing this file C:\Windows\System32\drivers\etc\hosts, I changed C:\Users\user_name\.ssh\config file as below: Host 10.100.128.00 Hostname 10.100.128.00 Port 22 User ec2-user IdentityFile ~/.ssh/my-ec2.pem then I ran: ssh -L 8182:<neptune endpoint>:8182 <bastion server ip> Now the status check worked but how can i use this connection to achieve other things in Neptune? Can someone help with that? – Argumentum

I get the response: "Access Denied: missing authentication token" – Plasmosome

Can you specify on which step, you got access denied? or create issues here with screenshot to help github.com/M-Thirumal/aws-cloud-tutorial/blob/main/neptune/… – Wamble

Answer 3

As already mentioned you can't access directly outside your VPC.
The following link describes another solution using a SSH tunnel: connecting-to-aws-neptune-from-local-environment. I find it much easier for testing and development purpose. You can create the SSH tunnel with Putty as well.

Spineless

This is the method I use, as the ALB is more expensive than a t3.nano bastion host – Logy

Followed the same method but at the end got no response on localhost:8182. Are there any ways to find out where I'm lagging? – Orwin

I suggest connecting with SSH to the bastion and checking you have access from there to the Neptune endpoint. If it works, this means you probably have some problem creating the tunnel to the bastion server (you can try creating the tunnel from MobaXterm or putty which provides UI for that). @Orwin – Spineless

Does this still work or is sigv4 required now? – Pneumoconiosis

Answer 4

Hi you can connect NeptuneDB by using gremlin console at your local machine. USE THIS LINK to setup your local gremlin server, it works for me gremlin 3.3.2 version Only you have to update the remote.yaml as per your url and port

Presurmise