Change remote IP wieh using L2TP VPN with docker

Question

I have an L2TP server set up with docker-compose, and nginx to filter certain hosts to a hostname, but when I try to connect, nginx is reading the original IP, not the IP proxied through the VPN.

Nginx showing x.x.x.x instead of 192.168.x.x for the IP.

As a result, it's giving me a 403 (forbidden) error when I try to connect on any remote IP that isn't the ones I allowed, even while connected to the VPN, and even when the VPN gives me an IP such like 192.168.43.12

And when I try network_mode: host on the VPN, it fails to route any web traffic at all.

docker-compose.yml:

services:
    vpn:
        image: hwdsl2/ipsec-vpn-server
        restart: always
        env_file:
          - ../config/vpn/vpn.env
        ports:
          - "500:500/udp"
          - "4500:4500/udp"
          - "1701:1701/udp"
        privileged: true
        hostname: example.com
        volumes:
          - /lib/modules:/lib/modules:ro
    nginx:
        build: ../config/nginx
        restart: unless-stopped
        ports:
         - "80:80"
        network_mode: host

nginx site conf:

server {
    listen *:80;

    server_name             bt.example.com;

    index                   index.html;

    access_log              /dev/stdout upstreamlog;
    error_log               /dev/stderr debug;

    location / {
        allow 127.0.0.1;
        allow 192.168.0.0/16;
        #allow x.x.x.x;      # one remote IP I want to allow, normally uncommented
        deny all;

        proxy_pass              http://localhost:9091;
        proxy_set_header        Host $host;
        proxy_set_header        X-Real-IP $remote_addr;
        proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header        X-Forwarded-Proto $scheme;
    }

}

Barbarian

VPN IP and external IP could still be different, so not sure if you are doing this correctly. The IP assigned to you by VPN is more like a private one, and then routed through another external IP. So outgoing traffic to/from VPN will still see a real ip. But for outgoing traffic it won't be the same as your internet external IP. – Stearn

@TarunLalwani It's mostly I need nginx run in docker to filter out all connections that aren't coming from the vpn (also in docker). – Barbarian