Nginx-Ingress Helm Deployment --tcp-services-configmap Argument not found

Question

I'm trying to do TCP/UDP port-forwarding with an ingress.

Following the docs: https://kubernetes.github.io/ingress-nginx/user-guide/exposing-tcp-udp-services/

It says to set: --tcp-services-configmap but doesn't tell you where to set it. I assume it is command line arguments. I then googled the list of command line arguments for nginx-ingress

https://kubernetes.github.io/ingress-nginx/user-guide/cli-arguments/

Here you can clearly see its an argument of the controller:

--tcp-services-configmap Name of the ConfigMap containing the definition of the TCP services to expose. The key in the map indicates the external port to be used. The value is a reference to a Service in the form "namespace/name:port", where "port" can either be a port number or name. TCP ports 80 and 443 are reserved by the controller for servicing HTTP traffic.

First Question: how do I dynamically add to the container arguments of the nginx-ingress helm chart I don't see that documented anywhere?

Second Question: What is the proper way to set this with the current version of nginx-ingress because setting the command line argument fails the container startup because the binary doesn't have that argument option.

Here in the default helm chart values.yaml there are some options about setting the namespace for the configmap for tcp-services but given the docs say I have to set it as an argument but that argument fails the startup I'm not sure how you actually set this.

https://github.com/kubernetes/ingress-nginx/blob/main/charts/ingress-nginx/values.yaml

I manually edited the deployment and set the flag on the container args:

      - args:
    - -nginx-plus=false
    - -nginx-reload-timeout=60000
    - -enable-app-protect=false
    - -nginx-configmaps=$(POD_NAMESPACE)/emoney-nginx-controller-nginx-ingress
    - -default-server-tls-secret=$(POD_NAMESPACE)/emoney-nginx-controller-nginx-ingress-default-server-tls
    - -ingress-class=emoney-ingress
    - -health-status=false
    - -health-status-uri=/nginx-health
    - -tcp-services-configmap=emoney-node/tcp-services-configmap
    - -nginx-debug=false
    - -v=1
    - -nginx-status=true
    - -nginx-status-port=8080
    - -nginx-status-allow-cidrs=127.0.0.1
    - -report-ingress-status
    - -external-service=emoney-nginx-controller-nginx-ingress
    - -enable-leader-election=true
    - -leader-election-lock-name=emoney-nginx-controller-nginx-ingress-leader-election
    - -enable-prometheus-metrics=true
    - -prometheus-metrics-listen-port=9113
    - -prometheus-tls-secret=
    - -enable-custom-resources=true
    - -enable-tls-passthrough=false
    - -enable-snippets=false
    - -enable-preview-policies=false
    - -ready-status=true
    - -ready-status-port=8081
    - -enable-latency-metrics=false
    env:

When I set this like the docs say should be possible the pod fails to start up because it errors out saying that argument isn't an option of the binary.

kubectl logs emoney-nginx-controller-nginx-ingress-5769565cc7-vmgrf -n emoney-node
flag provided but not defined: -tcp-services-configmap
Usage of /nginx-ingress:
  -alsologtostderr
        log to standard error as well as files
  -default-server-tls-secret string
        A Secret with a TLS certificate and key for TLS termination of the default server. Format: <namespace>/<name>.
            If not set, than the certificate and key in the file "/etc/nginx/secrets/default" are used.
            If "/etc/nginx/secrets/default" doesn't exist, the Ingress Controller will configure NGINX to reject TLS connections to the default server.
            If a secret is set, but the Ingress controller is not able to fetch it from Kubernetes API or it is not set and the Ingress Controller
            fails to read the file "/etc/nginx/secrets/default", the Ingress controller will fail to start.
  -enable-app-protect
        Enable support for NGINX App Protect. Requires -nginx-plus.
  -enable-custom-resources
        Enable custom resources (default true)
  -enable-internal-routes
        Enable support for internal routes with NGINX Service Mesh. Requires -spire-agent-address and -nginx-plus. Is for use with NGINX Service Mesh only.
  -enable-latency-metrics
        Enable collection of latency metrics for upstreams. Requires -enable-prometheus-metrics
  -enable-leader-election
        Enable Leader election to avoid multiple replicas of the controller reporting the status of Ingress, VirtualServer and VirtualServerRoute resources -- only one replica will report status (default true). See -report-ingress-status flag. (default true)
  -enable-preview-policies
        Enable preview policies
  -enable-prometheus-metrics
        Enable exposing NGINX or NGINX Plus metrics in the Prometheus format
  -enable-snippets
        Enable custom NGINX configuration snippets in Ingress, VirtualServer, VirtualServerRoute and TransportServer resources.
  -enable-tls-passthrough
        Enable TLS Passthrough on port 443. Requires -enable-custom-resources
  -external-service string
        Specifies the name of the service with the type LoadBalancer through which the Ingress controller pods are exposed externally.
            The external address of the service is used when reporting the status of Ingress, VirtualServer and VirtualServerRoute resources. For Ingress resources only: Requires -report-ingress-status.
  -global-configuration string
        The namespace/name of the GlobalConfiguration resource for global configuration of the Ingress Controller. Requires -enable-custom-resources. Format: <namespace>/<name>
  -health-status
        Add a location based on the value of health-status-uri to the default server. The location responds with the 200 status code for any request.
            Useful for external health-checking of the Ingress controller
  -health-status-uri string
        Sets the URI of health status location in the default server. Requires -health-status (default "/nginx-health")
  -ingress-class string
        A class of the Ingress controller.
        
            An IngressClass resource with the name equal to the class must be deployed. Otherwise, the Ingress Controller will fail to start.
            The Ingress controller only processes resources that belong to its class - i.e. have the "ingressClassName" field resource equal to the class.
        
            The Ingress Controller processes all the VirtualServer/VirtualServerRoute/TransportServer resources that do not have the "ingressClassName" field for all versions of kubernetes. (default "nginx")
  -ingress-template-path string
        Path to the ingress NGINX configuration template for an ingress resource.
            (default for NGINX "nginx.ingress.tmpl"; default for NGINX Plus "nginx-plus.ingress.tmpl")
  -ingresslink string
        Specifies the name of the IngressLink resource, which exposes the Ingress Controller pods via a BIG-IP system.
            The IP of the BIG-IP system is used when reporting the status of Ingress, VirtualServer and VirtualServerRoute resources. For Ingress resources only: Requires -report-ingress-status.
  -leader-election-lock-name string
        Specifies the name of the ConfigMap, within the same namespace as the controller, used as the lock for leader election. Requires -enable-leader-election. (default "nginx-ingress-leader-election")
  -log_backtrace_at value
        when logging hits line file:N, emit a stack trace
  -log_dir string
        If non-empty, write log files in this directory
  -logtostderr
        log to standard error instead of files
  -main-template-path string
        Path to the main NGINX configuration template. (default for NGINX "nginx.tmpl"; default for NGINX Plus "nginx-plus.tmpl")
  -nginx-configmaps string
        A ConfigMap resource for customizing NGINX configuration. If a ConfigMap is set,
            but the Ingress controller is not able to fetch it from Kubernetes API, the Ingress controller will fail to start.
            Format: <namespace>/<name>
  -nginx-debug
        Enable debugging for NGINX. Uses the nginx-debug binary. Requires 'error-log-level: debug' in the ConfigMap.
  -nginx-plus
        Enable support for NGINX Plus
  -nginx-reload-timeout int
        The timeout in milliseconds which the Ingress Controller will wait for a successful NGINX reload after a change or at the initial start. (default 60000) (default 60000)
  -nginx-status
        Enable the NGINX stub_status, or the NGINX Plus API. (default true)
  -nginx-status-allow-cidrs string
        Add IPv4 IP/CIDR blocks to the allow list for NGINX stub_status or the NGINX Plus API. Separate multiple IP/CIDR by commas. (default "127.0.0.1")
  -nginx-status-port int
        Set the port where the NGINX stub_status or the NGINX Plus API is exposed. [1024 - 65535] (default 8080)
  -prometheus-metrics-listen-port int
        Set the port where the Prometheus metrics are exposed. [1024 - 65535] (default 9113)
  -prometheus-tls-secret string
        A Secret with a TLS certificate and key for TLS termination of the prometheus endpoint.
  -proxy string
        Use a proxy server to connect to Kubernetes API started by "kubectl proxy" command. For testing purposes only.
            The Ingress controller does not start NGINX and does not write any generated NGINX configuration files to disk
  -ready-status
        Enables the readiness endpoint '/nginx-ready'. The endpoint returns a success code when NGINX has loaded all the config after the startup (default true)
  -ready-status-port int
        Set the port where the readiness endpoint is exposed. [1024 - 65535] (default 8081)
  -report-ingress-status
        Updates the address field in the status of Ingress resources. Requires the -external-service or -ingresslink flag, or the 'external-status-address' key in the ConfigMap.
  -spire-agent-address string
        Specifies the address of the running Spire agent. Requires -nginx-plus and is for use with NGINX Service Mesh only. If the flag is set,
                    but the Ingress Controller is not able to connect with the Spire Agent, the Ingress Controller will fail to start.
  -stderrthreshold value
        logs at or above this threshold go to stderr
  -transportserver-template-path string
        Path to the TransportServer NGINX configuration template for a TransportServer resource.
            (default for NGINX "nginx.transportserver.tmpl"; default for NGINX Plus "nginx-plus.transportserver.tmpl")
  -v value
        log level for V logs
  -version
        Print the version, git-commit hash and build date and exit
  -virtualserver-template-path string
        Path to the VirtualServer NGINX configuration template for a VirtualServer resource.
            (default for NGINX "nginx.virtualserver.tmpl"; default for NGINX Plus "nginx-plus.virtualserver.tmpl")
  -vmodule value
        comma-separated list of pattern=N settings for file-filtered logging
  -watch-namespace string
        Namespace to watch for Ingress resources. By default the Ingress controller watches all namespaces
  -wildcard-tls-secret string
        A Secret with a TLS certificate and key for TLS termination of every Ingress host for which TLS termination is enabled but the Secret is not specified.
                Format: <namespace>/<name>. If the argument is not set, for such Ingress hosts NGINX will break any attempt to establish a TLS connection.
                If the argument is set, but the Ingress controller is not able to fetch the Secret from Kubernetes API, the Ingress controller will fail to start.

Config Map

apiVersion: v1
data:
  "1317": emoney-node/emoney-api:1317
  "9090": emoney-node/emoney-grpc:9090
  "26656": emoney-node/emoney:26656
  "26657": emoney-node/emoney-rpc:26657
kind: ConfigMap
metadata:
  annotations:
    meta.helm.sh/release-name: emoney
    meta.helm.sh/release-namespace: emoney-node
  creationTimestamp: "2021-11-01T18:06:49Z"
  labels:
    app.kubernetes.io/managed-by: Helm
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:data:
        .: {}
        f:1317: {}
        f:9090: {}
        f:26656: {}
        f:26657: {}
      f:metadata:
        f:annotations:
          .: {}
          f:meta.helm.sh/release-name: {}
          f:meta.helm.sh/release-namespace: {}
        f:labels:
          .: {}
          f:app.kubernetes.io/managed-by: {}
    manager: helm
    operation: Update
    time: "2021-11-01T18:06:49Z"
  name: tcp-services-configmap
  namespace: emoney-node
  resourceVersion: "2056146"
  selfLink: /api/v1/namespaces/emoney-node/configmaps/tcp-services-configmap
  uid: 188f5dc8-02f9-4ee5-a5e3-819d00ff8b67






Name:              emoney
Namespace:         emoney-node
Labels:            app.kubernetes.io/instance=emoney
                   app.kubernetes.io/managed-by=Helm
                   app.kubernetes.io/name=ibcnode
                   app.kubernetes.io/version=1.16.0
                   helm.sh/chart=ibcnode-0.1.0
Annotations:       meta.helm.sh/release-name: emoney
                   meta.helm.sh/release-namespace: emoney-node
Selector:          app.kubernetes.io/instance=emoney,app.kubernetes.io/name=ibcnode
Type:              ClusterIP
IP:                172.20.30.240
Port:              p2p  26656/TCP
TargetPort:        26656/TCP
Endpoints:         10.0.36.192:26656
Session Affinity:  None
Events:            <none>


Name:              emoney-api
Namespace:         emoney-node
Labels:            app.kubernetes.io/instance=emoney
                   app.kubernetes.io/managed-by=Helm
                   app.kubernetes.io/name=ibcnode
                   app.kubernetes.io/version=1.16.0
                   helm.sh/chart=ibcnode-0.1.0
Annotations:       meta.helm.sh/release-name: emoney
                   meta.helm.sh/release-namespace: emoney-node
Selector:          app.kubernetes.io/instance=emoney,app.kubernetes.io/name=ibcnode
Type:              ClusterIP
IP:                172.20.166.97
Port:              api  1317/TCP
TargetPort:        1317/TCP
Endpoints:         10.0.36.192:1317
Session Affinity:  None
Events:            <none>


Name:              emoney-grpc
Namespace:         emoney-node
Labels:            app.kubernetes.io/instance=emoney
                   app.kubernetes.io/managed-by=Helm
                   app.kubernetes.io/name=ibcnode
                   app.kubernetes.io/version=1.16.0
                   helm.sh/chart=ibcnode-0.1.0
Annotations:       meta.helm.sh/release-name: emoney
                   meta.helm.sh/release-namespace: emoney-node
Selector:          app.kubernetes.io/instance=emoney,app.kubernetes.io/name=ibcnode
Type:              ClusterIP
IP:                172.20.136.177
Port:              grpc  9090/TCP
TargetPort:        9090/TCP
Endpoints:         10.0.36.192:9090
Session Affinity:  None
Events:            <none>


Name:                     emoney-nginx-controller-nginx-ingress
Namespace:                emoney-node
Labels:                   app.kubernetes.io/instance=emoney-nginx-controller
                          app.kubernetes.io/managed-by=Helm
                          app.kubernetes.io/name=emoney-nginx-controller-nginx-ingress
                          helm.sh/chart=nginx-ingress-0.11.3
Annotations:              meta.helm.sh/release-name: emoney-nginx-controller
                          meta.helm.sh/release-namespace: emoney-node
Selector:                 app=emoney-nginx-controller-nginx-ingress
Type:                     LoadBalancer
IP:                       172.20.16.202
LoadBalancer Ingress:     lb removed
Port:                     http  80/TCP
TargetPort:               80/TCP
NodePort:                 http  32250/TCP
Endpoints:                10.0.43.32:80
Port:                     https  443/TCP
TargetPort:               443/TCP
NodePort:                 https  32375/TCP
Endpoints:                10.0.43.32:443
Session Affinity:         None
External Traffic Policy:  Local
HealthCheck NodePort:     30904
Events:                   <none>


Name:              emoney-rpc
Namespace:         emoney-node
Labels:            app.kubernetes.io/instance=emoney
                   app.kubernetes.io/managed-by=Helm
                   app.kubernetes.io/name=ibcnode
                   app.kubernetes.io/version=1.16.0
                   helm.sh/chart=ibcnode-0.1.0
Annotations:       meta.helm.sh/release-name: emoney
                   meta.helm.sh/release-namespace: emoney-node
Selector:          app.kubernetes.io/instance=emoney,app.kubernetes.io/name=ibcnode
Type:              ClusterIP
IP:                172.20.42.163
Port:              rpc  26657/TCP
TargetPort:        26657/TCP
Endpoints:         10.0.36.192:26657
Session Affinity:  None
Events:            <none>





  helm repo add nginx-stable https://helm.nginx.com/stable --kubeconfig=./kubeconfig || echo "helm repo already added"

  helm repo update --kubeconfig=./kubeconfig || echo "helm repo already updated"

  helm upgrade ${app_name}-nginx-controller -n ${app_namespace} nginx-stable/nginx-ingress \
  --install \
  --kubeconfig=./kubeconfig \
  --create-namespace \
  --set controller.service.type=LoadBalancer \
  --set controller.tcp.configMapNamespace=${app_namespace} \
  --set controller.ingressClass="${app_name}-ingress"

  kubectl rollout status -w deployment/${app_name} --kubeconfig=./kubeconfig -n ${app_namespace}


  #- --tcp-services-configmap=emoney-node/tcp-services-configmap

Bleeder

Did provided answer help you? – Leadership

Answer 1

You could say the helm chart is biased in that it doesn't expose the option to set those args as chart value. It will set them by itself based on conditional logic when it's required according to the values.

When I check the nginx template in the repo, I see that additional args are passed from the template in the params helper file. Those seem to be generated dynamically. I.E.

{{- if .Values.tcp }}
- --tcp-services-configmap={{ default "$(POD_NAMESPACE)" .Values.controller.tcp.configMapNamespace }}/{{ include "ingress-nginx.fullname" . }}-tcp
{{- end }}

So, it seems it will use this flag only if the tcp value isn't empty. On the same condition, it will create the configmap.

Further, the tcp value allows you to set a key configMapNamespace. So if you were to set this key only, then the flag would be used as per paramaters helpers. Now you need to create your configmap only in the provided namespace and let it match the name {{ include "ingress-nginx.fullname" . }}-tcp.

So you could create the configmap in the default namespace and name it ingress-nginx-tcp or similar, depending on how you set the release name.

kubectl create configmap ingress-nginx-tcp --from-literal 1883=mqtt/emqx:1883 -n default
helm install --set controller.tcp.configMapNamespace=default ingress-nginx ingress-nginx/ingress-nginx

I think the only problem with that is that you cannot create it in the .Release.Namespace, since when tcp isn't empty it will attempt to create a configmap there by itself, which would result in conflicts. At least that's how I interpret the templates in the chart repo.

I personally, have configured TCP via values file that I pass to helm with -f.

helm install -f values.yaml ingress-nginx ingress-nginx/ingress-nginx

# configure the tcp configmap
tcp:
  1883: mqtt/emqx:1883
  8883: mqtt/emqx:8883

# enable the service and expose the tcp ports.
# be careful as this will pontentially make them 
# availble on the public web
controller:
  service:
    enabled: true
    ports:
      http: 80
      https: 443
      mqtt: 1883
      mqttssl: 8883
    targetPorts:
      http: http
      https: https
      mqtt: mqtt
      mqttssl: mqttssl

Sarmentose

Answer 2

You can use helm to proxy TCP with ingress-nginx (which is not the same thing as nginx-ingress) by passing the TCP ports as a value. For example, to proxy TCP 6005 to the service examplesvc in namespace examplens, you would use:

helm -n ingress-nginx upgrade ingress-nginx ingress-nginx/ingress-nginx \
    --set tcp.6005="examplens/examplesvc:6005"

Using the tcp value will create the configmap automatically, and pass --tcp-services-configmap to the controller. It will also add the port to the ingress LoadBalancer service. Passing --tcp-services-configmap alone (which you can do via controller.extraArgs) will not add the port to the ingress, which is probably not what you want.

Mistreat