How to get access-logs from OpenShift router (HAProxy)?

Question

How to get access-logs from openshift router (HAproxy).

I tried to use this command:

$ oc project default 
$ oc logs router-1-g...

I got output:

I0129 09:47:17.125616       1 router.go:554] Router reloaded:
 - Checking http://localhost:80 ...
 - Health check ok : 0 retry attempt(s).
I0129 09:47:54.356142       1 router.go:554] Router reloaded:
 - Checking http://localhost:80 ...
 - Health check ok : 0 retry attempt(s).

But there was no information about users traffic (client/server requests/responses).

Please give me advice on how I can debug how this proxy is working?

Bluing

Answer 1

You will need to point the router at a syslog server to debug the output. No access logs are output by default. You are seeing the logs of the Go process.

I created a rsyslog container some time ago to help debug issues with a custom router. This will log to stdout for debugging purposes only. Follow the instructions in the readme to deploy this within the default project. Shout if you need any further help.

Campy

Ok thank for your reply. I tried to do as you wrote in the readme, but when container starts I have error: Error: failed to start container "rsyslog": Error response from daemon: {"message":"linux spec user: unable to find user DEFAULT: no matching entries in passwd file"} – Bluing

Error: failed to start container "rsyslog": Error response from daemon: {"message":"linux spec user: unable to find user DEFAULT: no matching entries in passwd file"} – Bluing

Answer 2

On Openshift >=4.5 you can do it this way by edititing your ingresscontroller and add the following .spec.logging (see below, log format example included):

apiVersion: operator.openshift.io/v1
kind: IngressController
metadata:
  name: default
  namespace: openshift-ingress-operator
spec:
  logging:
    access:
      destination:
        type: Container
      # % formats see here: http://cbonte.github.io/haproxy-dconv/2.0/configuration.html#8.2.3
      httpLogFormat: log_source="haproxy-default" log_type="http" c_ip="%ci" c_port="%cp"
        req_date="%tr" fe_name_transport="%ft" be_name="%b" server_name="%s" res_time="%TR"
        tot_wait_q="%Tw" Tc="%Tc" Tr="%Tr" Ta="%Ta" status_code="%ST" bytes_read="%B"
        bytes_uploaded="%U" captrd_req_cookie="%CC" captrd_res_cookie="%CS" term_state="%tsc"
        actconn="%ac" feconn="%fc" beconn="%bc" srv_conn="%sc" retries="%rc" srv_queue="%sq"
        backend_queue="%bq" captrd_req_headers="%hr" captrd_res_headers="%hs" http_request="%r"

This solution is based on https://access.redhat.com/solutions/3250781.
There is a solution for Openshift 3.x as well.

You may access the logs by doing 'oc logs -n openshift-ingress <your-router-pod-name> -c logs'.

It is also possible to send those logs directly to a syslog server:

apiVersion: operator.openshift.io/v1
kind: IngressController
metadata:
  name: default
  namespace: openshift-ingress-operator
spec:
  logging:
    access:
      destination:
        type: Syslog
        syslog:
          address: 1.2.3.4
          port: 10514
      # % formats see here: http://cbonte.github.io/haproxy-dconv/2.0/configuration.html#8.2.3
      httpLogFormat: log_source="haproxy-default" log_type="http" c_ip="%ci" c_port="%cp"
        req_date="%tr" fe_name_transport="%ft" be_name="%b" server_name="%s" res_time="%TR"
        tot_wait_q="%Tw" Tc="%Tc" Tr="%Tr" Ta="%Ta" status_code="%ST" bytes_read="%B"
        bytes_uploaded="%U" captrd_req_cookie="%CC" captrd_res_cookie="%CS" term_state="%tsc"
        actconn="%ac" feconn="%fc" beconn="%bc" srv_conn="%sc" retries="%rc" srv_queue="%sq"
        backend_queue="%bq" captrd_req_headers="%hr" captrd_res_headers="%hs" http_request="%r"

Shanelleshaner

This is helpful as I couldn't much in OpenShift docs. Is it "user-agent":"%[capture.req.hdr(User-Agent)]" correct for adding User-Agent data to Router pod log? – Filings

Didn't notice "%hr" earlier. Is it possible to configure 'User-Agent' header alone for the HAProxy log-format in the operator instance configuration? – Filings

Answer 3

In the openshift 3.11, you could create a new router using oc adm router command with extended logging enabled

1. New router

oc adm router myrouter --extended-logging

2. Enable debug logging

oc set env dc/myrouter ROUTER_LOG_LEVEL=debug

3. tail the logs

oc logs -f myrouter-x-xxxxx -c syslog

Koy