Promtail error pipeline stage must only contain one key

Question

When trying to setup Promtail I get the following error:

level=error ts=2020-11-27T06:10:30.310583Z caller=main.go:104 msg="error creating promtail" error="failed to make file target manager: pipeline stage must contain only one key"

I am running the following from the command line.

promtail-windows-amd64.exe --config.file=../conf/promtail-local-config.yml

My log lines looks like this:

13:21:03.183 - INFO - Successfully received document from '127.0.0.1'. Saved as 'c:\test\test_file.txt' in 102ms from address '/127.0.0.1'

13:21:05.275 - WARN - Failed to receive document from '127.0.0.1'. Error creating file c:\test\error_file.txt'

My config looks as follows:

scrape_configs:
- job_name: promtailTest
  pipeline_stages:
  - match:
    selector: '{job="promtailTest"}'
    stages:
    - regex:
      expression: '^(?P<timestamp>\\d{2}:\\d{2}:\\d{2}\\.\\d{3})\\s\\-\\s(?P<logLevel>[A-Z]{4,5})\\s\\-\\s(?P<logMessage>.*)$'
    - labels:
      logLevel:
  static_configs:
  - targets:
      - localhost
    labels:
      job: promtailTest
      app: promtailTest
      host: LOCAL
      __path__: C:/test/logs/*log

When I take out the pipeline_stages: section then I do see the lines in grafana however I cannot get the regex part to work. I actually want to add a label for the logging Level (so I can count the errors)

Font

Answer 1

BAH!! I am a fool. I think the problem was with my format (YAML can be a real PITA!!!)

My new config now looks like this and it works.

scrape_configs:

  - job_name: promtailTest
    static_configs:
    - targets:
        - localhost
      labels:
        job: promtailTest
        app: promtailTest
        host: APOLLO99
        __path__: D:/SXI/XPress/Dispatch/logs/*log
    pipeline_stages:
    - match:
        selector: '{job="promtailTest"}'
        stages:
        - regex:
            expression: "^^(?P<myTime>\\d{2}:\\d{2}:\\d{2}\\.\\d{3})\\s\\-\\s(?P<logLevel>[A-Z]{4,5})\\s\\-\\s(?P<logMessage>.*)$$"
        - labels:
            logLevel:

Font

No, you're not. I don't think it's YAML as much as the fact that the different keys go together in non-obvious ways, and there aren't a lot of examples out there. I'm just trying to scrape a local multi-line log and I keep getting this error message. – Ocrea

I agree with @J.Gwinner you have just moved around your yml and changed some values.. But yea I'm getting the same error and it's not 100% clear what the issue is – Naturally

I'd post what I found to work - but as of the 10th I no longer have access to that source code :( – Ocrea

I got a similar error - for me it was the indentation – Horsepowerhour

Answer 2

I have been fighting with this error, and, even if the accepted answer is correct, it was not clear to me where the error was.

As many have said the problem is in the indentation, and the fact that YAML is quite confusing.

If you compare the YAML in the question and the YAML in the answer, you see that everything under the - match: is indented with two extra spaces in the correct answer. The same happens with the lines after the - regex: and - labels. These two extra spaces are important because otherwise match and selector would be at the same level (and they should be one inside the other).

Everything becomes clear when you convert it to JSON.

This is the YAML of the question converted to JSON:

{
  "pipeline_stages": [
    {
      "match": null,
      "selector": "{job=\"promtailTest\"}",
      "stages": [
        {
          "regex": null,
          "expression": "^(?P<timestamp>\\\\d{2}:\\\\d{2}:\\\\d{2}\\\\.\\\\d{3})\\\\s\\\\-\\\\s(?P<logLevel>[A-Z]{4,5})\\\\s\\\\-\\\\s(?P<logMessage>.*)$"
        },
        {
          "labels": null,
          "logLevel": null
        }
      ]
    }
  ]
}

As you can see, pipeline_stages is an array where the first item has 3 keys (at the same level): match, selector and stages. Promtail expects only 1 key here (match) and this is why it says "pipeline stage must only contain one key".

This is the correct answer converted to JSON :

{
  "pipeline_stages": [
    {
      "match": {
        "selector": "{job=\"promtailTest\"}",
        "stages": [
          {
            "regex": {
              "expression": "^^(?P<myTime>\\d{2}:\\d{2}:\\d{2}\\.\\d{3})\\s\\-\\s(?P<logLevel>[A-Z]{4,5})\\s\\-\\s(?P<logMessage>.*)$$"
            }
          },
          {
            "labels": {
              "logLevel": null
            }
          }
        ]
      }
    }
  ]
}

Now the pipeline_stages is an array as well, but the first item has only one key match which is what Promtail expects. Now selector and stages are inside the match.

Ordinate