Invalid header field value in Go ONLY on kubernetes/CoreOS

Question

I have a Go program that uses aws-sdk-go to talk to dynamodb. Dependencies are vendored. Go version 1.7.1. aws-sdk-go version 1.6.24. The program works as expected in all the following environments:

• dev box from shell (Arch Linux)
• docker container running on my dev box (Docker 1.13.1)
• Ec2 instance from shell (Ubuntu 16.04)

When I run the docker container on kubernetes (same one I tested on my dev box), I get the following error:

2017/03/02 22:30:13 DEBUG ERROR: Request dynamodb/GetItem:
---[ REQUEST DUMP ERROR ]-----------------------------
net/http: invalid header field value "AWS4-HMAC-SHA256 Credential=hidden\n/20170302/us-east-1/dynamodb/aws4_request, SignedHeaders=accept-encoding;content-length;content-type;host;x-amz-date;x-amz-target, Signature=483f56dd0b17d8945d3c2f2044b7f97e531190602f132a4d5f828264b3a2cff2" for key Authorization
-----------------------------------------------------

2017/03/02 22:30:13 DEBUG: Response dynamodb/GetItem Details:
---[ RESPONSE ]--------------------------------------
HTTP/0.0 000 status code 0
Content-Length: 0

Based on:
https://golang.org/src/net/http/transport.go
https://godoc.org/golang.org/x/net/lex/httplex#ValidHeaderFieldValue

It looks like the problem is with the header value validation, yet I am at a loss to understand why it works everywhere except on my k8s cluster. The cluster is composed of Ec2 instances running the latest CoreOS stable ami (CoreOS stable 1235.8.0)

The docker image that works on my dev machine is scratch based. To troubleshoot I created an image based on Ubuntu latest with a separate go program that just does a simple get item from dynamodb. When this image is run on my k8s cluster and the program run from an interactive shell, I get the same errors. I have confirmed I can ping the dynamodb endpoints from this env.

I am having a hard time troubleshooting this issue: am I missing something stupid here? Can someone point me in the right direction or have an idea of what is going on?

Hicks

Answer 1

The \n after hidden is certainly invalid. Not sure if it is actually there or somehow got inserted when you were cleansing for posting.

Consider:

package main

import (
    "fmt"
    "golang.org/x/net/lex/httplex"
)

func main() {
    fmt.Println("Is valid (without new line)", httplex.ValidHeaderFieldValue("AWS4-HMAC-SHA256 Credential=hidden/20170302/us-east-1/dynamodb/aws4_request, SignedHeaders=accept-encoding;content-length;content-type;host;x-amz-date;x-amz-target, Signature=483f56dd0b17d8945d3c2f2044b7f97e531190602f132a4d5f828264b3a2cff2"))
    fmt.Println("Is valid (with new line)", httplex.ValidHeaderFieldValue("AWS4-HMAC-SHA256 Credential=hidden\n/20170302/us-east-1/dynamodb/aws4_request, SignedHeaders=accept-encoding;content-length;content-type;host;x-amz-date;x-amz-target, Signature=483f56dd0b17d8945d3c2f2044b7f97e531190602f132a4d5f828264b3a2cff2"))
}

One guess would be wherever the real hidden value is getting pulled from (config file etc) mistakenly has the \n in there and it's happily getting pulled into your header, but only in this case.

Poon

the \n was in the header before I cleansed for posting. I would find it odd that it was inserted only on the k8s environment though since everything else should be the same (especially given the docker deploy), but it is a lead I will track down! – Hicks

creating a k8s secret from a file was adding a \n after the secret data, which was then getting inserted into the header. – Hicks

Answer 2

remember the "-n" when you do this: echo -n key | base64

Clamp

This does not seem related to the question – Quackenbush

This is related and was likely the origin of the issue: using echo without the -n flag adds a newline to the output (for pretty display in the terminal) and was getting piped along with the other data to base64 – Hicks

That was exactly the problem for me. I was encoding secrets doing "echo key | base64" and adding "-n" did the trick. Thank you! – Bethannbethanne

Never heard of this before, but indeed it helped me to fix the issue. Thanks! – Auspicate