Programmatic access from a service account to a Google IAP protected resource denied with invalid signature error

Question

I recently enabled IAP in GKE cluster.

• Cluster Version: 1.15.11-gke.11

I followed the instructions here: https://cloud.google.com/iap/docs/enabling-kubernetes-howto

Service config is as follows:

---
apiVersion: cloud.google.com/v1beta1
kind: BackendConfig
metadata:
  name: foo-bc-iap
  namespace: foo-test
spec:
  iap:
    enabled: true
    oauthclientCredentials:
      secretName: iap-client-secret
---
apiVersion: v1
kind: Service
metadata:
  name: foo-internal-service
  namespace: foo-test
  annotations:
    cloud.google.com/backend-config: '{"ports":{"80":"foo-bc-iap"}}'
spec:
  type: NodePort # To create Ingress using the service.
  selector:
    app: foo-test
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8081

The credential I used was OAuth 2.0 Client ID (Type: Web Application).

After making sure the IAP-protected API endpoint works differently when I activate IAP on the Kubernetes service, I wrote the following test program to make sure the endpoint is accessible from the service account given in the JSON file 'account.json'.

In writing this sample application, I consulted this doc: https://cloud.google.com/iap/docs/authentication-howto#iap_make_request-go

• google.golang.org/api v0.23.0
• go 1.12

func (m *myApp) testAuthz(ctx *cli.Context) error {
    audience := "<The client ID of the credential mentioned above>"
    serviceAccountOption := idtoken.WithCredentialsFile("account.json")

    client, err := idtoken.NewClient(ctx.Context, audience, serviceAccountOption)
    if err != nil {
        return fmt.Errorf("idtoken.NewClient: %v", err)
    }

    requestBody := `{
        <some JSON payload>
    }`

    request, err := http.NewRequest("POST", "https://my.iap.protected/endpoint",
        bytes.NewBuffer([]byte(requestBody)))
    if err != nil {
        return fmt.Errorf("http.NewRequest: %v", err)
    }

    request.Header.Add("Content-Type", "application/json")

    response, err := client.Do(request)
    if err != nil {
        return fmt.Errorf("client.Do: %v", err)
    }
    defer response.Body.Close()

    fmt.Printf("request header = %#v\n", response.Request.Header)
    fmt.Printf("response header = %#v\n", response.Header)

    body, err := ioutil.ReadAll(response.Body)
    if err != nil {
        return fmt.Errorf("ioutil.ReadAll: %v", err)
    }

    fmt.Printf("%d: %s\n", response.StatusCode, string(body))

    return nil
}

But when I run this, I could only see the following response.

request header = http.Header{"Authorization":[]string{"Bearer <jwt token>"}, "Content-Type":[]string{"application/json"}, "X-Cloud-Trace-Context":[]string{"c855757f20d155da1140fad1508ae3e5/17413578722158830486;o=0"}}

response header = http.Header{"Alt-Svc":[]string{"clear"}, "Content-Length":[]string{"49"}, "Content-Type":[]string{"text/html; charset=UTF-8"}, "Date":[]string{"Wed, 06 May 2020 22:17:43 GMT"}, "X-Goog-Iap-Generated-Response":[]string{"true"}}

401: Invalid IAP credentials: JWT signature is invalid

As you can see here, the access was denied.

So I thought that the signature used to sign the JWT token in the header might be wrong.

But I made sure the following using jwt.io:

• The JWT token used in the header is signed by the private key of the caller's service account
• The JWT token used in the header can be verified by the public key of the caller's service account
• The JWT token was signed using RS256 algorithm

And I also looked into the token:

{
  "alg": "RS256",
  "typ": "JWT",
  "kid": "<the service account's private key ID>"
}

{
  "iss": "<email address of the service account>",
  "aud": "",
  "exp": 1588806087,
  "iat": 1588802487,
  "sub": "<email address of the service acocunt>"
}

Nothing quite odd.

So I'm not sure what's going on here. If I disable IAP, the endpoint return the right response.

Can anyone give me some hint about what I am doing wrong?

Austronesian

Have you removed the "aud" content in your token? – Emrich

Thanks for the comment! But this is a token created by google library, I couldn't modify it. Further, because the signature verification itself is failing, I doubt that's the root cause. – Austronesian

Have you make sure that the service account is at least an IAP web user? – Riedel

hello @irvifa, yes. I made sure of it. – Austronesian

Answer 1

I tried your code, and I found that it doesn't work with google.golang.org/api v0.23.0, but it does work with google.golang.org/api v0.24.0 (latest one at the time of writing).

It is indeed a bug, the release notes mention the following:

When provided, use the TokenSource from options for NewTransport. This fixes a bug in idtoken.NewClient where the wrong TokenSource was being used for authentication.

Interestingly, 0.23.0 is sending a token signed with the service account's private key with the following claims:

{
  "iss":"[email protected]",
  "aud":"",
  "exp":1589596554,
  "iat":1589592954,
  "sub":"[email protected]"
}

and 0.24.0 sends a token signed with google's private key. (internally the previous token is exchanged for a google-signed token)

{
    "aud":"xxxxxxx-xxxxxxxxxxxxxxxxxx.apps.googleusercontent.com",
    "azp":"[email protected]",
    "email":"[email protected]",
    "email_verified":true,
    "exp":1589596508,
    "iat":1589592908,
    "iss":"https://accounts.google.com",
    "sub":"11524xxxxxxxxxxxxxxxx"
}

Giselegisella

Thanks @Dirbaio! I also agree that it's a bug. I will add an updated code as a workaround. – Austronesian

Answer 2

JWT token exp property seems to have an expired date (7th May 1:01) and your post was from (7th May 3:53) have you tried regenerating it?

[EDIT: I know it should be a comment. Posted as answer because of low rep]

Chinaware

Thanks for the comment! but it was not relevant to the issue. – Austronesian

Answer 3

As @Dirbaio pointed out, I think this is an issue specific to v0.23.0. Because I cannot upgrade the dependency right now, I chose to create a new IAP client that does not use idtoken.NewClient. Instead, it just uses idtoken.NewTokenSource to create an OIDC token. Adding the token to the Authorization header is easy so I could work around the issue in the client created by idtoken.NewClient.

package main

import (
    "context"
    "crypto/tls"
    "fmt"
    "io"
    "net/http"

    "golang.org/x/oauth2"
    "google.golang.org/api/idtoken"
    "google.golang.org/api/option"
)

// IAPClient is the default HTTPS client with Morse-Code KMS integration.
type IAPClient struct {
    client      *http.Client
    tokenSource oauth2.TokenSource
}

// NewIAPClient returns an HTTP client with TLS transport, but not doing the CA checks.
func NewIAPClient(audience string, opts ...option.ClientOption) *IAPClient {
    tokenSource, err := idtoken.NewTokenSource(context.Background(), audience, opts...)
    if err != nil {
        panic(fmt.Errorf("cannot create a new token source: %s", err.Error()))
    }

    return &IAPClient{
        client: &http.Client{
            Transport: &http.Transport{
                TLSClientConfig: &tls.Config{
                    InsecureSkipVerify: true,
                },
            },
        },
        tokenSource: tokenSource,
    }
}

// Do sends the http request to server with a morse-code JWT Authorization: Bearer header.
func (c *IAPClient) Do(request *http.Request) (*http.Response, error) {
    err := c.addAuthorizationHeader(request)
    if err != nil {
        return nil, fmt.Errorf("couldn't override the request with the new auth header: %s", err.Error())
    }

    return c.client.Do(request)
}

// Get sends the http GET request to server with a morse-code JWT Authorization: Bearer header.
func (c *IAPClient) Get(url string) (*http.Response, error) {
    request, err := http.NewRequest(http.MethodGet, url, nil)

    if err != nil {
        return nil, err
    }

    return c.Do(request)
}

// Post sends the http POST request to server with a morse-code JWT Authorization: Bearer header.
func (c *IAPClient) Post(url, contentType string, body io.Reader) (*http.Response, error) {
    request, err := http.NewRequest(http.MethodPost, url, body)
    if err != nil {
        return nil, err
    }

    request.Header.Add("Content-Type", contentType)

    return c.Do(request)
}

func (c *IAPClient) addAuthorizationHeader(request *http.Request) error {
    tkn, err := c.tokenSource.Token()
    if err != nil {
        return fmt.Errorf("cannot create a token: %s", err.Error())
    }

    tkn.SetAuthHeader(request)

    return nil
}

Austronesian