How can I read HTTP/2 push frames from a net/http request

Question

I am trying to write a Go client to test our http/2 infrastructure. I would like to make an http request to https://mydomain.tld/somePage and expect to receive an html response, along with several pushed resources. I would like to ensure those pushes are successful, and fail if they are not.

It is not clear to me if any part of the standard library exposes this functionality I need.

I can look at the response and check the protocol version to detect http2.

I can see Link headers in responses from sites like https://http2-push.appspot.com/ that send pushes, but I'm not quite clear on the relationship between Link headers and actual Push Promise frames. You can get link headers over http 1.1, so I'm not sure that alone ensures a push will happen.

The http2 package has a lower level Framer interface that I may be able to leverage to verify the raw frames, but honestly, I have no idea how to set one up and issue the initial request to it.

Is there any example of how a go client can verify the proper configuration of http2 pushed resources?

Drobman

I know they recently had a blog post (blog.golang.org/h2push) about server push, but I haven't seen anything about client yet. As I understand it H2 push works by pushing a response (by manufacturing a request and including that request and response together) to a client that stores the response directly in its cache. The result is that when the client makes the request rather than going over the network it refers to its local cache. This is expected to be built in to browsers that support h2, whether or not the language's http client supports it natively would require some digging. – Survival

Right. I don't actually need client support for receiving pushes, I just need to really validate that "If I was a browser, the server is configured to push these things properly" – Drobman

Looking at the test suite it appears as though private methods (e.g. decodeHeader) are required to actually get at that data, and there isn't much discussion about a proper client API. – Unhandled

That's a bit disheartening @nothingmuch. Maybe the answer is "sometime in the future"? – Drobman

Perhaps your case is a good motivating example to encourage progress on that front? I would encourage you participate in the discussion, since your test case seems like a very elementary thing to try to verify when starting to use that kind of feature, I think it's worth bringing up. Another idea is to use something like node-http2, it'd be pretty easy to call out to that from a Go test I suppose though I'm not sure if it's worth the headache of having another dev stack even if opportunistically. – Unhandled

Answer 1

Using the Framer in golang.org/x/net/http2 isn't hard, if we can get a copy of the bytes that are read naturally by the http.Client. We can do that by implementing our own net.Conn.

I made some progress with the program below, however I did not see the expected PUSH_PROMISE frames. After some digging around I found that the Go client explicitly disables Push. Servers are not allowed to send those frames in this case. I don't see an obvious way to change that setting (short of hacking the stdlib).

Thought I still share my code. Perhaps I missed something simple to make it work after all.

package main

import (
    "bytes"
    "crypto/tls"
    "io"
    "io/ioutil"
    "log"
    "net"
    "net/http"
    "os"

    "golang.org/x/net/http2"
)

func main() {
    buf := &bytes.Buffer{}
    transport := &http2.Transport{DialTLS: dialT(buf)}
    client := &http.Client{Transport: transport}

    res, err := client.Get("https://http2-push.appspot.com/")
    if err != nil {
            log.Fatal(err)
    }

    res.Body.Close()
    res.Write(os.Stdout)

    framer := http2.NewFramer(ioutil.Discard, buf)
    for {
            f, err := framer.ReadFrame()
            if err == io.EOF || err == io.ErrUnexpectedEOF {
                    break
            }
            switch err.(type) {
            case nil:
                    log.Println(f)
            case http2.ConnectionError:
                    // Ignore. There will be many errors of type "PROTOCOL_ERROR, DATA
                    // frame with stream ID 0". Presumably we are abusing the framer.
            default:
                    log.Println(err, framer.ErrorDetail())
            }
    }
}

// dialT returns a connection that writes everything that is read to w.
func dialT(w io.Writer) func(network, addr string, cfg *tls.Config) (net.Conn, error) {
    return func(network, addr string, cfg *tls.Config) (net.Conn, error) {
            conn, err := tls.Dial(network, addr, cfg)
            return &tConn{conn, w}, err
    }
}

type tConn struct {
    net.Conn
    T io.Writer // receives everything that is read from Conn
}

func (w *tConn) Read(b []byte) (n int, err error) {
    n, err = w.Conn.Read(b)
    w.T.Write(b)
    return
}

Twenty

A nice start. I'd hate to have to fork std lib to do this though. Perhaps I'll nudge that issue a bit. – Drobman

Answer 2

A patch was submitted for review.

"http2: support consuming PUSH_PROMISE streams in the client"

(The github issue has milestone "Unplanned", which hopefully won't give it significantly less priority in the review queue.)

Lummox

Go1.10 is frozen, this might make it for Go1.11 – Unglue

Go1.17 is out. Still waiting. Many people have looked for this over the years from what I can tell. Commits were almost done but not enough reviewers on the Golang side at the time. I think one of the problems with the go guarantees is once they support an API in the standard library, they guarantee to support it indefinitely - and just how a client accesses the push promise through the client api is rife with subjective opinions. An experimental package that is allowed to change would have made this more likely to be supported. – Maeda