Message Integrity Check with HTTP headers since Content-MD5 was deprecated?

Question

Making a REST web server mainly based on large files uploads / downloads, I want to be able to check the file integrity. I believed that the proper way to do it was using Content-MD5 HTTP header [0] as proved useful by aws experience [1].

However, much to my dismay, I recently learned that it was (to be ?) deprecated [2].

The deprecation discussion did not give any workaround hint, so I am asking you :

Should I still decide to use a Content-MD5 HTTP header ?

Should I use an ETag with the same meaning (base64 encoding of the md5sum) ?

Should I use an ?md5sum=XXX parameter ?

Is there a better solution altogether ?

Thanks for your insights.

Best Regards, B.

[0] https://webmasters.stackexchange.com/questions/2924/

[1] http://developer.amazonwebservices.com/connect/thread.jspa?threadID=22709

[2] http://trac.tools.ietf.org/wg/httpbis/trac/ticket/178

Nanosecond

Answer 1

Do not use Content-MD5: it has been deprecated because it leads to inconsistencies.

Use Digest with sha-256 or sha-512. We are updating RFC3230 to the latest HTTP specification (RFC7231) and added a lot of useful examples https://datatracker.ietf.org/doc/html/draft-ietf-httpbis-digest-headers-02

Digest: sha-256=4REjxQ4yrqUVicfSKYNO/cF9zNj5ANbzgDZt3/h3Qxo=

There Want-Digest allows requesting a specific Digest header.

Eg. The client requests a digest, supporting sha-256 and sha-512. The server replies with sha-256

Request:

GET /items/123 HTTP/1.1
Want-Digest: sha-256, sha-512

Response:

HTTP/1.1 200 OK
Content-Type: application/json
Digest: sha-256=X48E9qOokqqrvdts8nOJRJN3OWDUoyWxBf7kbu9DBPE=

{"hello": "world"}

---

2023 Note: The Digest field-name has been obsoleted per the (currently) latest draft (-13), there in 7.1. HTTP Field Name Registration has a nice overivew:

IANA is asked to update the "Hypertext Transfer Protocol (HTTP) Field Name Registry" registry ([HTTP]) according to the table below:

Field Name	Status	Reference
Content-Digest	permanent	Section 2 of this document
Repr-Digest	permanent	Section 3 of this document
Want-Content-Digest	permanent	Section 4 of this document
Want-Repr-Digest	permanent	Section 4 of this document
Digest	obsoleted	[RFC3230], Section 1.3 of this document
Want-Digest	obsoleted	[RFC3230], Section 1.3 of this document

Widget

Are there any web servers that do support Want-Digest as built-in option for static content? Or you always need to implement this header behaviour yourself every time? I am looking for a ready tool. – Monomolecular

Usually servers configured to return Digest always do it, but there are some plugins around, eg. github.com/search?q=want-digest&ref=opensearch Want-Digest is more useful when using Digest as a building block to implement some more logic in your API. – Widget

Answer 2

Add a custom header, called say X-YourService-Integrity. That makes it explicit that it's a system specific to your service, and allows you to use integrity check mechanisms other than MD5 in the future (for example, SHA1). It also avoids you having to "overload" existing mechanisms that are similar but not quite what you want.

Kyle

Thank you for your answer. However, a custom http header can be dropped during transfer and I don't see an advantage over ETag that could be used for any integrity check as the validator implementation is left open by the standard w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3 – Nanosecond

Why would a header be dropped during transfer? – Kyle

Thank you for following up on this matter. I'm currently searching the text (I'm sure I've been reading that implementations SHOULD only keep non standard headers, but I cannot find an authoritative reference. For the record, the "X-" naming convention is discouraged tools.ietf.org/html/draft-saintandre-xdash-03 . – Nanosecond

I'm not sure how many other desirable options you have left, then. ETag is meant to be server-assigned, and this is a client-assigned value. The URI is not the place for it either as the integrity hash is related to the representation, not the server resource. And while X- headers are discouraged, there are ways outlined in that link you posted to add custom headers. This is meta data, and meta data belongs in a header. How you structure that header is up to you and your application's needs. – Kyle

Answer 3

https://www.ietf.org/rfc/rfc3230.txt

4.3.2 Digest

The Digest message header field provides a message digest of the instance described by the message.

  Digest = "Digest" ":" #(instance-digest)

The instance described by a message might be fully contained in the message-body, partially-contained in the message-body, or not at all contained in the message-body. The instance is specified by the Request-URI and any cache-validator contained in the message.

A Digest header field MAY contain multiple instance-digest values. This could be useful for responses expected to reside in caches shared by users with different browsers, for example.

A recipient MAY ignore any or all of the instance-digests in a Digest header field.

A sender MAY send an instance-digest using a digest-algorithm without knowing whether the recipient supports the digest-algorithm, or even knowing that the recipient will ignore it.

Examples:

  Digest: md5=HUXZLQLMuI/KZ5KDcJPcOA==
  Digest: SHA=thvDyvhfIqlvFe+A9MYgxAfm1q5=,unixsum=30637

Viradis

Are there any web servers that do support Want-Digest as built-in option for static content? Or you always need to implement this header behaviour yourself every time? I am looking for a ready tool. – Monomolecular