Heroku H10 error but web dynos are not crashing

Question

A few hours ago we starting seeing a bunch of H10 errors in our logs and on our Heroku dashboard. Heroku says that H10 errors are due to web dyno crashes. Read more here. The strange part is that our 2 web dynos are running completely fine and not crashing at all.

Here's an example of what I'm seeing in the logs:

2021-08-12T20:50:35.934439+00:00 heroku[router]: at=error code=H10 desc="App crashed" method=POST path="/v1/graphql" host=api.rize.io request_id=abcd-ace1-44e8-99f5-03d38d8eaa5a fwd="1.1.1.1" dyno= connect= service= status=503 bytes= protocol=https

When I search for that request ID, abcd-ace1-44e8-99f5-03d38d8eaa5a in this example, in the logs (as Heroku recommends), nothing shows up except the router error listed above.

I've restarted all the dynos and the issue persists.

Anyone have any idea what could be going on and how we can fix this?

Gompers

I am seeing the same behavior. Intermittent H10s but app is not crashed. I was losing my mind trying to figure this one out, but I'm glad to see I'm not alone. Hoping this is an outage. – Rhebarhee

Same problem here going on 12+ hours. No response from support. heroku ps shows app up and running. We've tried maintenance mode, switching dyno types, scaling down to 0 and back, redeploying code, changing environment variables. Nothing helps. – Codding

Okay cool. Glad to know I'm not the only one seeing this issue. I'm going to file another ticket with Heroku to hopefully put more pressure on them to figure out what's going on. – Gompers

I'm also having this issue on several of my sites (typically serving ~100 rps, now sitting solidly at 0 rps due to 100% error rate). No word back from Heroku, also seemingly tried everything. Not sure what to do while I wait. – Warfare

Answer 1

I don't know if this is an answer for other people but we discovered that our DNS settings were not updated to match the changes documented here: https://devcenter.heroku.com/changelog-items/2151

We also needed to re-establish managed SSL certificates for our custom domains.

Our setup has some complications related to proxies we have in front of Heroku though and the fact that of the three custom domains we have configured on the app, only two of them had been migrated, which complicated the debugging of this situation.

Codding

Answer 2

This was due to a Heroku incident. Here is the incident status: https://status.heroku.com/incidents/2311

I asked Heroku support and he rolled back the infrastructure upgrade (https://devcenter.heroku.com/changelog-items/2151) for our site. After that, we don't have H10 errors.

Crosslink

Answer 3

Dunno if it helps, but I've cloned my app to a new one (letting all env vars on the old ones) and directed everything to the new Heroku app. It seems to be working just good enough while waiting for the problem to be solved by the Heroku team. No H10 errors but some H31, nothing is broken anymore though.

Welldefined