Creating a ManagedCertificate results in "Status: FailedNotVisible"

Question

Using Kubernetes 1.12.6-gke.7 or higher it is possible to create a ManagedCertificate which is then referenced from an Ingress Resource exposing a Service to the Internet.

Running kubectl describe managedcertificate certificate-name first indicates the certificate is in a Provisioning state but eventually goes to FailedNotVisible.

Despite using a Static IP and DNS that resolves fine to the http version of said service all ManagedCertificate's end up in a "Status: FailedNotVisible" state.

Outline of what I am doing:

1. Generating a reserved (static) external IP Address

2. Configuring DNS A record in CloudDNS to subdomain.domain.com to generated IP address from step 1.

3. Creating a ManagedCertificate named "subdomain-domain-certificate" with kubectl apply -f with spec:domains containing a single domain corresponding to subdomain.domain.com DNS record in step 2.
4. Creating a simple deployment and service exposing it
5. Creating Ingress resource referring to default backend of service in step 4 as well as annotations for static ip created in step 1 and managed certificate generated in step 3.
6. Confirm that Ingress is created and is assigned static IP
7. Visiting http://subdomain.domain.com serves the output from pod created in deployment in step 4

After a little while

kubectl describe managedcertificate subdomain-domain-certificate

results in "Status: FailedNotVisible".

Name:         subdomain-domain-certificate
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  networking.gke.io/v1beta1
Kind:         ManagedCertificate
Metadata:
  Creation Timestamp:  2019-04-15T17:35:22Z
  Generation:          1
  Resource Version:    52637
  Self Link:           /apis/networking.gke.io/v1beta1/namespaces/default/managedcertificates/subdomain-domain-certificate
  UID:                 d8e5a0a4-5fa4-11e9-984e-42010a84001c
Spec:
  Domains:
    subdomain.domain.com
Status:
  Certificate Name:    mcrt-ac63730e-c271-4826-9154-c198d654f9f8
  Certificate Status:  Provisioning
  Domain Status:
    Domain:  subdomain.domain.com
    Status:  FailedNotVisible
Events:
  Type    Reason  Age   From                            Message
  ----    ------  ----  ----                            -------
  Normal  Create  56m   managed-certificate-controller  Create SslCertificate mcrt-ac63730e-c271-4826-9154-c198d654f9f8

From what I understand if the Load Balancer is configured correctly (done under the hood in the ManagedCertificate resource) and the DNS (which resolves fine to the non https endpoint) checks out the certificate should go in to a Status: Active state?

Sputter

If you're the impatient type, you really may need to wait the full hour before the certificate becomes active – Blabber

Answer 1

The issue underlying my problem ended up being a DNSSEC misconfiguration. After running the DNS through https://dnssec-analyzer.verisignlabs.com/ I was able to identify and fix the issue.

Sputter

Had the same problem, thanks for the tool. Need to shell out money to Godaddy to enable DNSSEC. – Stoush

Don’t give Godaddy any more money, simply move your DNS nameservers in Godaddy to GCP Cloud DNS and enable it from there. I’m not sure how much it costs over time but no upfront payments are necessary. – Sputter

Just to be clear, was the issue that it was necessary for DNSSEC to be turned on? – Trouvaille

What was the issue exactly? – Rataplan

PSA: Enabling DNSSEC is not a requirement for using ManagedCertificates. However, having all of the domain names listed in the ManagedCertificate spec resolve to the global static IP address of the Load Balancer using them is. It is possible that if you have DNSSEC enabled and it is misconfigured, that could cause problems. – Vaporization

@Sputter What was the resolution. I am facing the same issue. I have registered my domain in Google and using Cloud DNS with DNSSEC on. – Pharmaceutics

Answer 2

DNSSEC was indeed not enabled for my domain but after configuring that, the ManagedCertificate configuration was still not going through and I had no clue what was going on. Deleting and re-applying the ManagedCertificate and Ingress manifests did not do the trick. But issuing the command gcloud beta compute ssl-certificates list showed several unused managed certificates hanging around and deleting them with cloud compute ssl-certificates delete NAME ..., and then restarting the configuration process did the trick in my case.

Ornithology

Answer 3

You need to make sure the domain name resolves to the IP address of your GKE Ingress, following the directions for "creating an Ingress with a managed certificate" exactly.

For more details, see the Google Cloud Load Balancing documentation. From https://cloud.google.com/load-balancing/docs/ssl-certificates#domain-status:

"The status FAILED_NOT_VISIBLE indicates that certificate provisioning failed for a domain because of a problem with DNS or the load balancing configuration. Make sure that DNS is configured so that the certificate's domain resolves to the IP address of the load balancer."

Pyrognostics

Yes Evan those are the obvious (well documented) deviations that could cause the "FAILED_NOT_VISIBLE" issue to arise. In my case the cause was with the DNSSEC delegation from GoDaddy being incorrect. – Sputter

Answer 4

I just ran into this problem when I was setting up a new service and my allowance of 8 external IPs was used up.

Following the trouble shooting guide, I checked whether there was a forwarding rule for port 443 to my ingress. There wasn't. When I tried to set it up manually, I got an error telling me I used up my 8 magic addresses. I deleted forwarding rules I didn't need et voila!

Now, why the forwarding rule for port 80 was successfully set up for the same ingress is beyond me.

Antibes

Answer 5

I ran across this same error and found that I had created the managedCertificate in the wrong Kubernetes namespace. Once the managedCertificate was placed in the correct namespace everything worked.

Imparipinnate

in general how long does it take to reflect ? i had a case that took around 2 hours – Indreetloire

Answer 6

After reading the trouble shooting guide, I still wasn't able to resolve my issue. When I checked the GCP ingress events, it showed that the ingress could not locate the SSL policy. Check if you missed something when creating the ingress.

and this is another reference useful to verify your k8s manifests to set up the managed certificate and ingress. Hope it helps someone.

Burnie