Google Managed SSL Certificate Stuck on FAILED_NOT_VISIBLE

Question

I'm trying to configure an HTTPS/Layer 7 Load Balancer with GKE. I'm following SSL certificates overview and GKE Ingress for HTTP(S) Load Balancing.

My config. has worked for some time. I wanted to test Google's managed service.

This is how I've set it up so far:

k8s/staging/staging-ssl.yml:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: my-staging-lb-ingress
  annotations:
    kubernetes.io/ingress.global-static-ip-name: "my-staging-global"
    ingress.gcp.kubernetes.io/pre-shared-cert: "staging-google-managed-ssl"
    kubernetes.io/ingress.allow-http: "false"
spec:
  rules:
  - host: staging.my-app.no
    http:
      paths:
      - path: /*
        backend:
          serviceName: my-svc
          servicePort: 3001

gcloud compute addresses list

#=>

NAME                   REGION  ADDRESS          STATUS
my-staging-global              35.244.160.NNN  RESERVED

host staging.my-app.no 

#=>

35.244.160.NNN

but it is stuck on FAILED_NOT_VISIBLE:

gcloud beta compute ssl-certificates describe staging-google-managed-ssl

#=>

creationTimestamp: '2018-12-20T04:59:39.450-08:00'
id: 'NNNN'
kind: compute#sslCertificate
managed:
  domainStatus:
    staging.my-app.no: FAILED_NOT_VISIBLE
  domains:
  - staging.my-app.no
  status: PROVISIONING
name: staging-google-managed-ssl
selfLink: https://www.googleapis.com/compute/beta/projects/my-project/global/sslCertificates/staging-google-managed-ssl
type: MANAGED

Any idea on how I can fix or debug this further?

---

I found a section in the doc I linked to at the beginning of the post Associating SSL certificate resources with a target proxy:

Use the following gcloud command to associate SSL certificate resources with a target proxy, whether the SSL certificates are self-managed or Google-managed.

gcloud compute target-https-proxies create [NAME] \
--url-map=[URL_MAP] \
--ssl-certificates=[SSL_CERTIFICATE1][,[SSL_CERTIFICATE2], [SSL_CERTIFICATE3],...]

Is that necessary when I have this line in k8s/staging/staging-ssl.yml?

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    . . .
    ingress.gcp.kubernetes.io/pre-shared-cert: "staging-google-managed-ssl"
    . . .

Capitulate

A work-around I used, was to "abuse" the lack of proper ipv6 support. I updated the SSL certificate manually via the dashboard to the target proxy. This isn't removed by gcloud / k8s and it allows for it to become ACTIVE, at the point where you can swap the k8s managed certificates for the managed GCP ones. – Unappealable

I'm having this same exact issue with a google-provisioned SSL, it's so frustrating, have you made any progress? – Noontime

Answer 1

It turns out that I had mistakenly done some changes to the production environment and others to staging. Everything worked as expected when I figured that out and followed the guide. :-)

Capitulate

Was it necessary for you to do the gcloud compute target-https-proxies create step to get it working? – Pipsissewa

It's not necessary. The target proxy gets created automatically when you create a GKE Ingress. – Pipsissewa

@JohnMichelau when not doing GKE Ingress, yes the target proxies create was the step that worked for me. Otherwise things just hung. – Stellastellar

Answer 2

I have faced this issue recently. You need to check whether your A Record correctly points to the Ingress static IP.

If you are using a service like Cloudflare, then disable the Cloudflare proxy setting so that ping to the domain will give the actual IP of Ingress. THis will create the Google Managed SSL certificate correctly with 10 to 15 minutes.

Once the certificate is up, you can again enable Cloudflare proxy setting.

Spindrift

This is the exact problem I had. Thanks for the answer. However, wondering how the current problem is solved - If i disable the cloud flare proxy, it exposes the ip address of the load balancer to public. This can be a potential risk ip attacks. I'm currently planning to use rate limit with CF. How this can be solved? – Alit

Thanks for your answer, I also had this problem :) – Gorget

Even I am also trying to find out solution for this. Actually Google Managed SSL should automatically detect the IP even if proxy services lioke CloudFlare are used. However, I am still not able to find any solution for this. – Spindrift

Thanks for this - I had exactly this issue :) – Paella

If you disable the proxy you open yourself up to botnet attacks. You can instead go to the SSL option in cloudflare and set it too "Full (strict)" – Tit

I think the reason GKE Ingress Controller "Managed Certificates" (GKE Ingress specific custom resource) doesn't like CloudFlare Proxy, is because it provisions certs with HTTP ACME challenge under the hood. And then CloudFlare Proxy(a L7 LB) is in front of the GCP LB (an L7 Proxy), which seems to mess up the HTTP challenge, because it's behind CF Proxy. Note GW API's gcloud cli provisioned/managed HTTPS certs, (that can be attached to gateway yaml objects), uses DNS based ACME challenge, so another good solution is to switch from GKE's Ingress Controller to GKE's GW API Controller. – Maymaya

Answer 3

I'm leaving this for anyone who might end up in the same situation as me. I needed to migrate from a self-managed certificate to a google-managed one.

I did create the google-managed certificate following the guide and was expecting to see it being activated before applying the certificate to my Kubernetes ingress (to avoid the possibility of a downtime)

Turns out, as stated by the docs,

the target proxy must reference the Google-managed certificate resource

So applying the configuration with kubectl apply -f ingress-conf.yaml made the load balancer use the newly created certificate, which became active shortly after (15 min or so)

Wowser

I don't know how nor why, but the 2 forwarding rules (http and https) created by k8s are using different IPs. I manually deleted one and recreated it forcing the right IP. I'll update this thread if this hack works. – Standee

Confirmed: manually adding an extra forwarding address with the proper {"kubernetes.io/ingress-name": "…"} description solved my problem. I still don't know why k8s insists in pointing HTTPS to a different IP. Hope that helps someone out there. – Standee

15min outages to use google managed SSL certificate is quite lame. Is there no way around this? I'm at 20mins and counting waiting on the SSL certificate to be created. – Instant

Answer 4

What worked for me after checking the answers here (I worked with a load balancer but IMO this is correct for all cases):

1. If some time passed this certificate will not work for you (It may be permamnently gone and it will take time to show that) - I created a new one and replaced it in the Load Balancer (just edit it)
2. Make sure that the certificate is being used a few minutes after creating it
3. Make sure that the DNS points to your service. And that your configuration is working when using http!! - This is the best and safest way (also if you just moved a domain - make sure that when you check it you reach to the correct IP)
4. After creating a new cert or if the problem was fixed - your domain will turn green but you still need to wait (can take an hour or more)

Orientalize

Answer 5

In my case, at work. We are leveraging the managed certificate a lot in order to provide dynamic environment for Developers & QA. As a result, we are provisioning & removing managed certificate quite a lot. This mean that we are also updating the Ingress resource as we are generating & removing managed certificate.

What we have founded out is that even if you delete the reference of the managed certificate from this annotation:

networking.gke.io/managed-certificates: <list>

It seems that randomly the Ingress does not remove the associated ssl-certificates from the LoadBalancer.

ingress.gcp.kubernetes.io/pre-shared-cert: <list>

As a result, when the managed certificate is deleted. The ingress will be "stuck" in a way, that no new managed certificate could be provision. Hence, new managed-ceritifcate will after some times transition from PROVISIONING state to FAILED_NOT_VISIBLE state

The only solution that we founded out so far, is that if a new certificate does not get provision after 30min. We will check if the annotation ingress.gcp.kubernetes.io/pre-shared-cert contains ssl-certificate that does not exist anymore.

You can check existing ssl-certificate with the command below

gcloud compute ssl-certificates list

If it happens that one ssl-certificate that does not exist anymore is still hanging around in the annotation. We'll then remove the unnecessary ssl-certificate from the ingress.gcp.kubernetes.io/pre-shared-cert annotation manually.

After applying the updated configuration, in about 5 minutes, the new managed certificate which was in FAILED_NOT_VISIBLE state should be provision and in ACTIVE state.

Retharethink

Turned out that was the reason for my issues as well. Thanks a lot! – Khan

Answer 6

As per the following documentation which you provided, this should help you out:

The status FAILED_NOT_VISIBLE indicates that certificate provisioning failed for a domain because of a problem with DNS or the load balancing configuration. Make sure that DNS is configured so that the certificate's domain resolves to the IP address of the load balancer.

Antigorite

Yes, I saw that, but the domain resolves correct. Which leads me to the load balancer. I can't see anything wrong with that either. :( – Capitulate

I'm also having the sample issue. Domain A record TTL is 5mins. Domain is resolved to up address indicated in Ingress status ip. "gcloud compute addresses list" shows IP address as external and IN_USE. "gcloud beta compute ssl-certificates list" says still PROVISIONING and domain is FAILED_NOT_VISIBLE. Do you really need to create a target proxy for this to work, cause cloud.google.com/kubernetes-engine/docs/how-to/managed-certs does not mention any target proxies? – Ethiop

Does this mean that a certificate can't be created without the DNS pointing to the IP address? I'm trying to migrate live services to Google managed certificates and I want to switch the DNS after the certificate has been created so there is no downtime. – Unplaced

@Unplaced it appears that way, I had to provision the cert before hand, wait til the FAILED_NOT_VISIBLE error flagged up, then switched the DNS over to the new LB and it worked straight away – Diverge

Answer 7

As already pointed by Mitzi https://mcmap.net/q/448468/-google-managed-ssl-certificate-stuck-on-failed_not_visible

This is what worked for me

1. Create cert with subdomains/domains
2. Must Add it load balancer ( I was waiting for it to become active but only when you add it becomes active !! )
3. Add static IP as A record for domains/subdomain

It worked in 5min

Ferriter

Answer 8

What is the TTL (time to live) of the A Resource Record for staging.my-app.no? Use, e.g.,

dig +nocmd +noall +answer staging.my-app.no

to figure it out.

In my case, increasing the TTL from 60 seconds to 7200 let the domainStatus finally arrive in ACTIVE.

Disassembly

Hi, I changed it as you suggested yesterday. Still no luck! :-( – Capitulate

Answer 9

In addition to the other answers, when migrating from self-managed to google-managed certs I had to:

• Enable http to my ingress service with kubernetes.io/ingress.allow-http: true
• Leave the existing SSL cert running in the original ingress service until the new managed cert was Active

I also had an expired original SSL cert, though I'm not sure this mattered.

Lucilucia

Answer 10

In my case I needed alter the healthcheck and point it to the proper endpoint ( /healthz on nginx-ingress) and after the healtcheck returned true I had to make sure the managed certificate was created in the same namespace as the gce-ingress. After these two things were done it finally went through, otherwise I got the same error. "FAILED_NOT_VISIBLE"

Creech

Answer 11

I met the same issue. I fixed it by re-looking at the documentation.

https://cloud.google.com/load-balancing/docs/ssl-certificates/troubleshooting?_ga=2.107191426.-1891616718.1598062234#domain-status

FAILED_NOT_VISIBLE  
Certificate provisioning failed for the domain. Either of the following might be the issue:
The domain's DNS record doesn't resolve to the IP address of the Google Cloud load balancer. To resolve this issue, update the DNS records to point to your load balancer's IP address.
The SSL certificate isn't attached to the load balancer's target proxy. To resolve this issue, update your load balancer configuration.
Google Cloud continues to try to provision the certificate while the managed status is PROVISIONING.

Because my loadbalancer is behind cloudflare. By default cloudflare has cdn proxy enabled, and i need to first disable it after the DNS verified by Google, the cert state changed to active.

Seniority

Answer 12

I had this problem for days. Even though the FQDN in Google Cloud public DNS zone correctly resolved to the IP of the HTTPS Load Balancer, certificate created failed with FAILED_NOT_VISIBLE. I eventually resolved the problem as my domain was set up in Google Domains with DNSSEC but had an incorrect DNSSEC record when pointing to the Google Cloud Public DNS zone. DNSSEC configuration can be verified using https://dnsviz.net/

Varini

Answer 13

I had the same problem. But my problem was in the deployment. I ran

kubectl describe ingress [INGRESS-NAME] -n [NAMESPACE]

The result shows an error in the resources.timeoutsec for the deployment. Allowed values must be less than 300 sec. My original value was above that. I reduced readinessProbe.timeoutSeconds to a lower number. After 30 mins the SSL cert was generated and the subdomain was verified.

Impulse

Answer 14

Having read tens of pages online, I finally solved this issue by creating a new certificate. Even though I waited 2+ days for the initial certificate to become ACTIVE, it got stuck at FAILED_NOT_VISIBLE. The new certificate became ACTIVE within 10 minutes! I then started using the new certificate in the load balancer and everything was fine.

Converse