WSO2 API Manager, invalid. unable to find valid certification path to requested target

Question

I have launched WSO2 API Manager locally.

I'm trying to add API endpoint with https connecttion. It shows me that kind of error.

It shows me Invalid. unable to find valid certification path to requested target error message.

Then I have downloaded certificate, which is cer file from website and in carbon page I have imported certificate to wso2carbon keystore:

After that I restart my WSO2 API Manager and try to test this endpoint and it still throws me that error.

What did I miss?

Console show that kind of error:

[2017-06-12 21:12:18,362] ERROR - APIProviderHostObject Error occurred while connecting to backend : https://example.com, reason : sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
    at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1949)
    at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:302)
    at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296)
    at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1514)
    at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
    at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1026)
    at sun.security.ssl.Handshaker.process_record(Handshaker.java:961)
    at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1062)
    at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403)
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387)
    at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:533)
    at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:401)
    at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:178)
    at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:304)
    at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:610)
    at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:445)
    at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
    at org.wso2.carbon.apimgt.hostobjects.APIProviderHostObject.sendHttpHEADRequest(APIProviderHostObject.java:4781)
    at org.wso2.carbon.apimgt.hostobjects.APIProviderHostObject.jsFunction_isURLValid(APIProviderHostObject.java:4173)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:126)
    at org.mozilla.javascript.FunctionObject.call(FunctionObject.java:386)
    at org.mozilla.javascript.optimizer.OptRuntime.callN(OptRuntime.java:52)
    at org.jaggeryjs.rhino.publisher.modules.api.c3._c_anonymous_10(/publisher/modules/api/add.jag:225)
    at org.jaggeryjs.rhino.publisher.modules.api.c3.call(/publisher/modules/api/add.jag)
    at org.mozilla.javascript.ScriptRuntime.applyOrCall(ScriptRuntime.java:2430)
    at org.mozilla.javascript.BaseFunction.execIdCall(BaseFunction.java:269)
    at org.mozilla.javascript.IdFunctionObject.call(IdFunctionObject.java:97)
    at org.mozilla.javascript.optimizer.OptRuntime.call2(OptRuntime.java:42)
    at org.jaggeryjs.rhino.publisher.modules.api.c0._c_anonymous_5(/publisher/modules/api/module.jag:18)
    at org.jaggeryjs.rhino.publisher.modules.api.c0.call(/publisher/modules/api/module.jag)
    at org.mozilla.javascript.optimizer.OptRuntime.callN(OptRuntime.java:52)
    at org.jaggeryjs.rhino.publisher.site.blocks.item_add.ajax.c0._c_anonymous_2(/publisher/site/blocks/item-add/ajax/add.jag:233)
    at org.jaggeryjs.rhino.publisher.site.blocks.item_add.ajax.c0.call(/publisher/site/blocks/item-add/ajax/add.jag)
    at org.mozilla.javascript.optimizer.OptRuntime.call0(OptRuntime.java:23)
    at org.jaggeryjs.rhino.publisher.site.blocks.item_add.ajax.c0._c_script_0(/publisher/site/blocks/item-add/ajax/add.jag:9)
    at org.jaggeryjs.rhino.publisher.site.blocks.item_add.ajax.c0.call(/publisher/site/blocks/item-add/ajax/add.jag)
    at org.mozilla.javascript.ContextFactory.doTopCall(ContextFactory.java:394)
    at org.mozilla.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3091)
    at org.jaggeryjs.rhino.publisher.site.blocks.item_add.ajax.c0.call(/publisher/site/blocks/item-add/ajax/add.jag)
    at org.jaggeryjs.rhino.publisher.site.blocks.item_add.ajax.c0.exec(/publisher/site/blocks/item-add/ajax/add.jag)
    at org.jaggeryjs.scriptengine.engine.RhinoEngine.execScript(RhinoEngine.java:567)
    at org.jaggeryjs.scriptengine.engine.RhinoEngine.exec(RhinoEngine.java:273)
    at org.jaggeryjs.jaggery.core.manager.WebAppManager.exec(WebAppManager.java:588)
    at org.jaggeryjs.jaggery.core.manager.WebAppManager.execute(WebAppManager.java:508)
    at org.jaggeryjs.jaggery.core.JaggeryServlet.doPost(JaggeryServlet.java:29)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:650)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:747)
    at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:485)
    at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:377)
    at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:337)
    at org.jaggeryjs.jaggery.core.JaggeryFilter.doFilter(JaggeryFilter.java:21)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at org.wso2.carbon.ui.filters.cache.ContentTypeBasedCachePreventionFilter.doFilter(ContentTypeBasedCachePreventionFilter.java:53)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at org.apache.catalina.filters.HttpHeaderSecurityFilter.doFilter(HttpHeaderSecurityFilter.java:120)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:218)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:505)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
    at org.wso2.carbon.tomcat.ext.valves.CompositeValve.continueInvocation(CompositeValve.java:99)
    at org.wso2.carbon.tomcat.ext.valves.CarbonTomcatValve$1.invoke(CarbonTomcatValve.java:47)
    at org.wso2.carbon.webapp.mgt.TenantLazyLoaderValve.invoke(TenantLazyLoaderValve.java:57)
    at org.wso2.carbon.event.receiver.core.internal.tenantmgt.TenantLazyLoaderValve.invoke(TenantLazyLoaderValve.java:48)
    at org.wso2.carbon.tomcat.ext.valves.TomcatValveContainer.invokeValves(TomcatValveContainer.java:47)
    at org.wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(CompositeValve.java:62)
    at org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.java:159)
    at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:958)
    at org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:57)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:452)
    at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1087)
    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1756)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1715)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:745)
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:387)
    at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:292)
    at sun.security.validator.Validator.validate(Validator.java:260)
    at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
    at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229)
    at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124)
    at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1496)
    ... 90 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
    at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
    at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
    at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:382)

Aquamanile

Just ideas - maybe you could try to add the whole certificate chain to the truststore (all certificates from the chain). The backend certificate should be in the client-truststore.jks as well you could check that's no an SNI issue – Danley

@Danley I have added whole chain that consists of two certificates, still get this error. – Aquamanile

Is there anything reasonable in the stacktrace? (wso2carbon.log) if not, you can switch on the handshake debug ( -Djavax.net.debug=ssl:handshake) , but you will have to dig deep – Danley

I have added stacktrace – Aquamanile

check raw TLS to the backend (openssl s_client -v -connect host:port) to see what certificate you REALLY get (I have bad feeling that the http client used doesn't support SNI) – Danley

Answer 1

I'm going to answer to my own question. Here is how I have solved this problem.

Here is the steps that I have used to solve this problem:

1. In <APIM_HOME>/repository/conf/axis2/axis2.xml where HostnameVerifier parameter is used I have written this:

   AllowAll

2. Then in <APIM_HOME>\repository\resources\security there are two keystores located wso2carbon.jks and client-truststore.jks. We can import certificates for wso2carbon.jks via carbon web page as I have shown in my question. However, for client-truststore.jks I should add this certificate manually via keytool

   keytool -importcert -file my_domain_sert.ser -keystore client-truststore.jks -alias "example.com"
   

==========================Important Warning!!!===========================

When you are going to download endpoints' certificate, by using your browser, you should check If your antivirus' firewall overwriting your certificate. If it overwrites, you mat have an error.

Here my ESET NOD32 Antivirus is changing original certificate.

Aquamanile

Excellent. My problem was "Antivirus"! After disable kaspersky, and download new ssl certificate, ssl handshaking error resolved. – Hoffmann