Firestore / gRPC behind a corporate firewall / proxy

J

2

11

Our company has built an electron application using Firestore and now we are trying to deploy the application behind a corporate proxy and firewall (customer environment). After setting the proxy authentication settings using electrons app.on('login') all network requests in the application are successful, except for the firestore connection.

We receive the following error:

[2018-09-21T09:09:13.556Z]  @firebase/firestore: Firestore (5.5.0) [Connection]: GRPC stream error. Code: 14 Message: 14 UNAVAILABLE: Connect Failed
[2018-09-21T09:09:13.557Z]  @firebase/firestore: Firestore (5.5.0) [PersistentStream]: close with error: FirebaseError: [code=unavailable]: 14 UNAVAILABLE: Connect Failed
[2018-09-21T09:09:13.557Z]  @firebase/firestore: Firestore (5.5.0): Could not reach Cloud Firestore backend. Connection failed 1 times. Most recent error: FirebaseError: [code=unavailable]: 14 UNAVAILABLE: Connect Failed
This typically indicates that your device does not have a healthy Internet connection at the moment. The client will operate in offline mode until it is able to successfully connect to the backend.

We have also tried to debug gRPC which is used by Firestore with the following log output:

I0921 11:39:18.014000000  8920 src/core/ext/filters/client_channel/lb_policy/subchannel_list.h:292] [pick_first 138D9818] subchannel list 08E19208 index 7 of 11 (subchannel 138B9138): starting watch: requesting connectivity change notification (from IDLE)
I0921 11:39:18.016000000  8920 connectivity_state.cc:116] CONWATCH: 138B91A8 subchannel: from IDLE [cur=IDLE] notify=08F0B0C4
I0921 11:39:18.016000000  8920 connectivity_state.cc:164] SET: 138B91A8 subchannel: IDLE --> CONNECTING [state_change] error=00000000 "No Error"
I0921 11:39:18.018000000  8920 connectivity_state.cc:190] NOTIFY: 138B91A8 subchannel: 08F0B0C4
I0921 11:39:18.019000000  8920 tcp_client_custom.cc:139] CLIENT_CONNECT: 134C5B98 ipv4:172.217.16.202:443: asynchronously connecting
I0921 11:39:18.020000000  8920 src/core/ext/filters/client_channel/lb_policy/subchannel_list.h:404] [pick_first 138D9818] subchannel list 08E19208 index 7 of 11 (subchannel 138B9138): connectivity changed: state=CONNECTING, error="No Error", shutting_down=0
I0921 11:39:18.021000000  8920 connectivity_state.cc:164] SET: 138D9848 pick_first: CONNECTING --> CONNECTING [connecting_changed] error=00000000 "No Error"
I0921 11:39:18.021000000  8920 src/core/ext/filters/client_channel/lb_policy/subchannel_list.h:313] [pick_first 138D9818] subchannel list 08E19208 index 7 of 11 (subchannel 138B9138): renewing watch: requesting connectivity change notification (from CONNECTING)
I0921 11:39:18.026000000  8920 connectivity_state.cc:116] CONWATCH: 138B91A8 subchannel: from CONNECTING [cur=CONNECTING] notify=08F0B0C4
I0921 11:39:18.027000000  8920 completion_queue.cc:851] grpc_completion_queue_next(cq=08DED7E0, deadline=gpr_timespec { tv_sec: -9223372036854775808, tv_nsec: 0, clock_type: 0 }, reserved=00000000)
I0921 11:39:18.028000000  8920 completion_queue.cc:951] RETURN_EVENT[08DED7E0]: QUEUE_TIMEOUT
I0921 11:39:18.030000000  8920 tcp_custom.cc:348] Creating TCP endpoint 134C5B98
I0921 11:39:18.030000000  8920 tcp_client_custom.cc:69] CLIENT_CONNECT: ipv4:172.217.16.202:443: on_alarm: error="Cancelled"
I0921 11:39:18.030000000  8920 handshaker.cc:141] handshake_manager 034B6840: adding handshaker http_connect [137A6430] at index 0
I0921 11:39:18.030000000  8920 ssl_transport_security.cc:211]      HANDSHAKE START -       TLS client start_connect  - !!!!!!
I0921 11:39:18.030000000  8920 ssl_transport_security.cc:211]                 LOOP -    TLS client enter_early_data  - !!!!!!
I0921 11:39:18.030000000  8920 ssl_transport_security.cc:211]                 LOOP -   TLS client read_server_hello  - !!!!!!
I0921 11:39:18.031000000  8920 handshaker.cc:141] handshake_manager 034B6840: adding handshaker security [08F078C8] at index 1
I0921 11:39:18.031000000  8920 handshaker.cc:212] handshake_manager 034B6840: error="No Error" shutdown=0 index=0, args={endpoint=13454A30, args=08E1AC60 {size=8: grpc.primary_user_agent=grpc-node/1.13.1, grpc.client_channel_factory=58CBB994, grpc.channel_credentials=134C5318, grpc.server_uri=dns:///firestore.googleapis.com, grpc.default_authority=firestore.googleapis.com, grpc.http2_scheme=https,
 grpc.security_connector=08EDF460, grpc.subchannel_address=ipv4:172.217.16.202:443}, read_buffer=08D55870 (length=0), exit_early=0}
I0921 11:39:18.031000000  8920 handshaker.cc:253] handshake_manager 034B6840: calling handshaker http_connect [137A6430] at index 0
I0921 11:39:18.031000000  8920 handshaker.cc:212] handshake_manager 034B6840: error="No Error" shutdown=0 index=1, args={endpoint=13454A30, args=08E1AC60 {size=8: grpc.primary_user_agent=grpc-node/1.13.1, grpc.client_channel_factory=58CBB994, grpc.channel_credentials=134C5318, grpc.server_uri=dns:///firestore.googleapis.com, grpc.default_authority=firestore.googleapis.com, grpc.http2_scheme=https,
 grpc.security_connector=08EDF460, grpc.subchannel_address=ipv4:172.217.16.202:443}, read_buffer=08D55870 (length=0), exit_early=0}
I0921 11:39:18.031000000  8920 handshaker.cc:253] handshake_manager 034B6840: calling handshaker security [08F078C8] at index 1
I0921 11:39:18.032000000  8920 tcp_custom.cc:234] WRITE 134C5B98 (peer=ipv4:172.217.16.202:443): 16 03 01 00 9f 01 00 00 9b 03 03 20 0d 33 b4 b9 36 9c bc b1 56 cf f9 8b 2c 96 35 7a 10 05 c7 42 8f 0b e8 f5 55 b9 5b d2 03 9e 40 00 00 08 c0 2b c0 2c c0 2f c0 30 01 00 00 6a ff 01 00 01 00 00 00 00 1d 00 1b 00 00 18 66 69 72 65 73 74 6f 72 65 2e 67 6f 6f 67 6c 65 61 70 69 73 2e 63 6f 6d 00 17 00 00 00
23 00 00 00 0d 00 14 00 12 04 03 08 04 04 01 05 03 08 05 05 01 08 06 06 01 02 01 33 74 00 00 00 10 00 0e 00 0c 08 67 72 70 63 2d 65 78 70 02 68 32 00 0b 00 02 01 00 00 0a 00 04 00 02 00 17 '........... .3..6...V...,.5z...B....U.[...@....+.,./.0...j..............firestore.googleapis.com.....#..........................3t.........grpc-exp.h2..............'
I0921 11:39:18.033000000  8920 completion_queue.cc:851] grpc_completion_queue_next(cq=08DED7E0, deadline=gpr_timespec { tv_sec: -9223372036854775808, tv_nsec: 0, clock_type: 0 }, reserved=00000000)
I0921 11:39:18.044000000  8920 completion_queue.cc:951] RETURN_EVENT[08DED7E0]: QUEUE_TIMEOUT
I0921 11:39:18.046000000  8920 tcp_custom.cc:217] write complete on 134C5B98: error="No Error"
I0921 11:39:18.046000000  8920 resource_quota.cc:795] RQ anonymous_pool_8f00c70 ipv4:172.217.16.202:443: alloc 8192; free_pool -> -8192
I0921 11:39:18.046000000  8920 resource_quota.cc:292] RQ: check allocation for user 08DDBA00 shutdown=0 free_pool=-8192
I0921 11:39:18.047000000  8920 resource_quota.cc:318] RQ anonymous_pool_8f00c70 ipv4:172.217.16.202:443: grant alloc 8192 bytes; rq_free_pool -> 9223372036854767615
I0921 11:39:18.048000000  8920 tcp_custom.cc:174] TCP:134C5B98 read_allocation_done: "No Error"
I0921 11:39:18.048000000  8920 tcp_custom.cc:191] Initiating read on 134C5B98: error="No Error"
I0921 11:39:18.050000000  8920 completion_queue.cc:851] grpc_completion_queue_next(cq=08DED7E0, deadline=gpr_timespec { tv_sec: -9223372036854775808, tv_nsec: 0, clock_type: 0 }, reserved=00000000)
I0921 11:39:18.052000000  8920 completion_queue.cc:951] RETURN_EVENT[08DED7E0]: QUEUE_TIMEOUT
I0921 11:39:18.056000000  8920 resource_quota.cc:818] RQ anonymous_pool_8f00c70 ipv4:172.217.16.202:443: free 8192; free_pool -> 8192
I0921 11:39:18.057000000  8920 tcp_custom.cc:128] TCP:134C5B98 call_cb 08F079B8 58B7D4B0:08F078C8
I0921 11:39:18.064000000  8920 tcp_custom.cc:132] read: error={"created":"@1537522758.056000000","description":"EOF","file":"..\deps\grpc\src\core\lib\iomgr\tcp_uv.cc","file_line":107}
D0921 11:39:18.065000000  8920 security_handshaker.cc:129] Security handshake failed: {"created":"@1537522758.065000000","description":"Handshake read failed","file":"..\deps\grpc\src\core\lib\security\transport\security_handshaker.cc","file_line":321,"referenced_errors":[{"created":"@1537522758.056000000","description":"EOF","file":"..\deps\grpc\src\core\lib\iomgr\tcp_uv.cc","file_line":107}]}
I0921 11:39:18.066000000  8920 tcp_custom.cc:286] TCP 134C5B98 shutdown why={"created":"@1537522758.065000000","description":"Handshake read failed","file":"..\deps\grpc\src\core\lib\security\transport\security_handshaker.cc","file_line":321,"referenced_errors":[{"created":"@1537522758.056000000","description":"EOF","file":"..\deps\grpc\src\core\lib\iomgr\tcp_uv.cc","file_line":107}]}
I0921 11:39:18.067000000  8920 handshaker.cc:212] handshake_manager 034B6840: error={"created":"@1537522758.065000000","description":"Handshake read failed","file":"..\deps\grpc\src\core\lib\security\transport\security_handshaker.cc","file_line":321,"referenced_errors":[{"created":"@1537522758.056000000","description":"EOF","file":"..\deps\grpc\src\core\lib\iomgr\tcp_uv.cc","file_line":107}]} shut
down=0 index=2, args={endpoint=00000000, args=00000000 {size=0: (null)}, read_buffer=00000000 (length=0), exit_early=0}
I0921 11:39:18.069000000  8920 handshaker.cc:240] handshake_manager 034B6840: handshaking complete -- scheduling on_handshake_done with error={"created":"@1537522758.065000000","description":"Handshake read failed","file":"..\deps\grpc\src\core\lib\security\transport\security_handshaker.cc","file_line":321,"referenced_errors":[{"created":"@1537522758.056000000","description":"EOF","file":"..\deps\
grpc\src\core\lib\iomgr\tcp_uv.cc","file_line":107}]}
I0921 11:39:18.072000000  8920 connectivity_state.cc:164] SET: 138B91A8 subchannel: CONNECTING --> TRANSIENT_FAILURE [connect_failed] error=08E97DF8 {"created":"@1537522758.071000000","description":"Connect Failed","file":"..\deps\grpc\src\core\ext\filters\client_channel\subchannel.cc","file_line":641,"grpc_status":14,"referenced_errors":[{"created":"@1537522758.065000000","description":"Handshake
read failed","file":"..\deps\grpc\src\core\lib\security\transport\security_handshaker.cc","file_line":321,"referenced_errors":[{"created":"@1537522758.056000000","description":"EOF","file":"..\deps\grpc\src\core\lib\iomgr\tcp_uv.cc","file_line":107}]}]}
I0921 11:39:18.075000000  8920 connectivity_state.cc:190] NOTIFY: 138B91A8 subchannel: 08F0B0C4
I0921 11:39:18.076000000  8920 subchannel.cc:646] Connect failed: {"created":"@1537522758.065000000","description":"Handshake read failed","file":"..\deps\grpc\src\core\lib\security\transport\security_handshaker.cc","file_line":321,"referenced_errors":[{"created":"@1537522758.056000000","description":"EOF","file":"..\deps\grpc\src\core\lib\iomgr\tcp_uv.cc","file_line":107}]}
I0921 11:39:18.076000000  8920 resource_quota.cc:508] RU shutdown 08DDBA00
I0921 11:39:18.078000000  8920 src/core/ext/filters/client_channel/lb_policy/subchannel_list.h:404] [pick_first 138D9818] subchannel list 08E19208 index 7 of 11 (subchannel 138B9138): connectivity changed: state=TRANSIENT_FAILURE, error={"created":"@1537522758.071000000","description":"Connect Failed","file":"..\deps\grpc\src\core\ext\filters\client_channel\subchannel.cc","file_line":641,"grpc_sta
tus":14,"referenced_errors":[{"created":"@1537522758.065000000","description":"Handshake read failed","file":"..\deps\grpc\src\core\lib\security\transport\security_handshaker.cc","file_line":321,"referenced_errors":[{"created":"@1537522758.056000000","description":"EOF","file":"..\deps\grpc\src\core\lib\iomgr\tcp_uv.cc","file_line":107}]}]}, shutting_down=0
I0921 11:39:18.079000000  8920 src/core/ext/filters/client_channel/lb_policy/subchannel_list.h:332] [pick_first 138D9818] subchannel list 08E19208 index 7 of 11 (subchannel 138B9138): stopping connectivity watch

How can this connectivity problem be solved? Which settings need to be changed in the corporate proxy or firewall in order for Firestore / gRPC to be able to connect?

Junker answered 21/9, 2018 at 10:5 Comment(2)

any progress on this one? having a similar problem. using firestore part of a chrome browser extesnion, clients who are connected to a corporate internet with firewall, cant use the extesnion because reads to firestore fail – Recall 24/10, 2018 at 13:19

In our case the problem seems to be that the gRPC node module does not use the proxy auth credentials given to electron. We are currently checking, if https_proxy env var is picked up by gRPC when setting the variable programmatically. – Junker 27/10, 2018 at 18:28

J

2

Warning: GRPC currently only supports HTTP Basic Auth for proxies. The proxy authentication method here will currently not work, if the proxy requires authentication via Digest. I have filed an issue for it here: https://github.com/grpc/grpc/issues/18250

There seem to be two workarounds currently:

Possibility 1: The solution is to set the environment variables for http_proxy and https_proxy before loading the Electron BrowserWindow that initiates the Firestore connection.

An possible way is to ask for proxy credentials in the BrowserWindow and return them to the main process using an event (in this example set-proxy-url). After setting the environment variables correctly for gRPC to pick them up, reload the BrowserWindow (or open a new one).

Example code (main):

ipcMain.on('set-proxy-url', (event: Electron.Event, arg: any) => {
  if (arg.authInfo.isProxy) {
    // In our case this is a http proxy (HTTPS is tunneled)
    const httpUrl = `http://${arg.user}:${arg.pass}@${arg.authInfo.host}:${arg.authInfo.port}`

    process.env.http_proxy = httpUrl
    process.env.https_proxy = httpUrl

    // TODO: Reload the BrowserWindow that uses Firestore
  }
})

Example code (renderer):

ipcRenderer.send('set-proxy-url', {
     authInfo, // This comes from app.on('login', (event, webContents, request, authInfo, callback) => { ... }
     user: usernameRetrievedFromUser,
     pass: passwordRetrievedFromUser
})

Possibility 2: Configure a VPN tunnel and configure CNAME in your internal DNS to point *.googleapis.com to private.googleapis.com as explained here.

Junker answered 7/3, 2019 at 9:25 Comment(0)

A

2

As can be seen in the source code, the node.js client library accesses Firestore through gRPC at firestore.googleapis.com on port 443.

So for your application to work with Firestore successfully, the firewall/proxy must allow access to firestore.googleapis.com on port 443 via TCP. Why this doesn't work at the moment is a question to the current configuration of the firewall/proxy, most probably it is blocked by default in some catch-all rule.

Angie answered 23/9, 2018 at 17:33 Comment(5)

Hi Mikhail, the customer environment allows our application to perform HTTPS requests via port 443 - otherwise the application would not be able to connect to our own backend server for login in the first place. This does not seem to be the problem. We will however double-check this again. Thanks! – Junker 23/9, 2018 at 17:38

@Junker I looked in more detail and understood that it uses gRPC, not REST. Maybe the problem is that the gRPC payload does not look like the payload of plain HTTP requests, and the firewall has some kind of DPI (deep packet introspection) which doesn't let the data pass through because it doesn't look like what the firewall expects for port 443. – Angie 23/9, 2018 at 17:44

@Junker In this case it will be needed to turn off deep packet introspection for firestore.googleapis.com:443 on the firewall. – Angie 23/9, 2018 at 17:45

That might be possible. Thanks for the hint. I guess we'll have to check this with the IT administrators who have configured the firewall. – Junker 23/9, 2018 at 17:46

We have asked - deep packet introspection is not being used. So it must be something different... – Junker 27/9, 2018 at 12:34

J

2