What are default VPC firewall rules for App Engine Flex

Question

I have a few services deployed in one App Engine flexible environment project. The service instances are running in the same VPC network (not default) which is specified in their app.yaml files and assigned IP addresses from the same subnetwork. By default both services are accessible from outside and I want to configure firewall rules to leave only the default service allowed for incoming traffic.

So as the first step I'm configuring a rule to deny all ingress for all instances in this VPC, with the idea of creating another rule with a higher priority to allow traffic to only one instance which is running the default service.
The problem is that when I create a firewall rule with priority 1001 to block all incoming traffic my default service still get requests. But if I specify the priority 1000 or bellow then the traffic is blocked and I get 502 Server Error.

The question is of course WHY? The VPC documentation states that there are only two default firewall rules used for all manually created VPC networks:

• A default "allow egress" rule.
  Allows all egress connections. Rule has a priority of 65535.
• A default "deny ingress" rule.
  Deny all ingress connection. Rule has a priority of 65535

Then how come that my 1001 priority rule fails to block the ingress and the exactly the same rule with priority 1000 (or below) works as expected? Are there any other default firewall rules for Flex instances or am I missing something else here?

All

There are two firewalls that can be in play: a GCP firewall that applies to Compute Engine instances, which flexible environment instances fall into, and an App Engine firewall that affects both standard environment and flexible environment at the HTTP (layer 7) level rather than the layer 3 TCP level. – Stroup

Right. The default firewall rule for App Engine allows all ingress and has priority 2147483647. How those firewalls play together then? My main question to put it simply is why deny-all-ingress rule with priority 1001 doesn't block incoming http for AE Flex instances, but the same rule with priority 1000 and below does. This behaviour is undocumented and looks very strange. – All

Hi, did you managed to solve this? We are trying to do the same on our default network but it seems that all the trafic is managed by the app-engines firewall. – Endsley

thanks, your question helped us troubleshoot this issue. Let me know if you ever find documentation for this – Facsimile

Answer 1

From what I understand, both firewalls must allow the traffic in order for a request to reach your App Engine. If any one has a good reason to block the request, then the request is blocked.

You can know which firewall is blocking by looking at the response code :

• A 502 -> VPC firewall
• A 403 -> App Engine Firewall

If logs are enabled, you can know which rule on the VPC firewall is causing the deny.

Regarding the "1000 against 1001 priority" rule, that's a very strange (and undocumented) behavior. So let's document it here for the folks who might stumble on it :

Rules on the VPC network for flexible App Engine instances seems to work only if priority is below 1000.

Zedekiah

Do you have any idea how the firewalls interact with each other? It seems like I can not "allow" on a specific IP in the VPC firewall because after it passes the App Engine firewall, it uses some load balancer so I have no way of distinguishing which IP the request is coming from. – Facsimile