Kubernetes and AWS: Set LoadBalancer to use predefined Security Group

Question

As the title says, I am looking for a way to force a LoadBalancer service to use a predefined security group in AWS. I do not want to have to manually edit the inbound/outbound rules of the security group that is created for the ELB by Kubernetes. I have not been able to find anything within the documentation, nor have I located anything that works elsewhere online. Here is my current template:

apiVersion: v1
kind: Service
metadata:
  name: ds-proxy
spec:
  type: LoadBalancer
  ports:
  - port: 8761 # the port that this service should serve on
    targetPort: 8761
    protocol: TCP
  selector:
    app: discovery-service

Diahann

Answer 1

EDIT: 2021 - I am told my answer is now out of date, refer to https://mcmap.net/q/719759/-kubernetes-and-aws-set-loadbalancer-to-use-predefined-security-group instead.

You cannot prevent Kubernetes from creating a new security group. But since Andonaeus' answer was submitted a new feature has been added which allows for explicitly defining inbound permissions via your service's configuration file.

See the user guide details for the specifics. The example provided there shows that by using spec.loadBalancerSourceRanges you can provide allow inbound IPs:

In the following example, a load blancer will be created that is only accessible to clients with IP addresses from 130.211.204.1 and 130.211.204.2.

apiVersion: v1
kind: Service
metadata:
  name: myapp
spec:
  ports:
    - port: 8765
      targetPort: 9376
  selector:
    app: example
  type: LoadBalancer
  loadBalancerSourceRanges:
  - 130.211.204.1/32
  - 130.211.204.2/32

Unfold

This is no longer true. Please see my answer: https://mcmap.net/q/719759/-kubernetes-and-aws-set-loadbalancer-to-use-predefined-security-group. – Scan

Answer 2

You can not restrict kubernetes from creating new security group, but you can specify existing security groups using annotations as mentioned in the documentation:

service.beta.kubernetes.io/aws-load-balancer-extra-security-groups: "sg-53fae93f,sg-42efd82e" -> A list of additional security groups to be added to ELB

Greengrocer

Link to documentation! kubernetes.io/docs/concepts/cluster-administration/… – Ukulele

I tried to use aws-load-balancer-extra-security-groups to add extra SG with the ELB. But to my surprise, ELB couldn't be created. – Larvicide

Answer 3

I realize this post is now a couple of years old, but it came up for me in a google search. It looks like it is now possible with k8s 1.7+ to prevent kubernetes from creating a security group. See https://github.com/kubernetes/kops/blob/release-1.9/docs/cluster_spec.md#cloudconfig for more info.

Toner

Answer 4

It looks like this is not currently possible. Via the following code in the api, https://github.com/kubernetes/kubernetes/blob/37b5726716231c13117c4b05a841e00417b92cda/pkg/cloudprovider/providers/aws/aws.go :

func (s *AWSCloud) EnsureLoadBalancer(name, region string, publicIP net.IP, ports []*api.ServicePort, hosts []string, affinity api.ServiceAffinity) (*api.LoadBalancerStatus, error) {
glog.V(2).Infof("EnsureLoadBalancer(%v, %v, %v, %v, %v)", name, region,    publicIP, ports, hosts)

.
.
.

// Create a security group for the load balancer
var securityGroupID string
{
    sgName := "k8s-elb-" + name
    sgDescription := "Security group for Kubernetes ELB " + name
    securityGroupID, err = s.ensureSecurityGroup(sgName, sgDescription, vpcId)
    if err != nil {
        glog.Error("Error creating load balancer security group: ", err)
        return nil, err
    }

    permissions := []*ec2.IpPermission{}
    for _, port := range ports {
        portInt64 := int64(port.Port)
        protocol := strings.ToLower(string(port.Protocol))
        sourceIp := "0.0.0.0/0"

        permission := &ec2.IpPermission{}
        permission.FromPort = &portInt64
        permission.ToPort = &portInt64
        permission.IpRanges = []*ec2.IpRange{{CidrIp: &sourceIp}}
        permission.IpProtocol = &protocol

        permissions = append(permissions, permission)
    }
    _, err = s.ensureSecurityGroupIngress(securityGroupID, permissions)
    if err != nil {
        return nil, err
    }
}
securityGroupIDs := []string{securityGroupID}

.
.
.

}

There is no way to prevent it from creating a security group.

Diahann

Answer 5

This is now possible and has been for a while.

See https://kubernetes.io/docs/concepts/services-networking/service/

        service.beta.kubernetes.io/aws-load-balancer-security-groups: "sg-53fae93f"
        # A list of existing security groups to be configured on the ELB created. Unlike the annotation
        # service.beta.kubernetes.io/aws-load-balancer-extra-security-groups, this replaces all other security groups previously assigned to the ELB and also overrides the creation 
        # of a uniquely generated security group for this ELB.
        # The first security group ID on this list is used as a source to permit incoming traffic to target worker nodes (service traffic and health checks).
        # If multiple ELBs are configured with the same security group ID, only a single permit line will be added to the worker node security groups, that means if you delete any
        # of those ELBs it will remove the single permit line and block access for all ELBs that shared the same security group ID.
        # This can cause a cross-service outage if not used properly

Pay close attention to the fact that if you have multiple load balancers and you destroy one the ingress rule that allows the worker SG to talk to the ELB SG will be revoked.

What I would like is a "service.beta.kubernetes.io/aws-load-balancer-stop-messing-with-my-security-groups: true" annotation.

We use service.beta.kubernetes.io/aws-load-balancer-target-node-labels also and as an experiment eventually where I denied ec2:RevokeSecurityGroupIngress but this can cause other issues.

Official AWS advice is to not share LB Security groups.

Scan

Do you know if there's a way to use some sort of selector for the security group? I created a SG with Terraform and I want to use it here, but if the SG changes I don't want to have to change the id in service.beta.kubernetes.io/aws-load-balancer-security-groups manually. Thanks! – Institutional

In practice that security group should not change (though yes, silly things like modifying the Group Description can trigger it to recreate and will likely fail if things are using it). If you want to make it more dynamic you can always use something like jinja to generate the values for service.beta.kubernetes.io/aws-load-balancer-security-groups by looking up the security group based on its name. Would be great if they could take the SG ID or the GroupName as an argument. – Scan

@Institutional - I discovered recently that (for ALBs and NLBs controlled by the aws load balancer controller at least) the Name tag is now a valid argument too! "Both name or ID of securityGroups are supported. Name matches a Name tag, not the groupName attribute." kubernetes-sigs.github.io/aws-load-balancer-controller/v2.4/… – Scan