Kubernetes on AWS cloud provider

Question

I installed CentOS Atomic Host as operating system for kubernetes on AWS.

Everything works fine, but it seems I missed something.

I did not configure cloud provider and can not find any documentation on that.

In this question I want to know: 1. What features cloud provider gives to kubernetes? 2. How to configure AWS cloud provider?

UPD 1: external load balancer does not work; I have not tested awsElasticBlockStore yet, but I also suspect it does not work.

UPD 2: Service details:

$ kubectl get svc nginx-service-aws-lb -o yaml
apiVersion: v1
kind: Service
metadata:
  creationTimestamp: 2016-01-02T09:51:40Z
  name: nginx-service-aws-lb
  namespace: default
  resourceVersion: "74153"
  selfLink: /api/v1/namespaces/default/services/nginx-service-aws-lb
  uid: 6c28b718-b136-11e5-9bda-06c2feb29b0d
spec:
  clusterIP: 10.254.172.185
  ports:
  - name: http-proxy-protocol
    nodePort: 31385
    port: 8080
    protocol: TCP
    targetPort: 8080
  - name: https-proxy-protocol
    nodePort: 31370
    port: 8443
    protocol: TCP
    targetPort: 8443
  selector:
    app: nginx
  sessionAffinity: None
  type: LoadBalancer
status:
  loadBalancer: {}

Foreleg

Answer 1

I can't speak to the ProjectAtomic bits, nor to the KUBERNETES_PROVIDER env-var, since my experience has been with the CoreOS provisioner. I will talk about my experiences and see if that helps you dig a little more into your setup.

Foremost, it is absolutely essential that the controller EC2 and the worker EC2 machines have the correct IAM role that will enable the machines to make AWS calls on behalf of your account. This includes things like provisioning ELBs and working with EBS Volumes (or attaching an EBS Volume to themselves, in the case of the worker). Without that, your cloud-config experience will go nowhere. I'm pretty sure the IAM payloads are defined somewhere other than those .go files, which are hard to read, but that's the quickest link I had handy to show what's needed.

Fortunately, the answer to that question, and the one I'm about to talk about, are both centered around the apiserver and the controller-manager. The configuration of them and the logs they output.

Both the apiserver and the controller-manager have an argument that points to an on-disk cloud configuration file that regrettably isn't documented anywhere except for the source. That Zone field is, in my experience, optional (just like they say in the comments). However, it was seeing the KubernetesClusterTag that led me to follow that field around in the code to see what it does.

If your experience is anything like mine, you'll see in the docker logs of the controller-manager a bunch of error messages about how it created the ELB but could not find any subnets to attach to it; (that "docker logs" bit is presuming, of course, that ProjectAtomic also uses docker to run the Kubernetes daemons).

Once I attached a Tag named KubernetesCluster and set every instance of the Tag to the same string (it can be anything, AFAIK), then the aws_loadbalancer was able to find the subnet in the VPC and it attached the Nodes to the ELB and everything was cool -- except for the part about it can only create Internet facing ELBs, right now. :-(

Just for clarity: the aws.cfg contains a field named KubernetesClusterTag that allows you to redefine the Tag that Kubernetes will look for; without any value in that file, Kuberenetes will use the Tag name KubernetesCluster.

I hope this helps you and I hope it helps others, because once Kubernetes is up, it's absolutely amazing.

Mission

Hi Daniel, thank you for your reply. I found AWS IAM roles in kubernetes repository . – Foreleg

regarding atomic host, check this document docs.openshift.org/latest/install_config/… – Plumbery

Answer 2

1. What features cloud provider gives to kubernetes?

Some features that I know: the external loadbalancer, the persistent volumes.

2. How to configure AWS cloud provider?

There is a environment var called KUBERNETES_PROVIDER, but it seems the env var only matters when people start a k8s cluster. Since you said "everything works fine", I guess you don't need any further configuration to use the features I mentioned above.

Yaker

Thanks for your answer. I tested external LB and it not works. – Foreleg

Did you follow the getting started guides for aws? github.com/kubernetes/kubernetes/blob/release-1.1/docs/…. Please also refer to github.com/kubernetes/kubernetes/blob/…. If creating a service with Type=Loadbalancer doesn't create an external loadbalancer on AWS, that sounds like a bug. Kubernetes does have ELB support, as mentioned in the "under the hood" doc. – Feleciafeledy

Can you say more what you mean that the LB is not working? Does it show up when you execute the following command: kubectl get svc -o yaml and see if it returns a hostname attribute? The problem is that AWS does not give ELBs IP addresses, just host names. – Pettitoes

Yes, I read that guide. Cluster created with thehttp://www.projectatomic.io guides. – Foreleg

You shouldn't have to configure the cloud provider. Can you paste in your service .yaml file? Also, can you paste in your kubernetes versions (both master and node)? kubctl get version – Pettitoes

Answer 3

These are some of the features that cloud provider provides to Kubernetes.

1. Load balancing integration.
2. Volume provisioning
3. Node autoscaling
4. Node management
5. Monitoring and Logging
6. Network
7. Security

Varian