How to setMasterUrl in Ignite XML config for Kubernetes IPFinder

Question

Using test config with Ignite 2.4 and k8s 1.9:

<?xml version="1.0" encoding="UTF-8"?>
    <beans xmlns="http://www.springframework.org/schema/beans"
           xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
           xmlns:util="http://www.springframework.org/schema/util"
           xsi:schemaLocation="
            http://www.springframework.org/schema/beans
            http://www.springframework.org/schema/beans/spring-beans.xsd
            http://www.springframework.org/schema/util
            http://www.springframework.org/schema/util/spring-util.xsd">

    <bean class="org.apache.ignite.configuration.IgniteConfiguration">
      <property name="discoverySpi">
        <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
          <property name="ipFinder">
            <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes.TcpDiscoveryKubernetesIpFinder"/>
          </property>
        </bean>
      </property>
    </bean>
</beans>

Unable to find Kubernetes API Server at https://kubernetes.default.svc.cluster.local:443 Can I set the API Server URL in the XML config file? How?

Margiemargin

I was able to provide masterUrl in XML config, but still receive the same 403 error: Caused by: java.io.IOException: Server returned HTTP response code: 403 for URL: 10.244.0.1:443/api/v1/namespaces/default/endpoints/ignite – Margiemargin

Answer 1

Take a look at this thread: http://apache-ignite-users.70518.x6.nabble.com/Unable-to-connect-ignite-pods-in-Kubernetes-using-Ip-finder-td18009.html

The problem of 403 error can be solved by granting more permissions to the service account.

Pelias

Thanks. This link lead to the correct answer. I was able to find Ignite nodes after creating RoleBinding with clusterrole=admin fir service account: kubectl create rolebinding igniteRoleBinding \ --clusterrole=admin \ --serviceaccount=default:ignite \ --namespace=default – Margiemargin

Still not sure why admin is needed, but new to RBAC and permissions in k8s so I will research further. – Margiemargin

Answer 2

@Denis was right.

Kubernetes using RBAC access controlling system and you need to authorize your pod to access to API.

For that, you need to add a Service Account to your pod.

So, for do that you need:

1. Create a service account and set role for it:

   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: ignite
     namespace: <Your namespace>
   

2. I am not sure that permissions to access only pods will be enough for Ignite, but if not - you can add as more permissions as you want. Here is example of different kind of roles with large list of permissions. So, now we create Cluster Role for your app:

   apiVersion: rbac.authorization.k8s.io/v1beta1
   kind: ClusterRole
   metadata:
     name: ignite
     namespace: <Your namespace>
   rules:
   - apiGroups:
     - ""
     resources:
     - pods # Here is resources you can access
     verbs: # That is what you can do with them
     - get
     - list
     - watch
   

3. Create binding for that role:

   kind: ClusterRoleBinding
   apiVersion: rbac.authorization.k8s.io/v1beta1
   metadata:
     name: ignite
   roleRef:
     kind: ClusterRole
     name: ignite
     apiGroup: rbac.authorization.k8s.io
   subjects:
   - kind: ServiceAccount
     name: ignite
     namespace: <Your namespace>
   

4. Now, you need to associate ServiceAccount to pods with your application:

   apiVersion: extensions/v1beta1
   kind: DaemonSet
   metadata:
     ....
   spec:
     template:
       spec:
         serviceAccountName: ignite
   

After that, your application will have an access to K8s API. P.S. Do not forget to change <Your namespace> to namespace where you running Ignition.

Laevorotatory

When I try to run the 4th step kubectl create -f daemon.yaml, I get the following error. error: error validating "daemon.yaml": error validating data: ValidationError(DaemonSet.spec.template.spec): missing required field "containers" in io.k8s.api.core.v1.PodSpec; if you choose to ignore these errors, turn validation off with --validate=false – Gender

Answer 3

Platform versions

• Kubernetes: v1.8
• Ignite: v2.4

@Anton Kostenko design is mostly right, but here's a refined suggestion that works and grants least access privileges to Ignite.

1. If you're using a Deployment to manage Ignite, then all of your Pods will launch within a single namespace. Therefore, you should really use a Role and a RoleBinding to grant API access to the service account associated with your deployment.

2. The TcpDiscoveryKubernetesIpFinder only needs access to the endpoints for the headless service that selects your Ignite pods. The following 2 manifests will grant that access.

   apiVersion: rbac.authorization.k8s.io/v1
   kind: Role
   metadata:
     name: ignite-endpoint-access
     namespace: <your-ns>
     labels:
       app: ignite
   rules:
     - apiGroups: [""]
       resources: ["endpoints"]
       resourceNames: ["<your-headless-svc>"]
       verbs: ["get"]
   
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: RoleBinding
   metadata:
     name: ignite-role-binding
     labels:
       app: ignite
   subjects:
     - kind: ServiceAccount
       name: <your-svc-account>
   roleRef:
     kind: Role
     name: ignite-endpoint-access
     apiGroup: rbac.authorization.k8s.io
   

Alidus

Answer 4

Tested Version:

Kubernetes: v1.8

Ignite: v2.4

This is going to be little bit more permissive.

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: ignite-rbac
subjects:
  - kind: ServiceAccount
    name: default
    namespace: <namespace>
roleRef:
  kind: ClusterRole
  name: cluster-admin
  apiGroup: rbac.authorization.k8s.io

Extenuate

Answer 5

If you're getting 403 unauthorized then your service account that made your resources may not have good enough permissions. you should update your permissions after you ensure that your namespace and service account and deployments/ replica sets are exactly the way you want it to be.

This link is very helpful to setting permissions for service accounts: https://kubernetes.io/docs/reference/access-authn-authz/rbac/#service-account-permissions

Gwen