Spinnaker on Kubernetes: Not able to start the spinnaker containers in kubernetes
Asked Answered
A

4

20

I was following this documentation to setup Spinnaker on Kubernetes. I ran the scripts as they specified. Then the replication controllers and services are started. But some of PODs are not started

root@nveeru~# kubectl get pods --namespace=spinnaker
NAME                           READY     STATUS             RESTARTS   AGE
data-redis-master-v000-zsn7e   1/1       Running            0          2h
spin-clouddriver-v000-6yr88    1/1       Running            0          47m
spin-deck-v000-as4v7           1/1       Running            0          2h
spin-echo-v000-g737r           1/1       Running            0          2h
spin-front50-v000-v1g6e        0/1       CrashLoopBackOff   21         2h
spin-gate-v000-9k401           0/1       Running            0          2h
spin-igor-v000-zfc02           1/1       Running            0          2h
spin-orca-v000-umxj1           0/1       CrashLoopBackOff   20         2h

Then I kubectl describe the pods

root@veeru:~# kubectl describe pod spin-orca-v000-umxj1 --namespace=spinnaker
Name:       spin-orca-v000-umxj1
Namespace:  spinnaker
Node:       172.25.30.21/172.25.30.21
Start Time: Mon, 19 Sep 2016 00:53:00 -0700
Labels:     load-balancer-spin-orca=true,replication-controller=spin-orca-v000
Status:     Running
IP:     172.16.33.8
Controllers:    ReplicationController/spin-orca-v000
Containers:
  orca:
    Container ID:   docker://e6d77e9fd92dc9614328d09a5bfda319dc7883b82f50cc352ff58dec2e933d04
    Image:      quay.io/spinnaker/orca:latest
    Image ID:       docker://sha256:2400633b89c1c7aa48e5195c040c669511238af9b55ff92201703895bd67a131
    Port:       8083/TCP
    QoS Tier:
      cpu:      BestEffort
      memory:       BestEffort
    State:      Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Mon, 19 Sep 2016 02:59:09 -0700
      Finished:     Mon, 19 Sep 2016 02:59:39 -0700
    Ready:      False
    Restart Count:  21
    Readiness:      http-get http://:8083/env delay=20s timeout=1s period=10s #success=1 #failure=3
    Environment Variables:
Conditions:
  Type      Status
  Ready     False 
Volumes:
  spinnaker-config:
    Type:   Secret (a volume populated by a Secret)
    SecretName: spinnaker-config
  default-token-6irrl:
    Type:   Secret (a volume populated by a Secret)
    SecretName: default-token-6irrl
Events:
  FirstSeen LastSeen    Count   From            SubobjectPath       Type        Reason      Message
  --------- --------    -----   ----            -------------       --------    ------      -------
  1h        3m      22  {kubelet 172.25.30.21}  spec.containers{orca}   Normal      Pulling     pulling image "quay.io/spinnaker/orca:latest"
  1h        3m      22  {kubelet 172.25.30.21}  spec.containers{orca}   Normal      Pulled      Successfully pulled image "quay.io/spinnaker/orca:latest"
  1h        3m      13  {kubelet 172.25.30.21}  spec.containers{orca}   Normal      Created     (events with common reason combined)
  1h        3m      13  {kubelet 172.25.30.21}  spec.containers{orca}   Normal      Started     (events with common reason combined)
  1h        3m      23  {kubelet 172.25.30.21}  spec.containers{orca}   Warning     Unhealthy   Readiness probe failed: Get http://172.16.33.8:8083/env: dial tcp 172.16.33.8:8083: connection refused
  1h        <invalid>   399 {kubelet 172.25.30.21}  spec.containers{orca}   Warning     BackOff     Back-off restarting failed docker container
  1h        <invalid>   373 {kubelet 172.25.30.21}              Warning     FailedSync  Error syncing pod, skipping: failed to "StartContainer" for "orca" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=orca pod=spin-orca-v000-umxj1_spinnaker(ee2511f0-7e3d-11e6-ab16-0022195df673)"

spin-front50-v000-v1g6e

root@veeru:~# kubectl describe pod spin-front50-v000-v1g6e  --namespace=spinnaker
Name:       spin-front50-v000-v1g6e
Namespace:  spinnaker
Node:       172.25.30.21/172.25.30.21
Start Time: Mon, 19 Sep 2016 00:53:00 -0700
Labels:     load-balancer-spin-front50=true,replication-controller=spin-front50-v000
Status:     Running
IP:     172.16.33.9
Controllers:    ReplicationController/spin-front50-v000
Containers:
  front50:
    Container ID:   docker://f5559638e9ea4e30b3455ed9fea2ab1dd52be95f177b4b520a7e5bfbc033fc3b
    Image:      quay.io/spinnaker/front50:latest
    Image ID:       docker://sha256:e774808d76b096f45d85c43386c211a0a839c41c8d0dccb3b7ee62d17e977eb4
    Port:       8080/TCP
    QoS Tier:
      memory:       BestEffort
      cpu:      BestEffort
    State:      Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Mon, 19 Sep 2016 03:02:08 -0700
      Finished:     Mon, 19 Sep 2016 03:02:15 -0700
    Ready:      False
    Restart Count:  23
    Readiness:      http-get http://:8080/env delay=20s timeout=1s period=10s #success=1 #failure=3
    Environment Variables:
Conditions:
  Type      Status
  Ready     False 
Volumes:
  spinnaker-config:
    Type:   Secret (a volume populated by a Secret)
    SecretName: spinnaker-config
  creds-config:
    Type:   Secret (a volume populated by a Secret)
    SecretName: creds-config
  aws-config:
    Type:   Secret (a volume populated by a Secret)
    SecretName: aws-config
  default-token-6irrl:
    Type:   Secret (a volume populated by a Secret)
    SecretName: default-token-6irrl
Events:
  FirstSeen LastSeen    Count   From            SubobjectPath           Type        Reason      Message
  --------- --------    -----   ----            -------------           --------    ------      -------
  1h        3m      24  {kubelet 172.25.30.21}  spec.containers{front50}    Normal      Pulling     pulling image "quay.io/spinnaker/front50:latest"
  1h        3m      24  {kubelet 172.25.30.21}  spec.containers{front50}    Normal      Pulled      Successfully pulled image "quay.io/spinnaker/front50:latest"
  1h        3m      15  {kubelet 172.25.30.21}  spec.containers{front50}    Normal      Created     (events with common reason combined)
  1h        3m      15  {kubelet 172.25.30.21}  spec.containers{front50}    Normal      Started     (events with common reason combined)
  1h        <invalid>   443 {kubelet 172.25.30.21}  spec.containers{front50}    Warning     BackOff     Back-off restarting failed docker container
  1h        <invalid>   417 {kubelet 172.25.30.21}                  Warning     FailedSync  Error syncing pod, skipping: failed to "StartContainer" for "front50" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=front50 pod=spin-front50-v000-v1g6e_spinnaker(edf85f41-7e3d-11e6-ab16-0022195df673)"

spin-gate-v000-9k401

root@n42-poweredge-5:~# kubectl describe pod spin-gate-v000-9k401 --namespace=spinnaker
Name:       spin-gate-v000-9k401
Namespace:  spinnaker
Node:       172.25.30.21/172.25.30.21
Start Time: Mon, 19 Sep 2016 00:53:00 -0700
Labels:     load-balancer-spin-gate=true,replication-controller=spin-gate-v000
Status:     Running
IP:     172.16.33.6
Controllers:    ReplicationController/spin-gate-v000
Containers:
  gate:
    Container ID:   docker://7507c9d7c00e5834572cde2c0b0b54086288e9e30d3af161f0a1dbdf44672332
    Image:      quay.io/spinnaker/gate:latest
    Image ID:       docker://sha256:074d9616a43de8690c0a6a00345e422c903344f6876d9886f7357505082d06c7
    Port:       8084/TCP
    QoS Tier:
      memory:       BestEffort
      cpu:      BestEffort
    State:      Running
      Started:      Mon, 19 Sep 2016 01:14:54 -0700
    Ready:      False
    Restart Count:  0
    Readiness:      http-get http://:8084/env delay=20s timeout=1s period=10s #success=1 #failure=3
    Environment Variables:
Conditions:
  Type      Status
  Ready     False 
Volumes:
  spinnaker-config:
    Type:   Secret (a volume populated by a Secret)
    SecretName: spinnaker-config
  default-token-6irrl:
    Type:   Secret (a volume populated by a Secret)
    SecretName: default-token-6irrl
Events:
  FirstSeen LastSeen    Count   From            SubobjectPath       Type        Reason      Message
  --------- --------    -----   ----            -------------       --------    ------      -------
  1h        <invalid>   696 {kubelet 172.25.30.21}  spec.containers{gate}   Warning     Unhealthy   Readiness probe failed: Get http://172.16.33.6:8084/env: dial tcp 172.16.33.6:8084: connection refused

what's wrong here?

UPDATE1

Logs (Please check the logs here)

2016-09-20 06:49:45.062 ERROR 1 --- [           main] o.s.boot.SpringApplication               : Application startup failed

org.springframework.context.ApplicationContextException: Unable to start embedded container; nested exception is org.springframework.boot.context.embedded.EmbeddedServletContainerException: Unable to start embedded Tomcat
    at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.onRefresh(EmbeddedWebApplicationContext.java:133)
    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:532)
    at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.refresh(EmbeddedWebApplicationContext.java:118)
    at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:690)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:322)
    at org.springframework.boot.builder.SpringApplicationBuilder.run(SpringApplicationBuilder.java:134)
    at org.springframework.boot.builder.SpringApplicationBuilder$run$0.call(Unknown Source)
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
    at com.netflix.spinnaker.front50.Main.main(Main.groovy:47)
Caused by: org.springframework.boot.context.embedded.EmbeddedServletContainerException: Unable to start embedded Tomcat
    at org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedServletContainer.initialize(TomcatEmbeddedServletContainer.java:99)
    at org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedServletContainer.<init>(TomcatEmbeddedServletContainer.java:76)
    at org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedServletContainerFactory.getTomcatEmbeddedServletContainer(TomcatEmbeddedServletContainerFactory.java:384)
    at org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedServletContainerFactory.getEmbeddedServletContainer(TomcatEmbeddedServletContainerFactory.java:156)
    at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.createEmbeddedServletContainer(EmbeddedWebApplicationContext.java:159)
    at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.onRefresh(EmbeddedWebApplicationContext.java:130)
    ... 10 common frames omitted
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.springframework.security.config.annotation.web.configuration.WebSecurityConfiguration'........
..........

UPDATE-1(02-06-2017)

I tried above setup again in latest version of K8

Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.4", GitCommit:"d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae", GitTreeState:"clean", BuildDate:"2017-05-19T18:44:27Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.4", GitCommit:"d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae", GitTreeState:"clean", BuildDate:"2017-05-19T18:33:17Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}

Still not all PODs are up

ubuntu@ip-172-31-18-78:~/spinnaker/experimental/kubernetes/simple$ kubectl get pods --namespace=spinnaker
NAME                           READY     STATUS             RESTARTS   AGE
data-redis-master-v000-rzmzq   1/1       Running            0          31m
spin-clouddriver-v000-qhz97    1/1       Running            0          31m
spin-deck-v000-0sz8q           1/1       Running            0          31m
spin-echo-v000-q9xv5           1/1       Running            0          31m
spin-front50-v000-646vg        0/1       CrashLoopBackOff   10         31m
spin-gate-v000-vfvhg           0/1       Running            0          31m
spin-igor-v000-8j4r0           1/1       Running            0          31m
spin-orca-v000-ndpcx           0/1       CrashLoopBackOff   9          31m

Here is the logs links

Front50 https://pastebin.com/ge5TR4eR

Orca https://pastebin.com/wStmBtst

Gate https://pastebin.com/T8vjqL2K

Deck https://pastebin.com/kZnzN62W

Clouddriver https://pastebin.com/1pEU6V5D

Echo https://pastebin.com/cvJ4dVta

Igor https://pastebin.com/QYkHBxkr

Did I miss any configuration? I have not touched yaml config(Updated Jenkins URL,uname, passwd), that's I'm getting errors?. I'm new to Spinnaker. I had little knowledge on normal Spinnaker installation. Please guide me the installation.

Thanks

Akanke answered 19/9, 2016 at 10:20 Comment(7)
First thing to check are the logs - what does bash scripts/logs.sh front50 show during startup?Aye
@Aye please check the update and log file linkAkanke
Have you configured S3 or GCS as a data store for front50? The reason front50 can't start is because it can't seem to communicate with either. It's probably a permissions issue.Aye
@Aye Ok, I configured S3 in spinnaker-local.yml. front50 is up, but still orca is down. I have checked the logs (docs.google.com/document/d/…) And where can I get the Spinnaker with Kuberneties deployement flow like AWS==> files.readme.io/06CDzZS2RhmmJ0PaMFgy_flow.pngAkanke
That's how the log ends? "Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.springframework.security.config.annotation.web.configuration.WebSecurityConfiguration'........". Please provide the rest if possibleFlunkey
@PiotrR Sorry I completly removed the setup. I guess, those are the complete logs at that time.Akanke
Hey looks like your question may have been answered in slack. In any case, the above scripts are being deprecated, this is the way to install Spinnaker from now on: spinnaker.github.io/setup/installAye
D
1

Based on these errors, it seems that the Front50 service cannot reach any backend(if there is one configured). I would not modify the spinnaker-local.yml file directly, but instead use Halyard to install Spinnaker services into Kubernetes.

I have setup Spinnaker services in Kubernetes successfully using the instructions and scripts in this repo. Just omit/skip the components that you don't need.

https://github.com/grizzthedj/kubernetes-spinnaker

Drawbridge answered 19/7, 2019 at 19:3 Comment(0)
C
1

use halyard to install spinnaker. it is the recommended approach for deploying spinnaker in kubernetes clsuter

Chasse answered 27/6, 2019 at 7:24 Comment(0)
D
1

Based on these errors, it seems that the Front50 service cannot reach any backend(if there is one configured). I would not modify the spinnaker-local.yml file directly, but instead use Halyard to install Spinnaker services into Kubernetes.

I have setup Spinnaker services in Kubernetes successfully using the instructions and scripts in this repo. Just omit/skip the components that you don't need.

https://github.com/grizzthedj/kubernetes-spinnaker

Drawbridge answered 19/7, 2019 at 19:3 Comment(0)
A
1

This question is very old. At the time I started working on Spinnaker, there was no Halyard or proper document on how to setup spinnaker on K8s. Since I see some attention getting on this question, I just want to provide some resource & tools to quickly setup.

Akanke answered 2/8, 2019 at 11:25 Comment(0)
R
0

Might be the coredns issue. Orca mainly deals with establishing communication across the services in the namespace.

Try getting into some running pod (or use a busybox pod) to nslookup the spin-* services. Check the logs of coredns pods. Try restarting coredns pods (last option)

Renault answered 24/1, 2020 at 12:8 Comment(0)

© 2022 - 2024 — McMap. All rights reserved.