Skaffold syncs files but pod doesn't refresh

Question

Kubernetes newbie here.

I have some strange Skaffold/Kubernetes behavior. I'm working in Google Cloud but I've changed to the local environment just for test and it's the same. So probably it's me how's doing something wrong. The problem is that though I see Skaffold syncing changes these changes aren't reflected. All the files inside the pods are the old ones.

Skaffold.yaml:

apiVersion: skaffold/v2alpha3
kind: Config
deploy:
  kubectl:
    manifests:
      - ./infra/k8s/*
build:
  # local:
  #   push: false
  googleCloudBuild:
    projectId: ts-maps-286111
  artifacts:
    - image: us.gcr.io/ts-maps-286111/auth
      context: auth
      docker:
        dockerfile: Dockerfile
      sync:
        manual:
          - src: 'src/**/*.ts'
            dest: .
    - image: us.gcr.io/ts-maps-286111/client
      context: client
      docker:
        dockerfile: Dockerfile
      sync:
        manual:
          - src: '**/*.js'
            dest: .
    - image: us.gcr.io/ts-maps-286111/tickets
      context: tickets
      docker:
        dockerfile: Dockerfile
      sync:
        manual:
          - src: 'src/**/*.ts'
            dest: .
    - image: us.gcr.io/ts-maps-286111/orders
      context: orders
      docker:
        dockerfile: Dockerfile
      sync:
        manual:
          - src: 'src/**/*.ts'
            dest: .
    - image: us.gcr.io/ts-maps-286111/expiration
      context: expiration
      docker:
        dockerfile: Dockerfile
      sync:
        manual:
          - src: 'src/**/*.ts'
            dest: .

When a file inside one of the directories is changed I see following logs:

time="2020-09-05T01:24:06+03:00" level=debug msg="Change detected notify.Write: \"F:\\projects\\lrn_microservices\\complex\\expiration\\src\\index.ts\""
time="2020-09-05T01:24:06+03:00" level=debug msg="Change detected notify.Write: \"F:\\projects\\lrn_microservices\\complex\\expiration\\src\\index.ts\""
time="2020-09-05T01:24:06+03:00" level=debug msg="Change detected notify.Write: \"F:\\projects\\lrn_microservices\\complex\\expiration\\src\\index.ts\""
time="2020-09-05T01:24:06+03:00" level=debug msg="Change detected notify.Write: \"F:\\projects\\lrn_microservices\\complex\\expiration\\src\\index.ts\""
time="2020-09-05T01:24:06+03:00" level=debug msg="Change detected notify.Write: \"F:\\projects\\lrn_microservices\\complex\\expiration\\src\\index.ts\""
time="2020-09-05T01:24:06+03:00" level=debug msg="Change detected notify.Write: \"F:\\projects\\lrn_microservices\\complex\\expiration\\src\\index.ts\""
time="2020-09-05T01:24:07+03:00" level=debug msg="Found dependencies for dockerfile: [{package.json /app true} {. /app true}]"
time="2020-09-05T01:24:07+03:00" level=debug msg="Skipping excluded path: node_modules"
time="2020-09-05T01:24:07+03:00" level=debug msg="Found dependencies for dockerfile: [{package.json /app true} {. /app true}]"
time="2020-09-05T01:24:07+03:00" level=debug msg="Skipping excluded path: .next"
time="2020-09-05T01:24:07+03:00" level=debug msg="Skipping excluded path: node_modules"
time="2020-09-05T01:24:07+03:00" level=debug msg="Found dependencies for dockerfile: [{package.json /app true} {. /app true}]"
time="2020-09-05T01:24:07+03:00" level=debug msg="Skipping excluded path: node_modules"
time="2020-09-05T01:24:07+03:00" level=debug msg="Found dependencies for dockerfile: [{package.json /app true} {. /app true}]"
time="2020-09-05T01:24:07+03:00" level=debug msg="Skipping excluded path: node_modules"
time="2020-09-05T01:24:07+03:00" level=debug msg="Found dependencies for dockerfile: [{package.json /app true} {. /app true}]"
time="2020-09-05T01:24:07+03:00" level=debug msg="Skipping excluded path: node_modules"
time="2020-09-05T01:24:07+03:00" level=info msg="files modified: [expiration\\src\\index.ts]"
Syncing 1 files for us.gcr.io/ts-maps-286111/expiration:2aae7ff-dirty@sha256:2e31caedf3d9b2bcb2ea5693f8e22478a9d6caa21d1a478df5ff8ebcf562573e
time="2020-09-05T01:24:07+03:00" level=info msg="Copying files: map[expiration\\src\\index.ts:[/app/src/index.ts]] to us.gcr.io/ts-maps-286111/expiration:2aae7ff-dirty@sha256:2e31caedf3d9b2bcb2ea5693f8e22478a9d6caa21d1a478df5ff8ebcf562573e"
time="2020-09-05T01:24:07+03:00" level=debug msg="getting client config for kubeContext: ``"
time="2020-09-05T01:24:07+03:00" level=debug msg="Running command: [kubectl --context gke_ts-maps-286111_europe-west3-a_ticketing-dev exec expiration-depl-5cb997d597-p49lv --namespace default -c expiration -i -- tar xmf - -C / --no-same-owner]"
time="2020-09-05T01:24:09+03:00" level=debug msg="Command output: [], stderr: tar: removing leading '/' from member names\n"
Watching for changes...
time="2020-09-05T01:24:11+03:00" level=info msg="Streaming logs from pod: expiration-depl-5cb997d597-p49lv container: expiration"
time="2020-09-05T01:24:11+03:00" level=debug msg="Running command: [kubectl --context gke_ts-maps-286111_europe-west3-a_ticketing-dev logs --since=114s -f expiration-depl-5cb997d597-p49lv -c expiration --namespace default]"
[expiration]
[expiration] > [email protected] start /app
[expiration] > ts-node-dev --watch src src/index.ts
[expiration]
[expiration] ts-node-dev ver. 1.0.0-pre.62 (using ts-node ver. 8.10.2, typescript ver. 3.9.7)
[expiration] starting expiration!kdd
[expiration] Connected to NATS!

NodeJS server inside the pod restarts. Sometimes I see this line, sometimes not, the result overall is the same

[expiration] [INFO] 22:23:42 Restarting: src/index.ts has been modified

But there are no changes made. If I cat the changed file inside a pod it's the old version, if I delete a pod it starts again with an old version.

My folder structure:

+---auth
|   \---src
|       +---models
|       +---routes
|       |   \---__test__
|       +---services
|       \---test
+---client
|   +---.next
|   |   +---cache
|   |   |   \---next-babel-loader
|   |   +---server
|   |   |   \---pages
|   |   |       +---auth
|   |   |       \---next
|   |   |           \---dist
|   |   |               \---pages
|   |   \---static
|   |       +---chunks
|   |       |   \---pages
|   |       |       +---auth
|   |       |       \---next
|   |       |           \---dist
|   |       |               \---pages
|   |       +---development
|   |       \---webpack
|   |           \---pages
|   |               \---auth
|   +---api
|   +---components
|   +---hooks
|   \---pages
|       \---auth
+---common
|   +---build
|   |   +---errors
|   |   +---events
|   |   |   \---types
|   |   \---middlewares
|   \---src
|       +---errors
|       +---events
|       |   \---types
|       \---middlewares
+---config
+---expiration
|   \---src
|       +---events
|       |   +---listeners
|       |   \---publishers
|       +---queue
|       \---__mocks__
+---infra
|   \---k8s
+---orders
|   \---src
|       +---events
|       |   +---listeners
|       |   |   \---__test__
|       |   \---publishers
|       +---models
|       +---routes
|       |   \---__test__
|       +---test
|       \---__mocks__
+---payment
\---tickets
    \---src
        +---events
        |   +---listeners
        |   |   \---__test__
        |   \---publishers
        +---models
        |   \---__test__
        +---routes
        |   \---__test__
        +---test
        \---__mocks__

Would be grateful for any help!

Dato

You could try running ts-node-dev --debug to show more output. Is it possible that your app is changing the current working directory outside of /app? It would help if you included your expiration/Dockerfile and showed how you checked that are no code changes. – Lifeboat

Answer 1

What worked for me was using --poll flag with ts-node-dev. My script looks like this

"start" : "ts-node-dev --respawn --poll --inspect --exit-child src/index.ts

Mcmillan

Answer 2

For your start script try to add --poll. For example if you start script is "start" : "nodemon src/index.js", change it to "nodemon --poll src/index.js"

Hollo

can you explain why? what does the --pull attr do? – Drugstore

Answer 3

It looks like no major error in the logs, my guess is that the files are actually being put in another directory. You can try running in the container.

find / -name "index.ts"

to see if that didn't land somewhere else.

Another thing to check is the WORKDIR value in your container(s). Check what directory you land on when you run:

kubectl exec -it -c <container-name> <pod-name>

✌️

Acadia