Gcloud - cannot provision many VM using one Service Account

Question

I am using Gcloud to run Prow (Continous Integration server). One of my job creates a virtual machine, perform some tests and then delete that instance. I use a service account to create VM, run tests.

#!/bin/bash

set -o errexit

cleanup() {
    gcloud compute instances delete kyma-integration-test-${RANDOM_ID}
}


gcloud config set project ...
gcloud auth activate-service-account --key-file ...

gcloud compute instances create <vm_name> \
    --metadata enable-oslogin=TRUE \
    --image debian-9-stretch-v20181009 \
    --image-project debian-cloud --machine-type n1-standard-4 --boot-disk-size 20 \

trap cleanup exit

gcloud compute scp --strict-host-key-checking=no --quiet <script.sh> <vm_name>:~/<script.sh>

gcloud compute ssh --quiet <vm_name> -- ./<script.sh>

After some time, I got following error:

ERROR: (gcloud.compute.scp) INVALID_ARGUMENT: Login profile size exceeds 32 KiB. Delete profile values to make additional space.

Indeed, for that service account, describe command returns a lot of data, for example ~70 entries in sshPublicKeys section.

gcloud auth activate-service-account --key-file ... gcloud compute os-login describe-profile

Most of this public keys refer to already removed VM instances. How to perform cleanup of this list? Or is it possible to not store that public keys at all?

Yielding

I would not use scp to copy files to the instance nor would I connect using ssh to run a script. Put files for your instance on Cloud Storage and then use the metadata startup script to download the file(s) and run your script. – Hanaper

Answer 1

The permanent solution is to use --ssh-key-expire-after 30s. You still need to cleanup the existing keys with the solutions above or a little more command kungfu like this (without grep).

for i in $(gcloud compute os-login ssh-keys list --format="table[no-heading](value.fingerprint)"); do 
  echo $i; 
  gcloud compute os-login ssh-keys remove --key $i || true; 
done

NOTE: you have to be using the offending account. gcloud config account activate ACCOUNT and/or gcloud auth activate-service-account --key-file=FILE or gcloud auth login

Need a new ssh key in a script:

# KEYNAME should be something like $HOME/.ssh/google_compute_engine
ssh-keygen -t rsa -N "" -f "${KEYNAME}" -C "${USERNAME}" || true
chmod 400 ${KEYNAME}*

cat > ssh-keys <<EOF
${USERNAME}:$(cat ${KEYNAME}.pub)
EOF

Testing this solution:

while :; do
  [email protected]
  KEYNAME=~/.ssh/google_compute_engine
  rm -f ~/.ssh/google_compute_engine*
  ssh-keygen -t rsa -N "" -f "${KEYNAME}" -C "${USERNAME}" || true
  chmod 400 ${KEYNAME}*
  cat > ssh-keys <<EOF
  ${USERNAME}:$(cat ${KEYNAME}.pub)
EOF
  gcloud --project=test-project compute ssh --ssh-key-expire-after 30s one-bastion-to-rule-them-all -- date
  gcloud --project=test-project compute os-login ssh-keys list --format="table[no-heading](value.fingerprint)" \
    |wc -l
done

Wheat

Doesn’t work for me as expired keys are never removed from the login profile – Peevish

Answer 2

A very crude way to do the above that worked for me was:

for i in $(gcloud compute os-login ssh-keys list); do echo $i; gcloud compute os-login ssh-keys remove --key $i; done

I stopped this (with Control-C) after deleting a few tens of keys and then it worked again.

Actually, in the project metadata in the GUI, I do not see a lot of key. Only :

• gke...cidr : network-name...
• sshKeys : gke-e9...
• SSH Keys => peter_v : ssh-rsa my public key

Lemoine

Thanks, this command worked perfectly using gcloud version Google Cloud SDK 331.0.0 alpha 2021.03.05 beta 2021.03.05 bq 2.0.65 core 2021.03.05 gsutil 4.59 – Sapless

Answer 3

In my case, I was using another service account to run ssh, so basically I'm using a impersonate. If you are using an impersonation too, you need to delete the ssh key list from the service account which you're impersonating.

for i in $(gcloud compute os-login ssh-keys list --impersonate-service-account="[email protected]" --format="table[no-heading](value.fingerprint)");
        do echo $i;
     gcloud compute os-login ssh-keys remove --key $i --impersonate-service-account="[email protected]" || true;
done

And then add "--ssh-key-expire-after=7m" the amount of time is defined by your needs

gcloud compute ssh ${MY_VM} --zone ${GKE_ZONE} --project ${PROJECT_ID} --tunnel-through-iap --ssh-key-expire-after=7m --impersonate-service-account="[email protected]"

Perfusion

Answer 4

These key are stored in your Project Metadata yo can remove them by deleting trough the Google Console UI

Overstate

Answer 5

Seeing as you were mentioning OS Login in your question: there is a way to delete specific SSH keys from a user's profile using this command. Alternatively, instead of performing SCP, I'd advise you, much like John Hanley has, to put the file you're copying into the instance in Storage and retrieve it via a startup script (you could also use a custom Compute image).

Cummins