Syncing Git repo to Google Cloud

Question

So suppose I have a git repository https://github.com/jc/ and I have a location for the google bucket gs://acme-sales/.

Is there a way to write a python program which updates the changes which have been made in github and sync them to google cloud each time we run it?

I suppose we have to use gitpython to read the file from github link but how do I just keep updating the files to google bucket.

Guideline

One possibility would be to setup a CI/CD pipeline, automatically triggered by commits to the repo and which would just pull the repo content and use, as a build command, a wrapper (python if you want) script around the gsutil rsync cloud SDK utility to sync the repo content to the bucket. – Japhetic

Answer 1

If you don't need the sync to be immediate (i.e., w/in seconds), you could set up a cron job to periodically pull down the .zip archive of your repo, and upload it to Google Cloud Storage.

In your app.yaml:

runtime: python37

In your cron.yaml:

cron:
- description: "sync from git repo"
  url: /tasks/sync
  schedule: every 1 minute

In your main.py:

from urllib.request import urlopen
from zipfile import ZipFile

from flask import Flask
from google.cloud import storage

app = Flask(__name__)

client = storage.Client(project='your-project-name')
bucket = client.get_bucket('your-bucket-name')

# Path to the archive of your repository's master branch
repo = 'https://github.com/your-username/your-repo-name/archive/master.zip'

@app.route('/tasks/sync')
def sync():
    with ZipFile(BytesIO(urlopen(repo).read())) as zipfile:
        for filename in zipfile.namelist():
            blob = storage.Blob(filename, bucket)
            blob.upload_from_string(zipfile.read(filename))

Deploy with:

$ gcloud app deploy
$ gcloud app deploy cron.yaml

Indefensible

Answer 2

I recommend you to use Google Cloud Build. It lets you syncing your repo and using gsutil you can automatically update your storage i.e gs://acme-sales on each commits to your GitHub repository:

steps:
- name: gcr.io/cloud-builders/gsutil
  args: ['cp', '/workspace', 'gs://acme-sales']

You may start on GitHub by connecting your repository i.e. https://github.com/jc/<repo_name> to Google Cloud Build using the link below:

https://github.com/apps/google-cloud-build

Symmetrical

Answer 3

Adding to the previous answer[s].

You can an option to "Automatically Mirror from GitHub or Bitbucket", which is self descriptive. Or you can use an "Automatic Build Trigger" directly with custom build steps to perform certain actions.

Hallett