How to pass parameters to Google Cloud Run Job

Question

I have ran this example using this job

Everything worked well.

Now I am trying to see if there is a way to pass parameters to jobs running on Cloud Run.

I understand I can use the command to create jobs with a --message-body argument like this:

 gcloud scheduler jobs create http JOB_NAME \
  --location REGION \
  --schedule="*/3 * * * *" \
  --uri="https://REGION-run.googleapis.com/apis/run.googleapis.com/v1/namespaces/PROJECT_ID/jobs/CLOUD_RUN_JOB_NAME:run" \
  --http-method POST \
  --oauth-service-account-email  [email protected]
  --message-body="This is the body"

However while checking the documentation for Cloud Run jobs here.

I don't see parameters being mentioned anywhere. The idea is that depending on a JSON that contains the parameters we can run different kind of jobs (it's a same job that changes its operation based on the parameters)

Titbit

I have not tested this but I believe you can pass parameters via message-body. For example, create a JSON object, convert to a string and then base64 encode for the body. On the receiving side, you would grab the POST message body, base64decode ... – Mountain

You can't (and I begged the Cloud Run Job PM to implement something, and, obviously, the other alpha testers had the same issue and something will happen :) ). My current solution is to wrap the current batch job in a web server and, as @JohnHanley said, provide a body, parse it, and invoque your batch with that input. – Armyn

Hello Guillaume, how would one go about wrapping the job in a web server? – Titbit

I suggest that you create a feature request to pass parameters from Cloud scheduler to Cloud Run Job. Please use this link for your reference. – Mustee

Hello @guillaumeblaquiere can you post the comment as an answer so I can accept it? – Titbit

@guillaumeblaquiere sorry but what do you mean by “invoque your batch”?, how can I invoke the job with that parsed JSON?, do I need to create a scheduler with that message body?, or does the job execute API support a body? – Clericalism

Answer 1

With Cloud Run Jobs, you can't provide parameters (entry points, args or env vars). I discussed that point with the Cloud Run Job PM to implement something, and, obviously, the other alpha testers had the same issue and something will happen :).

My current solution is to wrap the current batch job in a web server. Package that webserver in a container and deploy it on Cloud Run. Then, provide the correct body, with your parameters, parse it, and invoke your batch with those inputs

Armyn

What if I change my API from GET to POST? Move all query parameters to body, is it possible to fix this issue? – Albuminate

Why do you want to change from get to post? ANd you can let query parameters even if you perform a post request – Armyn

What's the point of the jobs then? For example we can't use the Google Video Intelligence API in Cloud Run because of timeouts. – Cladding

Things are coming! Stay tuned!! – Armyn

Cloud Run Jobs now allow you to set env vars on the job itself, and override env vars within the context of each execution. And it's easy to do via the CLI --set-env-vars on the create and execute subcommand, so easy to script. cloud.google.com/run/docs/execute/jobs – Mispickel

@guillaumeblaquiere Do you know if we can use arguments with the -- sign? I mean, the prompt --args --arg1,a,--arg2,b follows with an error because it sees --arg1 as the next argument. It doesn't response with an error message when I define all the args as string (""), but then in yaml job configuration, all the arguments are treated as a single argument. – Panay

Answer 2

This is now possible using the REST API for Cloud Run Jobs.

• The REST API allows passing in a request body
• The request body accepts an override defnition
• To make the call, you'll need to get the token of the current executing identity

A full working repo of how to do this is here: https://github.com/CharlieDigital/gcr-invoke-job-overrides

A more in depth writeup here: https://medium.com/@chrlschn/programmatically-invoke-cloud-run-jobs-with-runtime-overrides-4de96cbd158c

The sample is in C#/.NET, but it's easy enough to translate to any other language.

Here's the log output of the environment variables and entry point args:

Faso

Answer 3

[UPDATE] Aug 08 2023, this feature (If I'm reading right) is now in preview

Klump

This feature isn't available in the client libraries yet, but is available in the REST APIs for anyone that needs it: cloud.google.com/run/docs/reference/rest/v1/namespaces.jobs/run – Faso

Born too early to explore space, born too late to explore earth, born just in time to use arguments with google cloud jobs – Postmeridian

Answer 4

You can pass arguments via the args flag.

They provide an example in this codelabs course.

gcloud beta run jobs create screenshot \
    --image=$REGION-docker.pkg.dev/$PROJECT_ID/containers/screenshot:v1 \
    --args="https://example.com" \
    --args="https://cloud.google.com" \
    --tasks=2 \
    --task-timeout=5m \
    --region=$REGION \
    --set-env-vars=BUCKET_NAME=screenshot-$PROJECT_ID \
    --service-account=screenshot-sa@$PROJECT_ID.iam.gserviceaccount.com

This passes an array of args when creating the job and access the right argument by using the CLOUD_RUN_TASK_INDEX environment variable.

Redpencil

I don't think this is what's being asked for. This creates a job with arguments, it doesn't allow those arguments to be varied each time you execute that job. – Tamberg

I do think it solves the problem. You pass an array of args when creating the job and access the right argument by using the CLOUD_RUN_TASK_INDEX variable. – Redpencil

Forgive me David, I'd not understood that the task index variable could be specified at runtime. Whilst not fully dynamic, this can meet a number of peoples uses. Downvote->upvote :) – Tamberg

how to pass the CLOUD_RUN_TASK_INDEX variable during runtime @DavidSpiess? – Beaman

CLOUD_RUN_TASK_INDEX is available as environment variable @Beaman – Redpencil

Answer 5

For those using .NET libraries (Google.Cloud.Run.V2) to execute Cloud Run Jobs and want to pass env and args variables to the jobs you can use container override.

    // Create client
    Google.Cloud.Run.V2.JobsClient jobsClient = Google.Cloud.Run.V2.JobsClient.Create();

    // Initialize request 
    var request = new Google.Cloud.Run.V2.RunJobRequest
    {
        JobName = Google.Cloud.Run.V2.JobName.FromProjectLocationJob("[ProjectID]", "[Location]", "[JobName]"),
        ValidateOnly = false,
        Etag = "",
        Overrides = new Google.Cloud.Run.V2.RunJobRequest.Types.Overrides(),
    };
    
    // HERE STARTS: Create a Container Override
    
    var containerModificado = new Overrides.Types.ContainerOverride();
    
    // Create Enviroment Variables

    var enviros = new Google.Cloud.Run.V2.EnvVar();
    enviros.Name = "SALUDO";
    enviros.Value = "HOLA";

    // Add Env Variables to Container

    containerModificado.Env.Add(enviros);

    // Create and add Args Arguments to container
    
    containerModificado.Args.Add("CHIDO1");
    containerModificado.Args.Add("CHIDO2");
    
    // Finally we add the container to Job Request

    request.Overrides.ContainerOverrides.Add(containerModificado);

    // THEN Make the request
    var response = jobsClient.RunJob(request);

    // Poll until the returned long-running operation is complete
    var completedResponse = response.PollUntilCompleted();
    // Retrieve the operation result
    var result = completedResponse.Result;

Purdum

I tried this on my node cloud run job and I get this issue: #78280572 – Pickel