How to send multipart/mixed request for google indexing batch request in NodeJs?

Question

I am using Nodejs to connect with GoogleApis v35.0.0 to tell Google to update or remove pages from the Google index. And I stuck in the multipart/mixed request, the body of multipart when I send the request through Google indexing batch request.

I could able to send an individual page update request to Google by following the indexing API documentation. But since Google has the limited quota at maximum of 200 requests per day and I need to update more URL's than that. So, I am trying to use google indexing batch request which can group at maximum of 100 individual requests and it counts as 1 request.

I am having issue with the correct format of multipart body when I am trying to send the request by batch. I am using JWT (JSON Web Token) of GoogleApis which extended from oauth2 to authenticate my account and using the request library v2.88.0 to send the request to Google.

Since request library already handle the multipart boundary that's why I am not sending that as one of the request options information. I also check the information in the multipart/mixed of the request npm library but I only found the one similar but not the same which is the multipart/related (https://github.com/request/request#multipartrelated).

According to the piece of batch request body example from Google, I need to use multipart/mixed as content type in the main request:

POST /batch HTTP/1.1
Host: indexing.googleapis.com
Content-Length: content_length
Content-Type: multipart/mixed; boundary="===============7330845974216740156=="
Authorization: Bearer oauth2_token

--===============7330845974216740156==
Content-Type: application/http
Content-Transfer-Encoding: binary
Content-ID: <b29c5de2-0db4-490b-b421-6a51b598bd22+2>

POST /v3/urlNotifications:publish [1]
Content-Type: application/json
accept: application/json
content-length: 58

{ "url": "http://example.com/jobs/42", "type": "URL_UPDATED" }

Here's my code:

    return jwtClient.authorize(function(err, tokens) {
      if (err) {
        console.log(err);
        return;
      }

      let options = {
        url: 'https://indexing.googleapis.com/batch',
        method: 'POST',
        headers: {
          'Content-Type': 'multipart/mixed'
        },
        auth: { 'bearer': tokens.access_token },
        multipart: [
          {
            body: JSON.stringify({
              headers: {
                'Content-Type': 'application/http'
              },
              method: 'POST',
              url: 'https://indexing.googleapis.com/v3/urlNotifications:publish',
              body: {
                'Content-Type': 'application/json',
                url: 'https://www.test.com/es/1234',
                type: 'URL_UPDATED'
              }
            })
          }
        ]
      };

      request(options, function (error, response, body) {
        console.log(body);
      });

    });

I am getting error in the body of multipart, I don't know which kind of body google indexing batch request is waiting for. Seems like everything inside the body of multipart are considering as headers. But according to the documentation the format of batch request, it says that "Each part begins with its own Content-Type: application/http HTTP header. The body of each part is itself a complete HTTP request, with its own verb, URL, headers, and body". For more details information check: https://cloud.google.com/storage/docs/json_api/v1/how-tos/batch.

However, I am getting the following error when I execute my code:

{
  "error": {
    "code": 400,
    "message": "Failed to parse batch request, error: Failed in parsing HTTP headers: {\"Content-Type\":\"application/http\",\"method\":\"POST\",\"url\":\"https://indexing.googleapis.com/v3/urlNotifications:publish\",\"body\":{\"Content-Type\":\"application/json\",\"url\":\"https://www.test.com/es/1234\",\"type\":\"URL_UPDATED\"}}\n. Received batch body: ",
    "status": "INVALID_ARGUMENT"
  }
}

Does someone know which is the correct format of body inside the multipart when it request to google indexing batch request?

Thanks in the advance!

Hellgrammite

Answer 1

As @DalmTo says the quota will still apply, even to batch requests. But also you are not correctly constructing the payload, the following example works.

const items = batch
  .filter(x => x)
  .map(line => {
    return {
      'Content-Type': 'application/http',
      'Content-ID': batchId,
      body:
        'POST /v3/urlNotifications:publish HTTP/1.1\n' +
        'Content-Type: application/json\n\n' +
        JSON.stringify({
          url: line,
          type: 'URL_UPDATED',
        }),
    };
  });
const options = {
  url: 'https://indexing.googleapis.com/batch',
  method: 'POST',
  headers: {
    'Content-Type': 'multipart/mixed',
  },
  auth: { bearer: access_token },
  multipart: items,
};
request(options, (err, resp, body) => {
  //...
});

Fairfax

thanks a lot to point out my issue. It works correctly now. Thanks again! – Hellgrammite

Where does the batch object come from? – Grodno

Answer 2

Batching does not help avoid quota limits

I could able to send an individual page update request to Google by following the indexing API documentation. But since Google has the limited quota at maximum of 200 requests per day and I need to update more URL's than that. So, I am trying to use google indexing batch request which can group at maximum of 100 individual requests and it counts as 1 request.

There is nothing in batching that states it only counts as one against your quota.

While batching can save you on the overhead of constructing many HTTP requests each Google APIs request within a batch request will count against your daily project quota. By default a project can make up to 200 request per day; batching will not help you stay below this quota.

Apply for a higher quota

Have you considered applying for a higher quota? I know it can take time to get the response back, but you may just want to wait and see what they say.

Note google-apis-nodejs-client

The library does not support batching so you are going to have to do it yourself as you are currently #1130

Your Actual issue

Let me know if you want to continue trying to get batching working. I will see if i can help. With manual version.

Conroy

Thanks @daimto. (I'm involved in the same project as @jing). Two follow-ups: (1) Correct, our objective with batching was to be able to surpass the default quota. (2) We began developing the batched requests because we reached a dead-end with the quota limit increase (we submitted a request for an increase through the process you indicate in your answer on Dec 17, 2018). If you know how to give a push for the team at Google to review our request approved, we'd greatly appreciate it (and likely save us a bunch of hassle! ;-) – Inoculation

Thanks @daimto for answering and clarifying. We'd definitely appreciate if you could help us with what Paulo aforementioned. Last thing, I would appreciate if you could give me advice to solve my current issue just for personal goal. – Hellgrammite

Have you tried to go over 200 since you applied? I don't have any contacts on that team unfortunately. It can take months to be approved and sometimes you just get approved without being notified you just need to try. – Conroy

Yes I tried, and get an error once it goes over 200 :( We will wait for Google to get the approval. According to my current issue with the manual version, do you know why I am getting the INVALID_ARGUMENT with the multipart/mixed request? Thanks @DaImTo! – Hellgrammite

I have never tried to get batching working manually i have always used one of the libraries that supports it. – Conroy

Okay, thanks @DaImTo! We just got the limit request approved today from Google. – Hellgrammite

I am glad i could help please remember to accept the anwser if it helped you – Conroy