Google Apps Scripts — parallel HTTP requests with retries

While writing Google Data Studio connectors I needed to fire lots and lots of HTTP requests using UrlFetchApp in order to gather all the necessary data for my reports. Initially I used fetch method that allows only one, synchronous request.
Later, when my connector was ready I wanted to speed things up and started experimenting with fetchAll method, capable of sending multiple parallel requests. It really gave my app a boost, but I still needed to tackle one more problem…

Sometimes a single request in the requests batch would fail causing the whole script to fail as well. In many cases the execution would be saved by simply retrying the failed requests in the batch. I didn’t find any good solutions on the Internet, so decided to write my own retriable requests batch service.

The solution

The RetriableRequestsBatch service can be described by a following chart:

So: if errors occur in the requests batch we check if they can be retried. If all of the errors are retriable (response contains one of errors defined in RETRIABLE_ERROR_STRINGS) then we retry the failed requests and repeat the checks. If number of retries exceeds 3 then we return an error result (the response of first errored request). If errors are not retriable then we return error result immediately. If all the requests succeed we return their responses.

Check out the solution’s code:

Now all you need to do is simply add the file to your script and call e.g.:

var request1 = {
  'url': 'https://httpbin.org/post',
  'method' : 'post'
};
var request2 = {
  'url': 'https://httpbin.org/post',
  'method' : 'post'
};
var result = new RetriableRequestsBatch(UrlFetchApp, [request1, request2]).fetchWithRetries();
if (result.error) {
  console.error('Error happened when fetching batch:', result.error);
  return;
}
console.log('Succesfuly fetched requests batch. Responses: result.responses);

Summary

Hope this tutorial will save you some time. If you’ve already solved this problem in a different way (or have any concerns regarding mine) don’t hesitate to post it in the comment. Keep on coding :)