Most CRUD APIs follow the shape of:

  • Create
  • Retrieve
  • Update
  • Delete
  • List

These work fine, but if you have lots of requests per second, then batching (bulk APIs) could help.

There are essentially two types of bulk APIs:

  • Request/response style – get a result back quickly with what succeeded / failed
  • Async job – kick off a job and poll for status

In the examples below, most of the APIs are of the request/response style.

Examples

SQL

insert into

insert into products (product_no, name, price) values
  (1, 'Cheese', 9.99),
  (2, 'Bread', 1.99),
  (3, 'Milk', 2.99);

update

update films set kind = 'Dramatic' where kind = 'Drama';

Mongo Bulk API

The bulk write provides extensive flexiblity for batch changes via its operations: insertOne, updateOne, updateMany, deleteOne, deleteMany, replaceOne

It also provides a detailed result when it returns.

const result = await collection.bulkWrite([
  {
    insertOne: {
      document: {
        foo: bar
      }
    }
  },
  {
    updateOne: {
      filter: { someField: value },
      update: { $set: { updatedField: "newValue" } }
    }
  },
  {
    updateMany: {
      filter: { someField: value },
      update: { $set: { otherField: true } }
    }
  },
  {
    deleteOne: {
      filter: { someDifferentField: anotherValue }
    }
  },
  {
    deleteMany: {
      filter: { yetAnotherField: anotherValue }
    }
  },
  {
    replaceOne: {
      filter: { someReplaceField: someValue },
      replacement: { newField: "newValue" }
    }
  }
])

Algolia

Their search api has batching built in:

const searchResults = await client.search({
  requests: [
    { indexName, query },
    { indexName: index2, query: query2 }
  ]
})

S3 API

S3 has a few bulk APIs:

  • The delete_objects API supports bulk deleting up to 1000 objects at a time.

  • The batch operations API utilizes a CSV to target specific objects, providing even greater batch sizes.

  • The storage inventory api supports generating a CSV, ORC, or Parquet file of your S3 objects metadata, rather than having to list them manually.

Facebook

The graph API supports batching requests:

curl -i -X POST 'https://graph.facebook.com/me?batch=
  [
    {
      "method":"GET",
      "relative_url":"PAGE-A-ID"
    },
    {
      "method":"GET",
      "relative_url":"PAGE-B-ID"
    }
  ]
  &access_token=ACCESS-TOKEN'

They also support async and batch requests for their marketing APIs.

Tiger Beetle

All of their APIs support batch operations out of the box, e.g.

await client.createAccounts([account0, account1])

Salesforce

Supports bulk CRUD operations via an async job style bulk API.

Cloudflare

Supports creating a batch of redirects.

Google Analytics

Support for creating a batch of reports.

Mixpanel

Supports creating a batch of events.

Amplitude

Supports creating a batch of events.

Conclusion

Batch APIs allow for more efficent API calls and simplier integrations.