Bulk APIs
Most CRUD APIs follow the shape of:
- Create
- Retrieve
- Update
- Delete
- List
These work fine, but if you have lots of requests per second, then batching (bulk APIs) could help.
There are essentially two types of bulk APIs:
- Request/response style – get a result back quickly with what succeeded / failed
- Async job – kick off a job and poll for status
In the examples below, most of the APIs are of the request/response style.
Examples
SQL
insert into products (product_no, name, price) values
(1, 'Cheese', 9.99),
(2, 'Bread', 1.99),
(3, 'Milk', 2.99);
update films set kind = 'Dramatic' where kind = 'Drama';
Mongo Bulk API
The bulk write provides extensive flexiblity for batch changes via its operations: insertOne
, updateOne
, updateMany
, deleteOne
, deleteMany
, replaceOne
It also provides a detailed result when it returns.
const result = await collection.bulkWrite([
{
insertOne: {
document: {
foo: bar
}
}
},
{
updateOne: {
filter: { someField: value },
update: { $set: { updatedField: "newValue" } }
}
},
{
updateMany: {
filter: { someField: value },
update: { $set: { otherField: true } }
}
},
{
deleteOne: {
filter: { someDifferentField: anotherValue }
}
},
{
deleteMany: {
filter: { yetAnotherField: anotherValue }
}
},
{
replaceOne: {
filter: { someReplaceField: someValue },
replacement: { newField: "newValue" }
}
}
])
Algolia
Their search api has batching built in:
const searchResults = await client.search({
requests: [
{ indexName, query },
{ indexName: index2, query: query2 }
]
})
S3 API
S3 has a few bulk APIs:
-
The
delete_objects
API supports bulk deleting up to 1000 objects at a time. -
The batch operations API utilizes a CSV to target specific objects, providing even greater batch sizes.
-
The storage inventory api supports generating a CSV, ORC, or Parquet file of your S3 objects metadata, rather than having to list them manually.
The graph API supports batching requests:
curl -i -X POST 'https://graph.facebook.com/me?batch=
[
{
"method":"GET",
"relative_url":"PAGE-A-ID"
},
{
"method":"GET",
"relative_url":"PAGE-B-ID"
}
]
&access_token=ACCESS-TOKEN'
They also support async and batch requests for their marketing APIs.
Tiger Beetle
All of their APIs support batch operations out of the box, e.g.
await client.createAccounts([account0, account1])
Salesforce
Supports bulk CRUD operations via an async job style bulk API.
Cloudflare
Supports creating a batch of redirects.
Google Analytics
Support for creating a batch of reports.
Mixpanel
Supports creating a batch of events.
Amplitude
Supports creating a batch of events.
Conclusion
Batch APIs allow for more efficent API calls and simplier integrations.