Bulk Operations for read-only queries

Hello!

We use both Shopify and ShipHero for our store, and I recently ran across a feature available in the Shopify API that I would love to see made available in the ShipHero API. They allow the ability to create a “bulk_operations” mutation, which lets you send a read-only query as part of the mutation, and it takes care of the pagination for large data sets, giving back a URL from which to download the JSONL (JSON Lines) formatted results. Their intro can be found here.

I know a previous request was made for bulk operations, but a number of the use cases are for mutations rather than queries, so I wanted to add this feature request for query-only bulk operations.

Thanks,
Jeremy

Hi @jeremyw!
That looks amazing! I’m junior dev here, so not really sure how much work it will require to make that possible though, but I will be more than happy to bring that up to our Public API engineers and see what they think about it.
That would be something like the ones we have for inventory sync and inventory snapshot, right?

In that case what kind of Bulk Operations are you looking for? Orders?
Thanks in advance!
Tom

Hey @tomasw!

Thank you for at least considering it and passing it along!

The way I understand it and envision it is that it wouldn’t be like the inventory_sync because that is making changes to inventory, but this could be a replacement for inventory_snapshot, and be able to process any page-able queries thrown at it, whether that be orders, shipments, products, picks/packs_per_day, or whatever.

For example, I could use this to create my own inventory snapshot which contains not only sku and quantities on hand, but also bin location, kit component information, barcodes and tags. These are things that we frequently need for reporting and warehouse management. Right now I have python scripts that our users run to generate things like purchase orders, warehouse moves, take stock offline for production, etc, but in order for it to run, the user has to manually go to the inventory screen in the desktop UI and manually export and make sure the CSV is in the right location for the script to find it, and so on. It’s extra steps for them. Pulling an automated snapshot of products/inventory via normal API methods is currently slow, requires pagination, and so forth. On top of that, I’m still facing the code 22 issues, so a script may get through the first 800 products, and then suddenly error out with code 22. (I know I can catch that, but that’s a full minute wasted of the query trying to get results each time code 22 is hit) This idea of being able to supply the same query to a bulk_operations mutation and have it drop a URL that I can download in the background, process, and serve up the results for the user without them having to worry about getting it themselves would be huge.

Likewise, every day I pull data with the picks_per_day query to store in a database for user performance analysis. I’m also working on a packs_per_day version, but the nested pagination of shipments and lines are throwing me off there. If I didn’t have to worry about the nested pagination, I could more easily handle the URL drop of results, and that would be an amazing thing.

I hope that gives some idea of what it could be used for, and how it’s a generalized solution compared to having specific “snapshot” type queries that have to be separately maintained and updated.

Thanks for hearing me out, and let me know if you have any other questions!

Jeremy

I see what you are saying @jeremyw , and thanks for bringing that up!
To build this I’m not sure how much work will require, so I made the request and will ask our Software Architect to take a look at it and provide a better insight.
I do think it’s worth the try, sounds amazing!
Thanks again and I will let you know as soon as I get a response back!
Tom

1 Like

Hi @jeremyw!
Thanks again for suggesting this. I was able to confirm that this is on our radar, but for the moment we won’t be able to dedicate the resources to build it, so I don’t have a plan or ETA right now.
I will monitor our plans and let you know as soon as this gets back on our plans.
Thanks again!
Tom

1 Like

Thanks, @tomasw! Just knowing it’s on the radar is a good thing. :smile: