Inventory CSV sync in processing frozen

GraphQL API
1

We are experiencing what seems to be a significant bug.
The inventory CSV upload job froze yesterday and now we cannot submit a new one because the API claims “job already processing”.
Indeed it is still “processing” although this Status output was taken today, i.e. this is the exact same result we had almost 24H ago:

  "data": {
    "inventory_sync_status": {
      "data": {
        "customer_account_id": null,
        "total_count": 49199,
        "status": "processing",
        "error": null,
        "created_at": "2020-02-06T22:10:17+00:00",
        "updated_at": "2020-02-06T22:10:22+00:00",
        "success_count": 990,
        "error_count": 3430,
        "finished_count": 4420
      }
    }
  }
}

Has anyone experienced that? This is the second time. We had to abort the first test job that froze for a week (we didn’t pay attention to the status).
This time we’ll let it as it is in case that helps for debugging.

Has anyone else experienced that?
Are we doing something wrong?
Is there a bug in the ShipHero backend?

Thank you for your help! This is a big blocker for our project.

2

Hi @Ant
Is there any chance you could send me the request_id on the response you get?
Or the batch_id.
Thanks in advance!
Tom

3

@tomasw Thanks for your response! I work with @Ant on the same project, so following up on the above the request id as requested is:

"request_id": "5e3ddabfe5b5f99b540ff0d9"

Please let us know if you need more information!

4

Thanks for providing the info @karolinakondrat !

Yes, I just confirm it’s the same issue our engineering team is trying to fix.
Hopefully we’ll have it resolved in the next few days, but I’ll let you know as soon as I have an update.
Thanks for the patience on this!
Tom

1 Like
5

Thanks for getting back to us so quickly.
Glad to hear that it’s already in your radar.
Looking forward to the fix!

6

@tomasw - Is there an update on this ticket? We need to use this functionality this week.

7

Hey @tomasw

We are trying to determine what exactly is the issue occurring here, so that we can attempt to create a workaround.

We are aware that we are passing in inventory changes for some SKUs that do not exist in the particular Shiphero warehouse. Will your fix prevent the job from error/freeze in these cases? (i.e. skip/ignore SKUs we don’t have)

Is the issue that the job count is too high? Would reducing size and submitting multiple jobs help?

Any insight you can provide will be extremely helpful.

Thanks for your time,
Jon

8

Hi @dan & @jon !
I apologize for the delay on this. We are still working on a fix for it, but still don’t have an exact ETA.

A workaround in the meantime would be to use the UI Inventory Upload Feature

Again, sorry for the waiting and I will update as soon as I have something about it.
Thanks!
Tom

9

Hi @tomasw - Can you answer Jon’s questions so we can attempt to troubleshoot on our end?

Manually uploading inventory is not an option. It needs to be automated via API.

10

Hi @dan
Unfortunately, we don’t have a successful workaround at the moment because it’s not entirely related to invalid SKUs or the number of lines processed.
Making smaller uploads will help, but we can’t guarantee it will succeed all of the time.

I know this has gone on way too long, and we currently have an engineering team dedicated to fixing this.
I will let you know as soon as I have an update or possible workaround.

Thanks again for the patience,
Tom.

11

Hi @tomasw - We need a rough timeline on the fix so we can make a decision on how to move forward.

12

This seems to be getting worse …

Yesterday we started a small inventory sync of only 1000 items (to avoid the known issue reported here) and this morning it was still in “pending” status. Sometime this morning it started and went to processing but with 0 in all the *_count fields. and it is still frozen at this stage. Hopefully it’s just a one of.

We have now a mechanism to give up after a certain amount of time but it’s still worrying.