PO Update (closed) webhook notification being sent more than once?

GraphQL API
1

Hi,
It seems that my webhook set to receive PO Update notifications seems to be receiving multiple pings on a PO being closed. This happens every single time for me - could you pls see if this is something that is incorrectly set on my end or an SH issue?

Here’s the first and second webhook export of a test PO I closed:

[
  {
    "headers": {
      "host": "redacted",
      "x-forwarded-scheme": "https",
      "x-forwarded-proto": "https",
      "x-forwarded-for": "redacted",
      "x-real-ip": "redacted",
      "content-length": "1199",
      "cdn-loop": "cloudflare",
      "cf-ipcountry": "US",
      "accept-encoding": "gzip, br",
      "cf-ray": "898dc54219b68000-IAD",
      "cf-visitor": "{\"scheme\":\"https\"}",
      "sentry-trace": "e4f717e61767414494dd10a7940afd39-adb68c90aee8f12c-0",
      "baggage": "sentry-trace_id=e4f717e61767414494dd10a7940afd39,sentry-environment=prod,sentry-release=293c323,sentry-public_key=613bdf17979047029f3c4b2e7afe6a5d,sentry-transaction=customers_webhooks.process_webhooks",
      "user-agent": "ShipHero Request V1.0",
      "accept": "*/*",
      "content-type": "application/json",
      "x-shiphero-hmac-sha256": "xTWlAMaf9kKv+/KGIX9hShDoyQLH+byiUejccg3KGUM=",
      "cf-connecting-ip": "redacted"
    },
    "params": {},
    "query": {},
    "body": {
      "test": "0",
      "purchase_order": {
        "id": 1493614,
        "po_number": "multitest1",
        "po_id": 8,
        "po_uuid": "UHVyY2hhc2VPcmRlcjoxNDkzNjE0",
        "account_id": 69683,
        "line_items": [
          {
            "id": "23106a9b49684d04a6dc",
            "quantity": 1,
            "quantity_received": 0,
            "sku": "Test5",
            "vendor_id": null,
            "vendor_uuid": "VmVuZG9yOk5vbmU=",
            "vendor_account_number": null,
            "vendor_sku": null
          },
          {
            "id": "4ba7da7d92dbc5a7709e",
            "quantity": 1,
            "quantity_received": 0,
            "sku": "Test3",
            "vendor_id": null,
            "vendor_uuid": "VmVuZG9yOk5vbmU=",
            "vendor_account_number": null,
            "vendor_sku": null
          },
          {
            "id": "bcdfb49052f90f6f5155",
            "quantity": 1,
            "quantity_received": 0,
            "sku": "Test4",
            "vendor_id": null,
            "vendor_uuid": "VmVuZG9yOk5vbmU=",
            "vendor_account_number": null,
            "vendor_sku": null
          },
          {
            "id": "c908e2c064fb3d15989d",
            "quantity": 1,
            "quantity_received": 0,
            "sku": "Test2",
            "vendor_id": null,
            "vendor_uuid": "VmVuZG9yOk5vbmU=",
            "vendor_account_number": null,
            "vendor_sku": null
          },
          {
            "id": "fd5b40a73d516543c238",
            "quantity": 1,
            "quantity_received": 0,
            "sku": "Test1",
            "vendor_id": null,
            "vendor_uuid": "VmVuZG9yOk5vbmU=",
            "vendor_account_number": null,
            "vendor_sku": null
          }
        ],
        "warehouse_id": 74776,
        "status": "closed",
        "webhook_type": "PO Update"
      }
    },
    "webhookUrl": "http://localhost:5678/webhook/sh/po",
    "executionMode": "production"
  }
]

[
  {
    "headers": {
      "host": "redacted",
      "x-forwarded-scheme": "https",
      "x-forwarded-proto": "https",
      "x-forwarded-for": "redacted",
      "x-real-ip": "redacted",
      "content-length": "1199",
      "cdn-loop": "cloudflare",
      "cf-ipcountry": "US",
      "accept-encoding": "gzip, br",
      "cf-ray": "898dc5597b3581b5-IAD",
      "cf-visitor": "{\"scheme\":\"https\"}",
      "sentry-trace": "3e0bd6e10e384bc082d2235487a34549-bb1d5fbd3d6956b4-0",
      "baggage": "sentry-trace_id=3e0bd6e10e384bc082d2235487a34549,sentry-environment=prod,sentry-release=293c323,sentry-public_key=613bdf17979047029f3c4b2e7afe6a5d,sentry-transaction=customers_webhooks.process_webhooks",
      "user-agent": "ShipHero Request V1.0",
      "accept": "*/*",
      "content-type": "application/json",
      "x-shiphero-hmac-sha256": "xTWlAMaf9kKv+/KGIX9hShDoyQLH+byiUejccg3KGUM=",
      "cf-connecting-ip": "redacted"
    },
    "params": {},
    "query": {},
    "body": {
      "test": "0",
      "purchase_order": {
        "id": 1493614,
        "po_number": "multitest1",
        "po_id": 8,
        "po_uuid": "UHVyY2hhc2VPcmRlcjoxNDkzNjE0",
        "account_id": 69683,
        "line_items": [
          {
            "id": "23106a9b49684d04a6dc",
            "quantity": 1,
            "quantity_received": 0,
            "sku": "Test5",
            "vendor_id": null,
            "vendor_uuid": "VmVuZG9yOk5vbmU=",
            "vendor_account_number": null,
            "vendor_sku": null
          },
          {
            "id": "4ba7da7d92dbc5a7709e",
            "quantity": 1,
            "quantity_received": 0,
            "sku": "Test3",
            "vendor_id": null,
            "vendor_uuid": "VmVuZG9yOk5vbmU=",
            "vendor_account_number": null,
            "vendor_sku": null
          },
          {
            "id": "bcdfb49052f90f6f5155",
            "quantity": 1,
            "quantity_received": 0,
            "sku": "Test4",
            "vendor_id": null,
            "vendor_uuid": "VmVuZG9yOk5vbmU=",
            "vendor_account_number": null,
            "vendor_sku": null
          },
          {
            "id": "c908e2c064fb3d15989d",
            "quantity": 1,
            "quantity_received": 0,
            "sku": "Test2",
            "vendor_id": null,
            "vendor_uuid": "VmVuZG9yOk5vbmU=",
            "vendor_account_number": null,
            "vendor_sku": null
          },
          {
            "id": "fd5b40a73d516543c238",
            "quantity": 1,
            "quantity_received": 0,
            "sku": "Test1",
            "vendor_id": null,
            "vendor_uuid": "VmVuZG9yOk5vbmU=",
            "vendor_account_number": null,
            "vendor_sku": null
          }
        ],
        "warehouse_id": 74776,
        "status": "closed",
        "webhook_type": "PO Update"
      }
    },
    "webhookUrl": "http://localhost:5678/webhook/sh/po",
    "executionMode": "production"
  }
]
2

Hey @Raj, I’ll have to do some troubleshooting. I will get back to you as soon as I have more details.
Have a great day!

3

Yes sure, @tomasfd , I’ll wait for your findings.

4

Hello @raj!
Which steps are you taking to update the status to closed?
I still was not able to reproduce it on my end.
Have a nice day!

5

Hi @tomasfd, I just went to https://app.shiphero.com/dashboard/purchase-orders/details/XXXX and selected Status: Closed in the dropdown and clicked Save Changes.
But I believe the same thing happens using this page: https://purchase-orders.shiphero.com/purchase-orders/details/XXXX

If it helps, I can run the test again and send you the same exports with no redactions.
Thank you

6

Hi @Raj

Checking the logs, I see there are two registered webhooks for PO_UPDATE.
One to hook.us1.make.com and the other to n8n.shipbots.net and that is why you are getting the notifications doubled.

Have a nice day!

8

Hi @Luisperrone, Thank you for looking into this, but the exports I posted are only collected from the n8n url. That’s the only one I’ve been looking at while noticing the duplicates.
Does that information help?

10

Hi @Raj

Yes, it does help!

I found the logs for the duplicated notification.

I also see there was a fix about this on June 28.

Can you make a new test to see if this is solved?

11

Hi @Luisperrone, sorry to report this but my happiness lasted only a few seconds - then the second notification showed in the log.
Here are both exports:

[
  {
    "headers": {
      "host": "redacted",
      "x-forwarded-scheme": "https",
      "x-forwarded-proto": "https",
      "x-forwarded-for": "redacted",
      "x-real-ip": "redacted",
      "content-length": "817",
      "cdn-loop": "cloudflare",
      "cf-ipcountry": "US",
      "accept-encoding": "gzip, br",
      "cf-ray": "89d03012af8e4316-EWR",
      "cf-visitor": "{\"scheme\":\"https\"}",
      "sentry-trace": "4f928643f4ff498a936cf8d31e902a0b-9cb3898c3e28b415-0",
      "baggage": "sentry-trace_id=4f928643f4ff498a936cf8d31e902a0b,sentry-environment=prod,sentry-release=293c323,sentry-public_key=613bdf17979047029f3c4b2e7afe6a5d,sentry-transaction=customers_webhooks.process_webhooks",
      "user-agent": "ShipHero Request V1.0",
      "accept": "*/*",
      "content-type": "application/json",
      "x-shiphero-hmac-sha256": "xVCeBkaY7Qy5h+81pHKsCP5jPCZck74jyJjkfOyi2j8=",
      "cf-connecting-ip": "redacted"
    },
    "params": {},
    "query": {},
    "body": {
      "test": "0",
      "purchase_order": {
        "id": 1471109,
        "po_number": "354341",
        "po_id": 4,
        "po_uuid": "UHVyY2hhc2VPcmRlcjoxNDcxMTA5",
        "account_id": 69683,
        "line_items": [
          {
            "id": "5aaec7dcd1cca9b49eda",
            "quantity": 1,
            "quantity_received": 0,
            "sku": "UP-0012",
            "vendor_id": null,
            "vendor_uuid": "VmVuZG9yOk5vbmU=",
            "vendor_account_number": null,
            "vendor_sku": null
          },
          {
            "id": "7fe4d818a8758537a6eb",
            "quantity": 1,
            "quantity_received": 0,
            "sku": "UP-0010",
            "vendor_id": null,
            "vendor_uuid": "VmVuZG9yOk5vbmU=",
            "vendor_account_number": null,
            "vendor_sku": null
          },
          {
            "id": "eaf974803f7c945b5c13",
            "quantity": 1,
            "quantity_received": 0,
            "sku": "UP-0011",
            "vendor_id": null,
            "vendor_uuid": "VmVuZG9yOk5vbmU=",
            "vendor_account_number": null,
            "vendor_sku": null
          }
        ],
        "warehouse_id": 74776,
        "status": "closed",
        "webhook_type": "PO Update"
      }
    },
    "webhookUrl": "redacted",
    "executionMode": "production"
  }
]

and

[
  {
    "headers": {
      "host": "redacted",
      "x-forwarded-scheme": "https",
      "x-forwarded-proto": "https",
      "x-forwarded-for": "redacted",
      "x-real-ip": "redacted",
      "content-length": "817",
      "cdn-loop": "cloudflare",
      "cf-ipcountry": "US",
      "accept-encoding": "gzip, br",
      "cf-ray": "89d03065689b43f4-EWR",
      "cf-visitor": "{\"scheme\":\"https\"}",
      "sentry-trace": "423c0c68795e4a198e1e3a08538fb10b-904333ca42a51f7a-0",
      "baggage": "sentry-trace_id=423c0c68795e4a198e1e3a08538fb10b,sentry-environment=prod,sentry-release=293c323,sentry-public_key=613bdf17979047029f3c4b2e7afe6a5d,sentry-transaction=customers_webhooks.process_webhooks",
      "user-agent": "ShipHero Request V1.0",
      "accept": "*/*",
      "content-type": "application/json",
      "x-shiphero-hmac-sha256": "xVCeBkaY7Qy5h+81pHKsCP5jPCZck74jyJjkfOyi2j8=",
      "cf-connecting-ip": "redacted"
    },
    "params": {},
    "query": {},
    "body": {
      "test": "0",
      "purchase_order": {
        "id": 1471109,
        "po_number": "354341",
        "po_id": 4,
        "po_uuid": "UHVyY2hhc2VPcmRlcjoxNDcxMTA5",
        "account_id": 69683,
        "line_items": [
          {
            "id": "5aaec7dcd1cca9b49eda",
            "quantity": 1,
            "quantity_received": 0,
            "sku": "UP-0012",
            "vendor_id": null,
            "vendor_uuid": "VmVuZG9yOk5vbmU=",
            "vendor_account_number": null,
            "vendor_sku": null
          },
          {
            "id": "7fe4d818a8758537a6eb",
            "quantity": 1,
            "quantity_received": 0,
            "sku": "UP-0010",
            "vendor_id": null,
            "vendor_uuid": "VmVuZG9yOk5vbmU=",
            "vendor_account_number": null,
            "vendor_sku": null
          },
          {
            "id": "eaf974803f7c945b5c13",
            "quantity": 1,
            "quantity_received": 0,
            "sku": "UP-0011",
            "vendor_id": null,
            "vendor_uuid": "VmVuZG9yOk5vbmU=",
            "vendor_account_number": null,
            "vendor_sku": null
          }
        ],
        "warehouse_id": 74776,
        "status": "closed",
        "webhook_type": "PO Update"
      }
    },
    "webhookUrl": "redacted",
    "executionMode": "production"
  }
]
12

Hi @Raj

I’ve escalated this to our engineering department. I’ll keep you updated.

Have a nice day!

1 Like
14

Hi @Raj

The engineering team made a fix for this.
Can you confirm that it is working as expected?

Have a nice day!

1 Like
15

Hey @Luisperrone, YES!
Confirmed, it’s fixed - Thank you so much!

1 Like
16

Hi @Luisperrone ,

The problem is happening again. I’m getting duplicate notifications for some POs, and just yesterday I got FOUR notifications for a single PO!

Here’s the info on these (by the way, for tracking/researching these, do you need any specific part of data that’s received by the webhook, or the whole thing? - so next time I can give you just the specific bits you need)

– I had to move the data I added here since it far exceeded the character limit for a post: Hi @Luisperrone ,The problem is happening again. I'm getting duplicate notif - Pastebin.com –

17

Hi @Raj

I’ve looked into this and it was due to a push the engineering team did to one of our workers.

I checked the last PO notification, PO 64, and it looks good.

Can you please confirm is not happening anymore?

18

HI @Luisperrone , Pls let me monitor it for a day and I’ll confirm to you what I find. Thank you for looking into this.

19

Not a problem.
Let me know.

Have a nice day!

20

Hey @Luisperrone , yesterday no actual POs were received/closed at work, so today I just tested twice on dummy POs and I only received a single notification. So far it looks like the fix is working - thank you!
(I’ll add more info here in case any issues are noticed in future)

1 Like
21

Hi @Raj

I’m glad we found the root cause, and it’s no longer happening. The engineering team is aware of this and they’ll try to be more thorough when doing those updates.

Have a nice day!

22

Hi @Luisperrone, Sorry to say but this is still happening - confirmed by testing more than once.
Here are headers of two from the same PO close action, received around 10 seconds apart (last duplicate was 4 seconds apart).

content-length:423
cdn-loop:cloudflare
cf-ipcountry:US
accept-encoding:gzip, br
cf-ray:8b741041b8c42412-IAD
cf-visitor:{"scheme":"https"}
sentry-trace:30bb0f6d41434ac18de90a3b9625ed08-95049bc39d6e3fe3-0
baggage:sentry-trace_id=30bb0f6d41434ac18de90a3b9625ed08,sentry-environment=prod,sentry-release=b8eabd9,sentry-public_key=613bdf17979047029f3c4b2e7afe6a5d,sentry-transaction=customers_webhooks.process_webhooks
user-agent:ShipHero Request V1.0
accept:*/*
content-type:application/json
x-shiphero-hmac-sha256:jJHsCj8s3EyA4W2NnePT29Q0G+Sm/DwDuuC1KIwOjo4=

content-length:423
cdn-loop:cloudflare
cf-ipcountry:US
accept-encoding:gzip, br
cf-ray:8b7410c95f872039-IAD
cf-visitor:{"scheme":"https"}
sentry-trace:0be6e724d3aa408bb329c174d9665e8b-84538cf8bee27915-0
baggage:sentry-trace_id=0be6e724d3aa408bb329c174d9665e8b,sentry-environment=prod,sentry-release=b8eabd9,sentry-public_key=613bdf17979047029f3c4b2e7afe6a5d,sentry-transaction=customers_webhooks.process_webhooks
user-agent:ShipHero Request V1.0
accept:*/*
content-type:application/json
x-shiphero-hmac-sha256:jJHsCj8s3EyA4W2NnePT29Q0G+Sm/DwDuuC1KIwOjo4=

Pls let me know if you need more details - I only included the headers to save time+effort.
Thank you

23

Hi @Raj

Can you please give me the po_id and the repeated SKUs?