Resolved -
All hosts are now running at realtime, future work will segregate high volume hosts traffic to prevent wider system impact.
Oct 16, 21:55 UTC
Identified -
More event data being sent 'in' that can be processed, specifically, we're being rate limited by some customers Jira instances which is causing a backlog. Infrastructure limits mean this can impact all customers if sufficiently high volumes of hooks are present from such rated limited hosts.
We're going to create dedicated queues for such high volume hosts to solve this and isolate such problems in future.
Oct 16, 09:13 UTC
Investigating -
Large volumes of webhooks have accumulated overnight for some hosts, that can reduce throughput, we are upscaling resources to help.
Oct 16, 07:25 UTC