Syncro

Background Workers

  1. Overview
  2. Services
  3. Background Workers
  4. Issue

Resolved Background Workers: Scripting, Scheduled Reports, and Notifications

Postmortem

On Tuesday, February 20, at 5:49 AM PT, we experienced a delay in our asynchronous job processing, impacting the timely execution of key operations, including script runs, asset synchronization, and scheduled reporting. Although minor delays are expected for lower-priority tasks, our system is designed to ensure the prioritization of critical workflows.

To address this issue, we immediately increased our infrastructure capacity, resolving the backlog and returning queue times to normal by 7:20 AM PT. Subsequent investigation revealed the cause: a failure in a worker process responsible for scheduling report deliveries. The issue was rectified with a deployed fix by 1:04 PM PT.

Moving forward, we are focusing on measures to avoid such delays, including controls to prevent worker process overruns and improvements to our monitoring systems for quicker detection of critical workflow disruptions. These efforts are ongoing, with a commitment to invest necessary resources for future prevention.

Resolved

The team has deemed our Background Workers stable since our latest updates and are calling the all clear.

We will be sharing a retrospective on this incident and action items we will be taking to avoid incidents like this in the future.

This issue is now resolved, but if you have any further questions or concerns relating to this issue, please reach out to our support team at help@syncromsp.com.

Monitoring

UPDATE: The background workers are back online and operational. We are seeing improvement and stability with all operations and queues. The team is continuing to monitor until we call the all clear.

Investigating

We have identified an issue impacting background processing including: scripting, scheduled reports, and notifications.

Our team is currently working on resolving it. We apologize for any inconvenience this may have caused and appreciate your patience while we work to fix the problem. We will provide further updates as soon as possible. Thank you for your understanding.