Network outage on June 23, 2025
June 24, 2025 - Reading time: 2 minutes

We're writing to share details about a recent network outage that our users may have experienced, particularly those on the Eastern time zone. We understand how disruptive service interruptions can be, and we sincerely apologize for any inconvenience this may have caused.

What Happened

On June 22, 2025, our platform underwent migration to a new cloud infrastructure, following a schedule proposed by our cloud provider. After the migration, their network technician reviewed the traffic, and initial observations indicated activity across all cloud servers. What wasn't immediately apparent to the technician was that a significant portion of this traffic was internal server communication, essential for automated background tasks that keep our platform running smoothly.

Unfortunately, relying solely on these observations, the migration ticket was closed without a crucial step: user acceptance (UA) confirmation. This step, vital in any migration or upgrade, ensures that all user-facing functionalities are validated and working as expected by our team.

By the time ScheduleInterpreter.com, Inc. team identified the issue, it was already the start of the working day for the Eastern time zone. It then took an additional 45 minutes to pinpoint the exact problem, engage the necessary personnel on the cloud provider's side, and revert the network migration.

Our Commitment to Preventing Future Incidents

We've already taken steps to address this. We've communicated with our cloud provider to request a change in their policy:

  • All future infrastructure changes should be conducted exclusively on Saturdays.

  • A dedicated time slot on Sunday mornings must be provided for our team to conduct thorough user acceptance validation.

This incident, while frustrating, has provided valuable lessons. We're committed to refining our processes and working closely with our partners to ensure the reliability and stability of our services. We appreciate your understanding and continued trust.

Search

Sections