Service disruption for Automated Persisted Queries (APQs)

2024-01-08T13:45:00Z

Jan 8, 13:45 UTC
Resolved - - A bug was released on Jan 8th at 1.43 pm UTC while improving Persisted Operation support. The two areas of code overlap, and unfortunately, the change broke support for APQs.
- Our E2E test suite should have caught this bug.
- Unfortunately, we recently made many improvements to our E2E test suite and silently broke the validity of the APQ E2E tests. These tests were running and reporting successes, but under the hood, they were erroneously being run against a server that does not support APQ.
- The impact of this bug was not widespread enough to trigger alarms after release.
- At 11.45 pm UTC, a customer raised an issue with APQs, and our engineering team started investigating.
- On Jan 9th at 2.05 am UTC, a fix was deployed, and the issue was resolved.