Summary
On the morning of Saturday May 10th, 2025, we release a patch update for PoLo that caused the app to hang on the start screen for a significant number of users.
We got notified in our Discord server and promptly disabled the patch update, but the app failed to revert to the previous version.
About 300 users downloaded the update. A couple of dozen of them reported having issues on our Forums and/or Discord server.
No data was lost, but the app was unusable during the entire weekend for these users.
Investigation
On Saturday morning, a user reported a bug (unrelated to the incident)that was crashing the app when logging certain calls. We took a quick look, determined the cause and fixed it with a small change in a single line of code.
Seeing how this was a minor change, and it could affect other users, we decided to release it as “patch update 7” for the April release.
In the previous days we had been working on the upcoming May release, so our test devices were running that version meaning we could not quickly test the patch update.
When users reported issues on the Discord server, we were not home so all we could do was quickly disable the update. The patch update mechanism is supposed to revert automatically to the previous version, but users were still reporting the app was stuck on the start screen.
Early on Sunday morning, we started investigating the issue and found that the crash was caused by the patch update having been built against the May release instead of the April release. We tried to push a new patch update with the correct version, but it failed to install.
It’s still unclear why the patch update mechanism failed to revert to the previous version, but we believe it had to do with the app crashing earlier during the boot process, before the update mechanism was activated.
This forced us to do a “full update” in order to fix the issue. Patch updates are distributed under our own control and can be published instantly. Full releases are distributed through the Google Play Store and the Apple AppStore, and have to go through a formal review process.
Google approved it within an hour and was available by noon ET. Apple does not seem to be reviewing apps during weekends, so the update was not available to iOS users until Monday at 5am ET.
After the updates were published, we got reports confirming that it fixed the crash and that no data had been lost.
Lessons Learned
Avoid releasing updates on a weekend, and come up with a clear policy on when and how to release updates.
Assume that Apple does not review apps during weekends.
Keep at lease one test device running the public version of the app and always test patch updates on it.
Use the bleeding edge and unstable patch channels for testing patch updates with small numbers of users before rolling them out to the stable channel.
Encourage more users to be on the unstable patch channel.
Build mechanisms to ensure patch releases are always built against the correct version of the app.
Suggested release policy
- If the issue is not severe, test it on the unstable channel for at least a day, and do not release it on a Friday, Saturday or Sunday.
- If there’s an issue affecting a large number of users, test it on the unstable channel first with at least 50 downloads, and users on both Android and iOS confirming the results.
- If necessary to do a patch update to fix a problem for a small number of users, use the bleeding edge channel.