What happened?
On December 9, 2020 at 3:11 PM EST the CrewFire app started sending email and SMS notifications for some old activities. Those notifications shouldn't have gone out and, for some of our current and past customers, this meant having their email and phone inboxes flooded with emails
Nobody wants that. We sincerely apologize for this error.
Between 3:11 PM and 3:30 PM EST, 81 activities from 23 accounts had 48,522 notifications sent, 40,905 of which were for one larger CrewFire account.
Why did it happen?
Prior to the incident, we were doing maintenance on the CrewFire app, which was related to notifications for some Ambassador activities not being sent out properly.
During this process we were thoughtful and careful enough to update certain older activities, for which notifications never went out, to ensure your Ambassadors didn't get irrelevant notifications.
Unfortunately, as part of this process, there was a set of activities that had never received notifications in the past. When the CrewFire app was updated with the new code, notifications were triggered for these older activities and they were sent out immediately.
What are we doing to prevent it happening again?
Bugs likes this are hard to catch before they happen, when testing on a developer's computer it's not possible to account for the historical data present in the live/production CrewFire app.
We know the change we were making was potentially risky which lead us to double check activities it would affect and handle those, that made us think we were being cautious and handled the potential downsides of this change, we should have dug even deeper and spent more time analyzing the changes and live data.
We're going to institute a more stringent documentation and investigation process to changes similar to this one. We've only had one developer for a long time, but now that we are expanding out team, we'll introduce a habit of peer review, especially for the trickier changes like this bug fix.
One thing that is obvious to everybody involved is that we shouldn't have kept all this data related to cancelled CrewFire accounts. We used to keep that data as a convenience for customers that wanted to re-activate their accounts. Effective immediately we will adopt the practice of automatically deleting all customer data 30 days after their cancellation. This means we went and deleted all data for all cancelled CrewFire accounts this morning.
