Notifications Sent for Old Activities - Incident on December 9, 2020

Alan VanToai

December 10, 2020

What happened?

On December 9, 2020 at 3:11 PM EST the CrewFire app started sending email and SMS notifications for some old activities. Those notifications shouldn't have gone out and, for some of our current and past customers, this meant having their email and phone inboxes flooded with emails

Nobody wants that. We sincerely apologize for this error.

Between 3:11 PM and 3:30 PM EST, 81 activities from 23 accounts had 48,522 notifications sent, 40,905 of which were for one larger CrewFire account.

Why did it happen?

Prior to the incident, we were doing maintenance on the CrewFire app, which was related to notifications for some Ambassador activities not being sent out properly.

During this process we were thoughtful and careful enough to update certain older activities, for which notifications never went out, to ensure your Ambassadors didn't get irrelevant notifications.

Unfortunately, as part of this process, there was a set of activities that had never received notifications in the past. When the CrewFire app was updated with the new code, notifications were triggered for these older activities and they were sent out immediately.

What are we doing to prevent it happening again?

Bugs likes this are hard to catch before they happen, when testing on a developer's computer it's not possible to account for the historical data present in the live/production CrewFire app.

We know the change we were making was potentially risky which lead us to double check activities it would affect and handle those, that made us think we were being cautious and handled the potential downsides of this change, we should have dug even deeper and spent more time analyzing the changes and live data.

We're going to institute a more stringent documentation and investigation process to changes similar to this one. We've only had one developer for a long time, but now that we are expanding out team, we'll introduce a habit of peer review, especially for the trickier changes like this bug fix.

One thing that is obvious to everybody involved is that we shouldn't have kept all this data related to cancelled CrewFire accounts. We used to keep that data as a convenience for customers that wanted to re-activate their accounts. Effective immediately we will adopt the practice of automatically deleting all customer data 30 days after their cancellation. This means we went and deleted all data for all cancelled CrewFire accounts this morning.

Free Ebook!

The Ultimate Guide to Brand Ambassador Marketing.

We distilled years of brand ambassador marketing experience to show you how to promote your content, reach more people, and grow your brand with brand ambassadors.
Thanks! The ebook is on it's way to your inbox!
Oops! Something went wrong while submitting the form.

WRITTEN BY

Alan VanToai

Co-Founder of CrewFire