Github Notification Spam — Post Mortem

TLDR — Oops. We’re sorry and fixing it.

On Saturday morning between 8:29am UTC and 8:40am UTC, the Gitcoin network was hit by a morass of Github Notification Spam. Here’s what it looked like.

These notifications did not come from Gitcoin Bot, they came from a community member who had forked our repo to work on the dApp.

These notification were sent out to 100s of repos, and references localhost:8080, a clear sign that a new contributor was trying to sync the blockchain’s StandardBounty information about bounties to their local environment, but had some misconfigured .env variables that accidentally sent out the notifications.

As of 11am UTC on Saturday, we have reached out to the user who produced all the notifications, but have not heard back from them. We are working under the assumption that it was an accident.

Hindsight being 20/20, with as much contributor traction as Gitcoin has, this was bound to happen eventually. Assuming we’re correct about the user not having malignant intent, we do not fault them at all.

Our Response

The core team was alerted at 9:11am UTC Saturday and began to debate an action plan. We have two team members in India, and they were able to discuss it before the US team woke up.

Whats worse? Lumburgh telling you to come in on Saturday, or a massive glut of notifications in your inbox telling you you have to come in on Saturday?

When the US team was up, we continued triaging and began to notify our network that we were responding.

The Action Plan

One of the mantras we have internally is ‘fix it twice’. This means that when we have a bug, we not only solve the bug itself… but the conditions that caused it to become a bug in the first place. We fix it twice.

Gitcoin is a mission-driven organization.
Gitcoin is the easiest way to monetize or incentivize work in Open Source Software.gitcoin.co

The first fix was to notify users about this issue, delete as many comments as possible, and let the offending user know what happened and how to prevent it.

The “Fix it twice” was to deploy some updates to our codebase that:

  1. Make our repo setup instructions more clear, especially where there are faults like this scenario.
  2. Provide more stopping points in our blockchain sync management command for local environments.
  3. Break out the notification application and sync applications into their own separate repos which must be explicitly installed, with permission.
  4. Reminding users that they can manage their notification and privacy preferences on Gitcoin’s Privacy Page.

What does it all mean?

First off, it means that a bunch of our community woke up to some annoying notifications. You know who you are — thanks for rolling with us.

Second, it means that I’m not on my Saturday bike ride right now. And the core team is rallying, on a weekend, to fix the issue. Thank you to the team! 🙏

Third, and sorry (not sorry) for getting philosophical here but… it is early days for applications that are have both an Open Data Layer and an Open Code Layer. There are some interesting architecture lessons that we, one of the first dApps built upon a ‘fat protocol’, are learning.

Web 3.0 The open data layer. Thanks to our friends at Origin Protocol for lending us this gif!

For me, a big lesson is that when you combine Open Data and an Open Codeyou get a phenomena that is extremely powerful. As Uncle Ben from Spiderman put it, “with great power comes great responsibility”.

It is our responsibility to help members of the distributed network of OSS BUIDLers yield their power without bugs. We failed today, but we’re evolving along with others in the emergent BUIDL ecosystem.

We know notification spam is annoying. The notification spam was a negative UX for many of you, but the silver lining is our opportunity to take quick action that demonstrates the both (1) the mantra of decentralization and (2) the experience of users is very important to us.

Kevin Owocki
Gitcoin Founder


To learn more about Gitcoin, click below.

0
grants