Post Mortem — Production Issues 2018/02/20

In the spirit of transparency, I want to write up a small post-mortem on last week’s issues to tell you what happened, why, and to illuminate what we’re doing to evolve our development processes.

To set the tone for this post-mortem, I first want to borrow a line from the Retrospective Prime Directive:

“Regardless of what we discover, we understand and truly believe that everyone did the best job they could, given what they knew at the time, their skills and abilities, the resources available, and the situation at hand.”

TLDR — Hindsight is 20/20; Retrospectives should be blameless. You can’t have accountability if you have blame.

What happened?

If you are a Gitcoin user, you may have noticed one of the following

• The ‘Started Work’ flag disappeared on bounties where work had begun
• GitcoinBot left erroneous comments on GitHub threads
• We accidentally emailed people twice about the same funded issues
• No funds were put at risk by these issues

Why?

Why did this happen? Why did it reach the level of a production bug?

The team did a deployment in mid-February that rewrote the layer of the application which syncs the Ethereum blockchain to the Gitcoin.co application database. We migrated this layer of the application from web3js to web3py.

Before the rewrite (with web3js), users who submitted a bounty would have to keep their browser window open until their web browser and Metamask plugin had confirmed that their tx had been confirmed. This was a suboptimal UX because sometimes Ethereum transactions can take several minutes to confirm.

After the rewrite (with web3py), users who submit a bounty can close their browser window, because the application will be smart enough to automatically sync their changes once the tx is confirmed.

We are confident that the rewrite will deliver the best user experience in the medium and long term, but in the short term, we dropped the ball on properly QAing and maintaining the release.

This was compounded by:

• The core team traveling (to ETHDenver + After Party, Iceland, Portugal)
• A staging environment that didn’t mirror production as closely as it should

Root Cause Fixes

You can see the root cause fixes on the gitcoinco/web commit tree.

Getting Better At Deployments

We believe wholeheartedly that the UX for the application needs to be buttoned up as Gitcoin scales up to power incentivization for a wider array of Open Source Repos.

As a means to that end, we have 38 open issues meant to button up the UX for Gitcoin. These improvements to our copy, design, and aesthetics of the site, will help make the UX more grokk-able with less of a learning curve.

Here is what the core team is doing in the future to make sure this doesnt happen again:

• Better staging environment
• Getting our test coverage up
• Refactoring the way bounties (the database-object) relate to each other
• More transparent logging via Rollbar
• (Long term) Decentralizing knowledge of the app to the community so we’re not so dependent on myself and the Core Dev team.

In Summation

Transparency is an important value for Gitcoin. We believe that in the future of work, accountability will flow fluidly from between networks of individuals with aligned incentivizes.

We hope that you’ll join us in making Gitcoin better by:

1. Working on paid work on the platform.. Especially some of the paid work that helps us button up UX 😊(Gitcoin Issue Explorer)
2. Joining our community of like-minded developers (Gitcoin Slack)

See you around the community 👋,

@owocki

Thanks to Vivek Singh.