Gitcoin Incident Report

Hey Gitcoiners – this is a brief post to explain an outage we had this morning, what caused it, and the steps we’ve taken to correct it and prevent it from happening in the future.

Starting around 5am PT, we deployed some follow-on fixes to our recent integration of the Web3Connect / Web3Modal library. At the time of this deploy, one of our production machines was experiencing degraded performance due to a periodic background job that consumed all of the system IOPS. This background job was written in such a way that a flat database file was reread from disk tens of thousands of times, which quickly drained the performance on the system. Because the available IOPS was consumed by this process, during the deploy some of the new static file references were not updated.

If you accessed the Gitcoin site from this time until we noticed the issue and properly updated the references (about 2.5 hours of downtime) you may not been able to interact with the Ethereum blockchain through the Gitcoin dapp. We apologize for this degraded performance and we’ve now taken steps to mitigate this issue by disabling the background job, and creating a fix that will be deployed before we enable the background job again.
If you notice any further issues, please don’t hesitate to contact us at engineering@gitcoin.co. Also, if you find a security issue on the site, we have a security bounty program you can disclose through and receive a reward – details on this program are here: https://docs.gitcoin.co/mk_securitybounty/

image from Alexis Lê-Quôc from New York, United States / CC BY-SA (https://creativecommons.org/licenses/by-sa/2.0) https://commons.wikimedia.org/wiki/File:Half_filled_server_racks.jpg