Big project release finally happened on Tuesday evening. 20 million transactions to migrate and a real time feed to stitch into the site and app behind a whitelist while keeping everything else running. It went really well until about 6.30am on Wednesday when the database began using up loads of resources, locking and basically being very unhappy. Our initial joy evaporated and after very long days for some people (I was awake for about 30 hours Tuesday over into Wednesday and others did far more than I) putting things live and even longer months building the thing it was a real gut punch.
We carried on having passed the point of rollback return a few hours earlier but the problems persisted for most of the week and we are still not fully back to normal as I type this although the customer problems we experienced Wednesday and Thursday have thankfully been resolved.
A shitty week that had a few silver linings. The way people came together in engineering to rally round this issue was great to see. We have made changes in several areas that will benefit a number of future projects as well as just making stuff run better. A big chunk of technical debt has been paid off but it has caused some other debts to need paying too after a lack of focus on them for too long.
Away from mega releases we finally agreed teams for our new Business Outcome Orientated Teams (BOOTs). 3 teams focused on different parts of the business and a singular outcome instead of project Tetris.
Watched Killing Eve season 2 in very short order. Brilliant TV. Ludicrous in parts but still head shoulders above most things on TV. I also watched Detective Pikachu with the kids this morning. Surprisingly enjoyable. Ludicrous consistently but still…
Let’s hope next week will be less ludicrous all round.