This is far more complex than the most complex CD pipeline I have ever had! Just cos it's complex, doesn't mean it's over-engineered though. Given the blast radius, I'm glad they do releases carefully and safely.
If you missed @clare_liguori's Continuous Delivery session this week (like I did) then good news, it's available on-demand now 🎊
https://t.co/78b8sI2hHs
And here's my play-by-play for the session
🧵...
This is far more complex than the most complex CD pipeline I have ever had! Just cos it's complex, doesn't mean it's over-engineered though. Given the blast radius, I'm glad they do releases carefully and safely.
More details in this post:
https://t.co/l0f44tunJD
Personally, I love what you can do with @LaunchDarkly such a slick control panel and super easy to use ❤️
It can get expensive though, because you're hitting DynamoDB a lot!
"One box used to mean one VM, but over time it has also come to mean one container or a small percentage of Lambda function invocations"
ha, so they use weighted-alias for microservices that run on Lambda too, starting at 10% at first
And within each region, apply the same pattern to AZs too.
First few waves deploy to one region and one AZ at a time, later waves (after you build some confidence) deploy to multiple regions in parallel
Deployments are staggered, so multiple deployments can be in motion and at different stages at once.
How does this affect rollback I wonder 🤔 e.g. if v1 deployment craps out at wave 5 and triggers rollback, what of wave 1 which is deploying v5?
Have thresholds on a bunch of metrics (regional, zonal aggregates as well as per-box) to trigger automatic rollbacks.
haha, been there... once had a slow memory leak that showed up 2 weeks after a deployment 🤦♂️
1. auto-rollback only in regions/zones that tripped the threshold
2. eng has to decide whether to roll back the whole thing or retry
e.g. something else could have happened in the offending region, which is why thresholds were crossed
Or, they can roll forward and push out a v3 deployment instead, which fixes whatever problem that got picked up in that failed region.
Anyway, Claire moves onto how to design your changes so they can be rolled back automatically.
As much as possible, make backward-compatible changes 💯
This mirrors a lot of database migrations when you move from one database to another one and you can't do it with downtime.
It's not an option at AWS scale of course, but you're not AWS.
And they basically practice the one-box deployment in the gamma (production-like) environment.
If you wanna catch the session yourself, here's the on-demand video: https://t.co/78b8sI2hHs
More from Software
forgive my indulgence but 2020's been a big year for @shmuplations, so here's a look back at everything that went up over the last twelve months—there's a lot of stuff I'm sure you all read & other things you'd be forgiven for missing, so let's recap (thread)
the year kicked off with shmuplations' first big video project: a subtitled translation of a 2016 NHK documentary on the 30th anniversary of Dragon Quest which features interviews with Yuji Horii, Koichi Nakamura, Akira Toriyama, and Koichi Sugiyama https://t.co/JCWA15RTlx
following DQ30 was one of the most popular articles of the year: an assortment of interviews with composers Yuzo Koshiro and Motohiro Kawashima concerning the music of Streets of Rage 1, 2 & 3 https://t.co/QUtyC9W12Z their comments on SoR3 in particular were full of gems
Game Designers: The Next Generation profiled six potential successors to the likes of Shigeru Miyamoto & Hironobu Sakaguchi, some of who you may recognise: Kazuma Kaneko, Takeshi Miyaji (1966-2011), Noboru Harada, Kan Naitou, Takashi Tokita & Ryoji Amano https://t.co/lWZU3PLvwX
from the 2010 Akumajou Dracula Best Music Collections Box, a subbed video feature on long-time Castlevania composer Michiru Yamane https://t.co/NMJe4ROozR sadly, Chiruru has since passed; Yamane wrote these albums in his honor
https://t.co/orlgPTDsKK
https://t.co/QnQl8KI9IX
the year kicked off with shmuplations' first big video project: a subtitled translation of a 2016 NHK documentary on the 30th anniversary of Dragon Quest which features interviews with Yuji Horii, Koichi Nakamura, Akira Toriyama, and Koichi Sugiyama https://t.co/JCWA15RTlx
following DQ30 was one of the most popular articles of the year: an assortment of interviews with composers Yuzo Koshiro and Motohiro Kawashima concerning the music of Streets of Rage 1, 2 & 3 https://t.co/QUtyC9W12Z their comments on SoR3 in particular were full of gems
Game Designers: The Next Generation profiled six potential successors to the likes of Shigeru Miyamoto & Hironobu Sakaguchi, some of who you may recognise: Kazuma Kaneko, Takeshi Miyaji (1966-2011), Noboru Harada, Kan Naitou, Takashi Tokita & Ryoji Amano https://t.co/lWZU3PLvwX
from the 2010 Akumajou Dracula Best Music Collections Box, a subbed video feature on long-time Castlevania composer Michiru Yamane https://t.co/NMJe4ROozR sadly, Chiruru has since passed; Yamane wrote these albums in his honor
https://t.co/orlgPTDsKK
https://t.co/QnQl8KI9IX