Continuous Deployment

Foreword
Preface
Who This Book Is For
What You Need to Read This Book
What This Book Will Cover
Conventions Used in This Book
O’Reilly Online Learning
How to Contact Us
Acknowledgments

I. Continuous Deployment
1. Continuous Deployment
Deploying Every Few Months or Years
Deploying Every Few Days
Deploying Continuously
eXtreme Programming
If It Hurts, Do It More Often
DevOps
The Barrier Between Dev and Ops
Joining Dev and Ops
Automation, Automation, Automation
Continuous Integration
Continuous Delivery
A Final Gate to Production
One Step Further: Continuous Deployment
Implementation
Implications
Is It Dangerous?
Summary

2. Benefits
One-Piece Flow and Lean Manufacturing
Origins of Lean Manufacturing
Lean and Software Development
DORA Metrics
Throughput Metrics
Stability Metrics
Shifting Quality Left
How Continuous Deployments Force a Quality Shift to the Left
The Effects of Automating Quality Gates
Summary

3. The Mindset Shift
Defining a Change Versus Applying a Change
Hiding Work in Progress
Version Control Branches
Execution Branches
Distributed Systems
Contracts Between Systems
Contracts Between Paths to Production
Adding New Features: When Order Matters
Refactoring: When Timing Matters
A Deployment Is Not a Release
Releases
Deployments
Differences
Overlap
End-to-End Delivery Life Cycle
Without Continuous Deployment
With Continuous Deployment
Summary

4. You Must Be This Tall
Cross-Functional, Autonomous Teams
Fast Decision Making
Implementation Autonomy
Frequent Integration
Frequent Code Reviews
Automated Code Analysis
Test Automation
Zero-Downtime Deployments
Observability and Monitoring
Stakeholder Trust
How Do We Convince the Boss?
When Are We Ready?
Summary

5. Challenges
Systems That Are Sensitive to Deployments
Interruption of Long-Running Processes
Sticky Sessions
Invalidation of Client-Side Caches
Scaling Interruptions
A Constant Stream of Cold Instances
User-Installed Software
Desktop Applications
Mobile Apps
Appliances and Other Devices
Libraries and Frameworks
Regulated Industries
Cognitive Load
Overly Busy Path to Production
Inattention During Deployments
Breadth of Knowledge Required
Steep Onboarding Curve
Scheduling Development Work
Summary

II. Before Development
6. Slicing Upcoming Work
Horizontal Versus Vertical Slicing
Horizontal Slicing
Vertical Slicing
With Continuous Deployment
Effective Vertical Slicing
MVP
INVEST
Small Slices
Example: The Groceroo Company
The Feature: “Last-Minute Items”
The User Interface
The Admin Interface
The Problems of Implementing with Horizontal Slicing
Implementation with Vertical Slicing
Summary

7. Building for Production
Deployability Requirements
Hiding with Feature Toggles
Hiding with Expand and Contract
Hiding in Version Control Branch
Unhidden
Pausing the Pipeline
Testability Requirements
High-Level Automated Tests
Manual Exploratory Testing
Observability Requirements
Maintaining Logs and Metrics
Maintaining Dashboards and Alerts
Security Requirements
New User Input
Storing New Data
New Dependencies
New Infrastructure
Performance Requirements
New Network Requests
Data Size
Persistence Layer
A (More) Complete User Story Template
Example: Adding CFRs to Groceroo User Stories
User Story 1: Add Simple Carousel
User Story 2: Admin Area
User Story 3: “Add to Basket” Button
User Story 4: Quantity Selector
Summary

III. During Development
8. Adding New Features
Our User Story
The Groceroo Application
Current State
Target State
How Do We Get There?
Implementing with a Feature Toggle
Summary

9. Refactoring Live Features
Our Task
The Product Identifier System
The Problem
The Solution
Current State
Frontend
Backend
Persistence
Target State
Frontend
Backend
Persistence
How Do We Get There?
Expand and Contract
Multiple Layers: Inside Out
Implementing with Expand and Contract
Outer Expand Phase: The Product Table
Migrating the POST /product Endpoint
Migrating the GET /product Endpoint
Migrating the Basket Table
Outer Contract Phase: Cleaning Up the Product Table
Summary

10. Data and Data Loss
Our Task
The Problem
The Solution
Current State
Persistence
Backend
Target State
Persistence
Backend
How Do We Get There?
Failure Mode: Simultaneous Change
Failure Mode: Simple Expand and Contract
Solution: Temporary Database Trigger
Solution: Double-Write
Solution: Double-Read
Implementing with Double-Write
Deployment 1: Expand the Database Column
Deployment 2: Double-Write to Both Columns
Deployment 3: Synchronize the Data
Deployment 4: Migrate Write and Read to Both Columns
Deployment 5: Contract the Columns
Implementing with Double-Read
Deployment 1: Expand the Database Column
Deployment 2: Double-Read and Migrate Write the Columns
Deployment 3: Synchronize the Data
Deployment 4: Migrate Read the Column
Deployment 5: Contract the Columns
NoSQL
Migrate on Read
Custom Batch Update
Conclusion on NoSQL
Summary

IV. After Development
11. Testing in Production
Why You Should Test in Production
Data Volume Accuracy
Data Shape Accuracy
Realistic Request Patterns
Realistic Incoming Traffic Volume
Realistic Outgoing Traffic Volume
Realistic Size and Number of Servers
Realistic Application Configuration
Realistic Network Configuration
Real Version of Other Teams’ Services
Real Version of Third-Party Services
Lower Cost
Better Data Hygiene
How to Test in Production
Feature Toggle Activation Strategies
Challenges
Life After Staging
Summary

12. Releasing
Antipattern: Big Bang Releases
Comparing Before Versus After States
Comparing Simultaneous States
Antipattern: Partial Releases Through Partial Deployments
Using Feature Flags for Releases
Coordinating Feature Flag Releases in Distributed Systems
Antipattern: Independent Flag State for Each Service
Propagating Flag State Down the Call Chain
Centralized Feature Flag State
Canary Releases
By Traffic Percentage
By Device
By Country
By User Segment
A/B Testing
Analytics
Experiment Best Practices
Schedule Delivery Around Experiments
Types of A/B Tests
Do You Need a Canary Release or an A/B Test?
Summary
Conclusion and Next Steps

V. Case Studies
Case Study: AutoScout24
AutoScout24’s Context
Adoption of Continuous Deployment at AutoScout24
Overcoming Organizational Obstacles
Overcoming Technical Obstacles
AutoScout24’s Implementation of Continuous Deployment
How AutoScout24 Makes Continuous Deployment Safe
How AutoScout24 Supports Junior Engineers
Case Study: OTTO
OTTO’s Context
Adoption of Continuous Deployment at OTTO
Working with Organizational Barriers
Working with Technical Barriers
OTTO’s Implementation of Continuous Deployment
OTTO’s Path to Production
How OTTO Makes Continuous Deployment Safe
How OTTO Supports Junior Engineers with Continuous Deployment
Links and Resources
Case Study: N26
N26’s Context
Adoption of Continuous Deployment at N26
Working Within Industry Regulations
Working with Technical Barriers
N26’s Implementation of Continuous Deployment
How N26 Makes Continuous Deployment Safe
How N26 Supports Junior Engineers
Links and Resources
Case Study: ClimatePartner
ClimatePartner’s Context
Adoption of Continuous Deployment at ClimatePartner
ClimatePartner’s Implementation of Continuous Deployment
How ClimatePartner Makes Continuous Deployment Safe
How ClimatePartner Supports Junior Engineers with Continuous Deployment
Case Study: Motability Operations
Motability Operations’ Context
Adoption of Continuous Deployment at Motability Operations
Working with Organizational Barriers
Working with technical barriers
Motability Operations’ Implementation of Continuous Deployment
How Motability Makes Continuous Deployment Safe
How Motability Operations Supports Teams
Case Study: REA Group
REA’s Context
Adoption of Continuous Deployment at REA
Overcoming organizational barriers
REA’s Implementation of Continuous Deployment
How REA Makes Continuous Deployment Safe
How REA Supports Junior Engineers with Continuous Deployment
Case Study: Maze
Maze’s Context
Adoption of Continuous Deployment at Maze
Overcoming Organizational Challenges
Overcoming Technical Challenges
Maze’s Implementation of Continuous Deployment
How Maze Makes Continuous Deployment Safe
How Maze Supports Junior Engineers with Continuous Deployment
Case Study: TravelPerk
TravelPerk’s Context
Adoption of Continuous Deployment at TravelPerk
TravelPerk’s Implementation of Continuous Deployment
How TravelPerk Makes Continuous Deployment Safe
How TravelPerk Onboards Engineers to Continuous Deployment

Index
About the Author