Easing into continuous deployment

Transcript

1. Continuous
   Deployment
   Chris Keathley / @ChrisKeathley / [email protected]
   

   View Slide

2. I work with a
   distributed team
   

   View Slide

3. I work with a
   distributed team
   

   View Slide

4. View Slide

5. View Slide

6. View Slide

7. View Slide

8. View Slide

9. Warehouse
   

   View Slide

10. Warehouse
    API
    

    View Slide

11. Warehouse
    API
    Apps
    

    View Slide

12. The problem
    

    View Slide

13. Slow Iteration Cycle
    Deployment Deployment
    2 weeks
    

    View Slide

14. Slow Iteration Cycle
    Deployment Deployment Deployment
    2 weeks
    2 weeks
    

    View Slide

15. Slow Iteration Cycle
    Deployment Deployment Deployment
    3 weeks
    

    View Slide

16. Slow Iteration Cycle
    Deployment Deployment Deployment
    Hopefully someday
    

    View Slide

17. Large PRs
    

    View Slide

18. Unsure about state of the application
    

    View Slide

19. Unsure about state of the application
    

    View Slide

20. Unsure about state of the application
    

    View Slide

21. Unsure about state of the application
    

    View Slide

22. Unsure about state of the application
    

    View Slide

23. Rollbacks
    are a scam
    

    View Slide

24. View Slide

25. View Slide

26. Data Migration
    

    View Slide

27. Data Migration
    ?
    

    View Slide

28. Don’t do this
    

    View Slide

29. always Move forward
    

    View Slide

30. always Move forward
    

    View Slide

31. always Move forward
    

    View Slide

32. The goal should
    never be to roll
    back a deployment
    

    View Slide

33. The goal is to minimize
    the damage done by
    any given deployment
    

    View Slide

34. There are bugs
    in your system
    

    View Slide

35. Solutions
    

    View Slide

36. We needed to
    deploy more often
    

    View Slide

37. So we did
    

    View Slide

38. Automated
    Deployment
    

    View Slide

39. What do you
    deploy?
    

    View Slide

40. Commit Sha
    

    View Slide

41. View Slide

42. Jars
    

    View Slide

43. Artifacts
    

    View Slide

44. Git Tags
    

    View Slide

45. Containers
    

    View Slide

46. Your App
    

    View Slide

47. Your App
    Server
    

    View Slide

48. View Slide

49. Master
    Branch
    How we merge our code
    PR
    

    View Slide

50. CI
    Github Registry
    Container
    Slack
    PR
    Notification
    

    View Slide

51. CI Kubernetes
    Deploy
    Auto-deploy Green builds of master
    

    View Slide

52. CI Kubernetes
    Deploy
    Auto-deploy Green builds of master
    Service A
    Service B
    

    View Slide

53. CI Kubernetes
    Deploy
    Auto-deploy Green builds of master
    Service B
    

    View Slide

54. CI Kubernetes
    Deploy
    Auto-deploy Green builds of master
    Service B
    Service A
    

    View Slide

55. CI Kubernetes
    Deploy
    Auto-deploy Green builds of master
    Service A
    

    View Slide

56. CI Kubernetes
    Deploy
    Auto-deploy Green builds of master
    Service A
    Service B
    

    View Slide

57. Tests
    Metrics
    &
    

    View Slide

58. Integration Tests
    +
    Property Tests
    

    View Slide

59. Integration Tests
    TEst App
    DB
    Service
    

    View Slide

60. Modeling Users as FSMs
    logged_out logged_in
    login
    logout
    vote
    

    View Slide

61. Property Tests
    Add Todo Edit Todo
    Delete
    Todo
    

    View Slide

62. Property Tests
    Add Todo Edit Todo
    Delete
    Todo
    

    View Slide

63. Property Tests
    Add Todo Edit Todo
    Delete
    Todo
    

    View Slide

64. Property Tests
    Add Todo Edit Todo
    Delete
    Todo
    

    View Slide

65. Generate Commands
    

    View Slide

66. Generated Commands
    [{:add_todo, “Test Todo”, 1},
    {:edit_todo, "Edited", 2},
    {:delete_todo, "", 1},
    {:add_todo, “New Todo", 3},
    {:delete_todo, "", 2}
    {:edit_todo, “Edited Todo”, 2}]
    

    View Slide

67. Generate Commands
    

    View Slide

68. Generate Commands
    

    View Slide

69. Generate Commands
    

    View Slide

70. Generate Commands
    

    View Slide

71. Generate Commands
    

    View Slide

72. Generated Commands
    [{:add_todo, “Test Todo”, 1},
    {:edit_todo, "Edited", 2},
    {:delete_todo, "", 1},
    {:add_todo, “New Todo", 3},
    {:delete_todo, "", 2}
    {:edit_todo, “Edited Todo”, 2}]
    

    View Slide

73. Generated Commands
    [{:add_todo, “Test Todo”, 1},
    {:delete_todo, "", 2}]
    [{:add_todo, “Test Todo”, 1},
    {:edit_todo, "Edited", 2},
    {:delete_todo, "", 1},
    {:add_todo, “New Todo", 3},
    {:delete_todo, "", 2}
    {:edit_todo, “Edited Todo”, 2}]
    

    View Slide

74. Prometheus
    Service
    A
    Grafana
    Service
    B
    Service
    C
    

    View Slide

75. Prometheus
    Service
    A Grafana
    Service
    B
    Service
    C
    Slack
    

    View Slide

76. # Alert for any instance that have a 95th percentile > 200ms.
    ALERT APIHighRequestLatency
    IF api_http_request_latencies_second{quantile="0.95"} > 0.2
    FOR 5m
    ANNOTATIONS {
    summary = "High request latency on {{ $labels.instance }}",
    description = "{{ $labels.instance }}
    has a median request latency above 1s (current value: {{ $value }}s)",
    }
    

    View Slide

77. Track “Business” Metrics
    

    View Slide

78. View Slide

79. Feature releases
    and flags
    

    View Slide

80. View Slide

81. Features aren’t
    all or nothing
    

    View Slide

82. Features
    !=
    Deployments
    

    View Slide

83. Deployment
    

    View Slide

84. Deployment Features
    

    View Slide

85. Deployment Features
    

    View Slide

86. User
    

    View Slide

87. User
    staff?(user) == true
    

    View Slide

88. User
    staff?(user) == false
    

    View Slide

89. User
    staff?(user) == false
    

    View Slide

90. defmodule MyApp.FeatureFlags do
    alias MyApp.User
    def foo_enabled?(%User{staff: is_staff}), do: is_staff
    def foo_enabled?(_), do: false
    def bar_enabled?(%User{staff: is_staff}), do: is_staff
    def bar_enabled?(_), do: false
    end
    

    View Slide

91. Browser
    Feature
    Service
    

    View Slide

92. Feature
    Service
    Feature
    Service
    Feature
    Service
    

    View Slide

93. Feature
    Service
    Feature
    Service
    Feature
    Service
    

    View Slide

94. Feature
    Service
    Feature
    Service
    Feature
    Service
    

    View Slide

95. You have updates ready! Reset
    

    View Slide

96. View Slide

97. With larger Traffic
    numbers you could
    use percentages
    

    View Slide

98. Alchemy
    

    View Slide

99. “Transmute lead
    code into gold in
    production”
    

    View Slide

100. Prior Art: https://github.com/github/scientist
     

     View Slide

101. Users_Controller DB
     User.all
     

     View Slide

102. DB
     User.all
     UserService.all
     

     View Slide

103. User.all UserService.all
     ==
     

     View Slide

104. def index(conn) do
     users = old_query()
     render(conn, "index.json", users: users)
     end
     

     View Slide

105. def index(conn) do
     users =
     experiment("users-query")
     |> control(&old_query/0)
     |> candidate(&new_query/0)
     |> run
     render(conn, "index.json", users: users)
     end
     

     View Slide

106. def index(conn) do
     users =
     experiment("users-query")
     |> control(&old_query/0)
     |> candidate(&new_query/0)
     |> candidate(&fancy_query/0)
     |> run
     render(conn, "index.json", users: users)
     end
     

     View Slide

107. 1) Shuffles test order
     2) Runs Each test in parallel
     3) exports the data
     Alchemy
     

     View Slide

108. DB
     User.all
     UserService.all
     Control
     Candidate
     Control
     UserController
     

     View Slide

109. View Slide

110. 1) Do the results match?
     2) How long does each test take to return?
     Measure
     

     View Slide

111. No more
     cutovers
     

     View Slide

112. DB
     User.all
     UserService.all
     

     View Slide

113. DB
     User.all
     UserService.all
     User
     service
     

     View Slide

114. Migrations
     

     View Slide

115. http://blog.datomic.com/2017/01/the-ten-rules-of-schema-growth.html
     

     View Slide

116. DB
     Schema
     App
     Application Coupling
     

     View Slide

117. Your application knows
     about your schema
     

     View Slide

118. Lets remove a column
     

     View Slide

119. Lets remove a column
     1) all application code needs to stop using that column
     

     View Slide

120. Lets remove a column
     1) all application code needs to stop using that column
     2) Update all ETL processes
     

     View Slide

121. Lets remove a column
     1) all application code needs to stop using that column
     2) Update all ETL processes
     3) Update Reporting
     

     View Slide

122. Lets remove a column
     1) all application code needs to stop using that column
     2) Update all ETL processes
     3) Update Reporting
     4) Remove the column
     

     View Slide

123. Lets remove a column
     1) all application code needs to stop using that column
     2) Update all ETL processes
     3) Update Reporting
     4) Remove the column
     Split all of these up
     

     View Slide

124. Lets Add a column
     

     View Slide

125. Lets Add a column
     1) Add the column
     

     View Slide

126. Lets Add a column
     1) Add the column
     2) Eventually start using it
     

     View Slide

127. Prefer Additive Migrations
     

     View Slide

128. CI Kubernetes
     Deploy
     Auto-deploy Green builds of master
     

     View Slide

129. CI Kubernetes
     Deploy
     Auto-deploy Green builds of master
     Migration
     

     View Slide

130. CI Kubernetes
     Deploy
     Auto-deploy Green builds of master
     Migration DB
     

     View Slide

131. Chat-Ops
     

     View Slide

132. View Slide

133. Chat is…
     

     View Slide

134. Chat is…
     Centralized
     

     View Slide

135. Chat is…
     Centralized
     Transparent
     

     View Slide

136. Chat is…
     Centralized
     Transparent
     Open
     

     View Slide

137. Try to do operational tasks
     in chat
     

     View Slide

138. View Slide

139. defmodule Hedwig.Responders.Ping do
     use Hedwig.Responder
     @usage """
     hedwig: ping - Responds with 'pong'
     """
     respond ~r/ping$/i, msg do
     reply msg, "pong"
     end
     end
     

     View Slide

140. View Slide

141. Generate
     grafana graphs
     

     View Slide

142. View Slide

143. Deploy
     

     View Slide

144. View Slide

145. Team
     Building
     

     View Slide

146. Conclusion
     

     View Slide

147. These are tools
     at our disposal
     

     View Slide

148. Deploy more
     often, safely
     

     View Slide

149. Thanks
     Chris Keathley / @ChrisKeathley / [email protected]
     

     View Slide