Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Consistent, Distributed Elixir

Consistent, Distributed Elixir

An explanation of consistency guarantees in erlang and what we might be able to do with a consistency protocol like Raft.

Chris Keathley

February 23, 2018
Tweet

More Decks by Chris Keathley

Other Decks in Programming

Transcript

  1. Consistent,
    Distributed
    Elixir
    Chris Keathley / @ChrisKeathley / [email protected]

    View Slide

  2. I have a problem…

    View Slide

  3. View Slide

  4. “Phoenix is not your application”

    View Slide

  5. I don’t know what my application
    is any more

    View Slide

  6. Elixir is
    awesome

    View Slide

  7. Processes

    View Slide

  8. Problem:
    “We need to limit access
    to an external resource”

    View Slide

  9. Solution:
    “Lets just use processes!”

    View Slide

  10. Global lock

    View Slide

  11. :unlocked
    Global lock

    View Slide

  12. :unlocked
    Global lock
    Client

    View Slide

  13. Global lock
    Client :lock :unlocked

    View Slide

  14. Global lock
    Client :lock :unlocked

    View Slide

  15. Global lock
    Client :locked

    View Slide

  16. Global lock
    Client :ok :locked

    View Slide

  17. Global lock
    Client :ok :locked

    View Slide

  18. Global lock
    Client :locked

    View Slide

  19. Global lock
    Client :unlock :locked

    View Slide

  20. :locked
    Global lock
    Client :unlock

    View Slide

  21. :locked
    Global lock
    Client

    View Slide

  22. :unlocked
    Global lock
    Client

    View Slide

  23. :unlocked
    Global lock
    Client :ok

    View Slide

  24. :unlocked
    Global lock
    Client :ok

    View Slide

  25. :unlocked
    Global lock
    Client

    View Slide

  26. :unlocked
    Global lock
    Client
    Client

    View Slide

  27. :unlocked
    Global lock
    Client
    Client

    View Slide

  28. :unlocked
    Global lock
    Client
    Client

    View Slide

  29. :unlocked
    Global lock
    Client
    Client

    View Slide

  30. Global lock
    Client
    Client
    :locked

    View Slide

  31. Global lock
    Client
    Client
    :locked

    View Slide

  32. Global lock
    Client
    Client
    :locked

    View Slide

  33. Global lock
    Client
    Client
    :locked

    View Slide

  34. Global lock
    Client
    Client
    :locked

    View Slide

  35. Global lock
    Client
    Client
    :locked

    View Slide

  36. Global lock
    Client
    Client
    :locked

    View Slide

  37. Global lock
    Client
    Client
    :locked

    View Slide

  38. Global lock
    Client
    Client
    :locked

    View Slide

  39. Global lock
    Client
    Client
    :locked

    View Slide

  40. Global lock
    Client
    Client
    :locked

    View Slide

  41. Global lock
    Client
    Client
    :locked

    View Slide

  42. Global lock
    Client
    Client
    :unlocked

    View Slide

  43. Global lock
    Client
    Client
    :unlocked

    View Slide

  44. Global lock
    Client
    Client
    :unlocked

    View Slide

  45. Global lock
    Client
    Client
    :unlocked

    View Slide

  46. Global lock
    Client
    Client
    :locked

    View Slide

  47. Global lock
    Client
    Client
    :locked

    View Slide

  48. Global lock
    Client
    Client
    :locked

    View Slide

  49. defmodule Demo.Lock do
    use GenServer
    end

    View Slide

  50. defmodule Demo.Lock do
    use GenServer
    def init(:ok) do
    state = {:unlocked, nil}
    {:ok, state}
    end
    end

    View Slide

  51. defmodule Demo.Lock do
    use GenServer
    def init(:ok) do
    state = {:unlocked, nil}
    {:ok, state}
    end
    def handle_call({:lock, client}, _from, {:unlocked, nil}) do
    {:reply, :ok, {:locked, client}}
    end
    end

    View Slide

  52. defmodule Demo.Lock do
    use GenServer
    def init(:ok) do
    state = {:unlocked, nil}
    {:ok, state}
    end
    def handle_call({:lock, client}, _from, {:unlocked, nil}) do
    {:reply, :ok, {:locked, client}}
    end
    def handle_call({:lock, client}, _from, {:locked, other_client}) do
    {:reply, :error, {:locked, other_client}}
    end
    end

    View Slide

  53. defmodule Demo.Lock do
    use GenServer
    def init(:ok) do
    state = {:unlocked, nil}
    {:ok, state}
    end
    def handle_call({:lock, client}, _from, {:unlocked, nil}) do
    {:reply, :ok, {:locked, client}}
    end
    def handle_call({:lock, client}, _from, {:locked, other_client}) do
    {:reply, :error, {:locked, other_client}}
    end
    def handle_call({:unlock, client}, _from, {:locked, client}) do
    {:reply, :ok, {:unlocked, nil}}
    end
    def handle_call({:unlock, client}, _from, {:locked, other_client}) do
    {:reply, :error, {:locked, other_client}}
    end
    end

    View Slide

  54. Global lock

    View Slide

  55. Problem:
    “We need to run multiple Nodes”

    View Slide

  56. Multiple Nodes

    View Slide

  57. Multiple Nodes Node

    View Slide

  58. Node
    Multiple Nodes

    View Slide

  59. Node
    Node
    Multiple Nodes

    View Slide

  60. Node
    Client
    Client
    Multiple Nodes
    Node

    View Slide

  61. Node
    Node
    Client
    Multiple Nodes
    Client

    View Slide

  62. Node
    Node
    Client
    Multiple Nodes
    Client

    View Slide

  63. Node
    Node
    Client
    Multiple Nodes
    Client

    View Slide

  64. Node
    Node
    Client
    Multiple Nodes
    Client

    View Slide

  65. Node
    Node
    Client
    Multiple Nodes
    Client

    View Slide

  66. Node
    Node
    Client
    Multiple Nodes
    Client

    View Slide

  67. Node
    Node
    Client
    Multiple Nodes
    Client
    This is bad

    View Slide

  68. Solution:
    “Lets just use a global process”

    View Slide

  69. GenServer.start_link(Lock, :ok, name: Lock)

    View Slide

  70. GenServer.start_link(Lock, :ok, name: {:global, Lock})

    View Slide

  71. Solution: Global Process Node
    Node
    Client
    Client

    View Slide

  72. Solution: Global Process Node
    Node
    Client
    Client
    Remove this

    View Slide

  73. Solution: Global Process Node
    Node
    Client
    Client
    Global

    View Slide

  74. Solution: Global Process Node
    Node
    Client
    Client

    View Slide

  75. Solution: Global Process Node
    Node
    Client
    Client

    View Slide

  76. Solution: Global Process Node
    Node
    Client
    Client

    View Slide

  77. Solution: Global Process Node
    Node
    Client
    Client

    View Slide

  78. Solution: Global Process Node
    Node
    Client
    Client

    View Slide

  79. Solution: Global Process Node
    Node
    Client
    Client

    View Slide

  80. Solution: Global Process Node
    Node
    Client
    Client

    View Slide

  81. Solution: Global Process Node
    Node
    Client
    Client

    View Slide

  82. Solution: Global Process Node
    Node
    Client
    Client
    What if this goes away?

    View Slide

  83. Solution: Global Process Node
    Client
    Client

    View Slide

  84. Solution: Global Process Node
    Client
    Client

    View Slide

  85. Solution: Global Process Node
    Client
    Client
    Start a new lock process

    View Slide

  86. Solution: Global Process Node
    Client
    Client
    Start a new lock process

    View Slide

  87. Solution: Global Process Node
    Client
    Client

    View Slide

  88. Solution: Global Process Node
    Client
    Client

    View Slide

  89. Solution: Global Process Node
    Client
    Client

    View Slide

  90. Solution: Global Process Node
    Client
    Client

    View Slide

  91. Problem:
    “what if the node isn’t really down?”

    View Slide

  92. Solution: Global Process Node
    Node
    Client
    Client

    View Slide

  93. Solution: Global Process Node
    Node
    Client
    Client

    View Slide

  94. Solution: Global Process Node
    Node
    Client
    Client
    Partition

    View Slide

  95. Solution: Global Process Node
    Node
    Client
    Client

    View Slide

  96. Solution: Global Process Node
    Node
    Client
    Client

    View Slide

  97. Solution: Global Process Node
    Node
    Client
    Client

    View Slide

  98. Solution: Global Process Node
    Node
    Client
    Client
    Guess it must be down

    View Slide

  99. Solution: Global Process Node
    Node
    Client
    Client
    Start a new lock process

    View Slide

  100. Solution: Global Process Node
    Node
    Client
    Client
    Start a new lock process

    View Slide

  101. Solution: Global Process Node
    Node
    Client
    Client

    View Slide

  102. Solution: Global Process Node
    Node
    Client
    Client

    View Slide

  103. Solution: Global Process Node
    Node
    Client
    Client

    View Slide

  104. Solution: Global Process Node
    Node
    Client
    Client

    View Slide

  105. Solution: Global Process Node
    Node
    Client
    Client
    This Lock still exists

    View Slide

  106. Solution: Global Process Node
    Node
    Client
    Client

    View Slide

  107. Solution: Global Process Node
    Node
    Client
    Client

    View Slide

  108. Solution: Global Process Node
    Node
    Client
    Client

    View Slide

  109. Solution: Global Process Node
    Node
    Client
    Client

    View Slide

  110. Solution: Global Process Node
    Node
    Client
    Client

    View Slide

  111. Solution: Global Process Node
    Node
    Client
    Client

    View Slide

  112. Solution: Global Process Node
    Node
    Client
    Client

    View Slide

  113. Solution: Global Process Node
    Node
    Client
    Client
    This is bad

    View Slide

  114. Problem:
    “What happens when the partition
    heals?”

    View Slide

  115. Solution: Global Process Node
    Node
    Client
    Client

    View Slide

  116. Solution: Global Process Node
    Node
    Client
    Client
    Heals

    View Slide

  117. Solution: Global Process Node
    Node
    Client
    Client

    View Slide

  118. Solution: Global Process Node
    Node
    Client
    Client
    Who should win?

    View Slide

  119. Solution: Global Process Node
    Node
    Client
    Client
    This is bad

    View Slide

  120. Solution:
    “Lets just use The Database”

    View Slide

  121. Solution: Databases Node
    Node
    Client
    Client

    View Slide

  122. Solution: Databases
    Node
    Node
    Client
    Client

    View Slide

  123. Solution: Databases
    Node
    Node
    Client
    Client Redis

    View Slide

  124. Solution:
    “Lets have a way to consistently
    manage state in elixir”

    View Slide

  125. When a network is partitioned
    you can either be available or
    consistent
    CAP Theorem

    View Slide

  126. “Every request receives a response
    without guarantee that it contains the
    most recent write”
    Available

    View Slide

  127. “Every read receives the most recent
    write or it errors”
    Consistent

    View Slide

  128. AP CP
    Available during partitions Consistent during partitions

    View Slide

  129. Available

    View Slide

  130. Problem:
    “We need to keep track of
    counts”

    View Slide

  131. Counters Node
    Client
    Node
    Client

    View Slide

  132. Counters Node
    Client
    Node
    Client +0
    +0

    View Slide

  133. Counters Node
    Client
    Node
    Client +0
    +0
    Store additions

    View Slide

  134. Counters Node
    Client
    Node
    Client +0
    +0

    View Slide

  135. Counters Node
    Client
    Node
    Client +2 +0
    +0

    View Slide

  136. Counters Node
    Client
    Node
    Client +2 +0
    +0

    View Slide

  137. Counters Node
    Client
    Node
    Client +0
    +0
    +2

    View Slide

  138. Counters Node
    Client
    Node
    Client
    +2
    +0
    +0
    +2

    View Slide

  139. Counters Node
    Client
    Node
    Client
    +2
    +0
    +0
    +2

    View Slide

  140. Counters Node
    Client
    Node
    Client +0
    +2
    +0
    +2

    View Slide

  141. Counters Node
    Client
    Node
    Client +0
    +2
    +0
    +2
    :read

    View Slide

  142. Counters Node
    Client
    Node
    Client +0
    +2
    +0
    +2
    :read

    View Slide

  143. Counters Node
    Client
    Node
    Client +0
    +2
    +0
    +2
    2

    View Slide

  144. Counters Node
    Client
    Node
    Client +0
    +2
    +0
    +2
    2

    View Slide

  145. Counters Node
    Client
    Node
    Client +0
    +2
    +0
    +2

    View Slide

  146. Counters Node
    Client
    Node
    Client +0
    +2
    +0
    +2

    View Slide

  147. Counters Node
    Client
    Node
    Client +0
    +2
    +0
    +2
    +3

    View Slide

  148. Counters Node
    Client
    Node
    Client +0
    +2
    +0
    +2
    +3

    View Slide

  149. Counters Node
    Client
    Node
    Client +0
    +2
    +0
    +2
    +3

    View Slide

  150. Counters Node
    Client
    Node
    Client +0
    +2
    +0
    +2
    +3
    +3

    View Slide

  151. Counters Node
    Client
    Node
    Client +0
    +2
    +0
    +2
    +3
    +3

    View Slide

  152. Counters Node
    Client
    Node
    Client +0
    +2
    +0
    +2
    +3
    :read

    View Slide

  153. Counters Node
    Client
    Node
    Client +0
    +2
    +0
    +2
    +3
    :read

    View Slide

  154. Counters Node
    Client
    Node
    Client +0
    +2
    +0
    +2
    +3
    2

    View Slide

  155. Counters Node
    Client
    Node
    Client +0
    +2
    +0
    +2
    +3
    2

    View Slide

  156. Counters Node
    Client
    Node
    Client +0
    +2
    +0
    +2
    +3

    View Slide

  157. Counters Node
    Client
    Node
    Client +0
    +2
    +0
    +2
    +3
    +5

    View Slide

  158. Counters Node
    Client
    Node
    Client +0
    +2
    +0
    +2
    +3
    +5

    View Slide

  159. Counters Node
    Client
    Node
    Client +0
    +2
    +0
    +2
    +3

    View Slide

  160. Counters Node
    Client
    Node
    Client +0
    +2
    +0
    +2
    +3
    +5

    View Slide

  161. Counters Node
    Client
    Node
    Client +0
    +2
    +0
    +2
    +3
    +5
    Inconsistent

    View Slide

  162. Counters Node
    Client
    Node
    Client +0
    +2
    +0
    +2
    +3
    +5

    View Slide

  163. Counters Node
    Client
    Node
    Client +0
    +2
    +0
    +2
    +3
    +5

    View Slide

  164. Counters Node
    Client
    Node
    Client +0
    +2
    +0
    +2
    +3
    +5

    View Slide

  165. Counters Node
    Client
    Node
    Client +0
    +2
    +0
    +2
    +3
    +5

    View Slide

  166. Counters Node
    Client
    Node
    Client
    +0
    +2
    +0
    +2
    +3
    +5
    +5
    +3

    View Slide

  167. Sometimes you’re
    going to be wrong
    (And thats ok)

    View Slide

  168. Phoenix
    presence

    View Slide

  169. Some problems need
    consistency
    Distributed Locking
    Databases
    Distributed scheduling and coordination
    Configuration and metadata storage
    Transactions

    View Slide

  170. Consistent

    View Slide

  171. Node
    Partitions in consistent systems
    Node
    Node

    View Slide

  172. Node
    Partitions in consistent systems
    Node
    Node

    View Slide

  173. Node
    Partitions in consistent systems
    Node
    Node

    View Slide

  174. Node
    Partitions in consistent systems
    Node
    Node

    View Slide

  175. Node
    Partitions in consistent systems
    Node
    Node

    View Slide

  176. Consensus

    View Slide

  177. Paxos

    View Slide

  178. Leslie Lamport

    View Slide

  179. Leslie Lamport

    View Slide

  180. Paxos (but simpler)

    View Slide

  181. Raft (this time a lot simpler)

    View Slide

  182. Raft (this time a lot simpler)

    View Slide

  183. Problem:
    “We need to limit access
    to an external resource”

    View Slide

  184. defmodule Demo.Lock do
    use GenServer
    def init(:ok) do
    {:ok, {:unlocked, nil}}
    end
    def handle_call({:lock, client}, _from, {:unlocked, nil}) do
    {:reply, :ok, {:locked, client}}
    end
    def handle_call({:lock, client}, _from, {:locked, other_client}) do
    {:reply, :error, {:locked, other_client}}
    end
    def handle_call({:unlock, client}, _from, {:locked, client}) do
    {:reply, :ok, {:unlocked, nil}}
    end
    def handle_call({:unlock, client}, _from, {:locked, other_client}) do
    {:reply, :error, {:locked, other_client}}
    end
    end

    View Slide

  185. defmodule Demo.Lock do
    use Raft.StateMachine
    def init(:ok) do
    {:ok, {:unlocked, nil}}
    end
    def handle_call({:lock, client}, _from, {:unlocked, nil}) do
    {:reply, :ok, {:locked, client}}
    end
    def handle_call({:lock, client}, _from, {:locked, other_client}) do
    {:reply, :error, {:locked, other_client}}
    end
    def handle_call({:unlock, client}, _from, {:locked, client}) do
    {:reply, :ok, {:unlocked, nil}}
    end
    def handle_call({:unlock, client}, _from, {:locked, other_client}) do
    {:reply, :error, {:locked, other_client}}
    end
    end

    View Slide

  186. defmodule Demo.Lock do
    use Raft.StateMachine
    def init(:ok) do
    {:unlocked, nil}
    end
    def handle_call({:lock, client}, _from, {:unlocked, nil}) do
    {:reply, :ok, {:locked, client}}
    end
    def handle_call({:lock, client}, _from, {:locked, other_client}) do
    {:reply, :error, {:locked, other_client}}
    end
    def handle_call({:unlock, client}, _from, {:locked, client}) do
    {:reply, :ok, {:unlocked, nil}}
    end
    def handle_call({:unlock, client}, _from, {:locked, other_client}) do
    {:reply, :error, {:locked, other_client}}
    end
    end

    View Slide

  187. defmodule Demo.Lock do
    use Raft.StateMachine
    def init(:ok) do
    {:unlocked, nil}
    end
    def handle_write({:lock, client}, _from, {:unlocked, nil}) do
    {:reply, :ok, {:locked, client}}
    end
    def handle_call({:lock, client}, _from, {:locked, other_client}) do
    {:reply, :error, {:locked, other_client}}
    end
    def handle_call({:unlock, client}, _from, {:locked, client}) do
    {:reply, :ok, {:unlocked, nil}}
    end
    def handle_call({:unlock, client}, _from, {:locked, other_client}) do
    {:reply, :error, {:locked, other_client}}
    end
    end

    View Slide

  188. defmodule Demo.Lock do
    use Raft.StateMachine
    def init(:ok) do
    {:unlocked, nil}
    end
    def handle_write({:lock, client}, {:unlocked, nil}) do
    {:reply, :ok, {:locked, client}}
    end
    def handle_call({:lock, client}, _from, {:locked, other_client}) do
    {:reply, :error, {:locked, other_client}}
    end
    def handle_call({:unlock, client}, _from, {:locked, client}) do
    {:reply, :ok, {:unlocked, nil}}
    end
    def handle_call({:unlock, client}, _from, {:locked, other_client}) do
    {:reply, :error, {:locked, other_client}}
    end
    end

    View Slide

  189. defmodule Demo.Lock do
    use Raft.StateMachine
    def init(:ok) do
    {:unlocked, nil}
    end
    def handle_write({:lock, client}, {:unlocked, nil}) do
    {:ok, {:locked, client}}
    end
    def handle_call({:lock, client}, _from, {:locked, other_client}) do
    {:reply, :error, {:locked, other_client}}
    end
    def handle_call({:unlock, client}, _from, {:locked, client}) do
    {:reply, :ok, {:unlocked, nil}}
    end
    def handle_call({:unlock, client}, _from, {:locked, other_client}) do
    {:reply, :error, {:locked, other_client}}
    end
    end

    View Slide

  190. defmodule Demo.Lock do
    use Raft.StateMachine
    def init(:ok) do
    {:unlocked, nil}
    end
    def handle_write({:lock, client}, {:unlocked, nil}) do
    {:ok, {:locked, client}}
    end
    def handle_write({:lock, client}, {:locked, other_client}) do
    {:error, {:locked, other_client}}
    end
    def handle_call({:unlock, client}, _from, {:locked, client}) do
    {:reply, :ok, {:unlocked, nil}}
    end
    def handle_call({:unlock, client}, _from, {:locked, other_client}) do
    {:reply, :error, {:locked, other_client}}
    end
    end

    View Slide

  191. defmodule Demo.Lock do
    use Raft.StateMachine
    def init(:ok) do
    {:unlocked, nil}
    end
    def handle_write({:lock, client}, {:unlocked, nil}) do
    {:ok, {:locked, client}}
    end
    def handle_write({:lock, client}, {:locked, other_client}) do
    {:error, {:locked, other_client}}
    end
    def handle_write({:unlock, client}, {:locked, client}) do
    {:ok, {:unlocked, nil}}
    end
    def handle_write({:unlock, client}, {:locked, other_client}) do
    {:error, {:locked, other_client}}
    end
    end

    View Slide

  192. View Slide

  193. Raft.start_peer(Demo.Lock, name: :s1)
    Raft.start_peer(Demo.Lock, name: :s2)
    Raft.start_peer(Demo.Lock, name: :s3)

    View Slide

  194. Raft.start_peer(Demo.Lock, name: :s1)
    Raft.start_peer(Demo.Lock, name: :s2)
    Raft.start_peer(Demo.Lock, name: :s3)
    Raft.set_configuration(:s1, [:s1, :s2, :s3])

    View Slide

  195. Raft.start_peer(Demo.Lock, name: :s1)
    Raft.start_peer(Demo.Lock, name: :s2)
    Raft.start_peer(Demo.Lock, name: :s3)
    Raft.set_configuration(:s1, [:s1, :s2, :s3])
    :ok = Raft.write(:s1, {:lock, :s1})

    View Slide

  196. Raft.start_peer(Demo.Lock, name: :s1)
    Raft.start_peer(Demo.Lock, name: :s2)
    Raft.start_peer(Demo.Lock, name: :s3)
    Raft.set_configuration(:s1, [:s1, :s2, :s3])
    :ok = Raft.write(:s1, {:lock, :s1})
    :error = Raft.write(:s2, {:lock, :s2})
    :error = Raft.write(:s2, {:unlock, :s2})

    View Slide

  197. Raft.start_peer(Demo.Lock, name: :s1)
    Raft.start_peer(Demo.Lock, name: :s2)
    Raft.start_peer(Demo.Lock, name: :s3)
    Raft.set_configuration(:s1, [:s1, :s2, :s3])
    :ok = Raft.write(:s1, {:lock, :s1})
    :error = Raft.write(:s2, {:lock, :s2})
    :error = Raft.write(:s2, {:unlock, :s2})
    :ok = Raft.write(:s1, {:unlock, :s1})

    View Slide

  198. Raft.start_peer(Demo.Lock, name: :s1)
    Raft.start_peer(Demo.Lock, name: :s2)
    Raft.start_peer(Demo.Lock, name: :s3)
    Raft.set_configuration(:s1, [:s1, :s2, :s3])
    :ok = Raft.write(:s1, {:lock, :s1})
    :error = Raft.write(:s2, {:lock, :s2})
    :error = Raft.write(:s2, {:unlock, :s2})
    :ok = Raft.write(:s1, {:unlock, :s1})
    :ok = Raft.write(:s2, {:lock, :s2})

    View Slide

  199. Demo

    View Slide

  200. How does
    this work?

    View Slide

  201. Node Node
    Node
    Consensus & leader election

    View Slide

  202. Consensus & leader election
    Leader
    Follower
    Follower
    Client

    View Slide

  203. Consensus & leader election
    Leader
    Follower
    Follower
    Client

    View Slide

  204. Consensus & leader election
    Leader
    Follower
    Follower
    Client

    View Slide

  205. Consensus & leader election
    Leader
    Follower
    Follower
    Client
    Replicated

    View Slide

  206. Consensus & leader election
    Leader
    Follower
    Follower
    Client

    View Slide

  207. Consensus & leader election
    Leader
    Follower
    Follower
    Client

    View Slide

  208. Consensus & leader election
    Leader
    Follower
    Follower
    Client

    View Slide

  209. Consensus & leader election
    Leader
    Follower
    Follower
    Client
    Committed

    View Slide

  210. Consensus & leader election
    Leader
    Follower
    Follower
    Client

    View Slide

  211. Consensus & leader election
    Leader
    Follower
    Follower
    Client

    View Slide

  212. Consensus & leader election
    Leader
    Follower
    Follower
    Client

    View Slide

  213. Consensus & leader election
    Leader
    Follower
    Follower
    Client
    Heartbeats

    View Slide

  214. Consensus & leader election
    Leader
    Follower
    Follower
    Client

    View Slide

  215. Consensus & leader election
    Leader
    Follower
    Follower
    Client

    View Slide

  216. Consensus & leader election
    Leader
    Follower
    Follower
    Client

    View Slide

  217. Consensus & leader election
    Leader
    Follower
    Follower
    Client

    View Slide

  218. Consensus & leader election
    Leader
    Follower
    Client
    Follower

    View Slide

  219. Consensus & leader election
    Leader
    Follower
    Client
    Follower
    Starts a new election

    View Slide

  220. Consensus & leader election
    Leader
    Follower
    Client
    Follower

    View Slide

  221. Consensus & leader election
    Leader
    Candidate
    Client
    Follower

    View Slide

  222. Consensus & leader election
    Leader
    Candidate
    Client
    Follower

    View Slide

  223. Consensus & leader election
    Leader
    Candidate
    Client
    Follower

    View Slide

  224. Consensus & leader election
    Leader
    Candidate
    Client
    Follower

    View Slide

  225. Consensus & leader election
    Leader
    Leader Follower
    Client

    View Slide

  226. Consensus & leader election
    Leader
    Leader Follower
    Client

    View Slide

  227. Consensus & leader election
    Leader
    Leader Follower
    Client

    View Slide

  228. Consensus & leader election
    Leader
    Leader Follower
    Client

    View Slide

  229. Consensus & leader election
    Leader
    Leader Follower
    Client

    View Slide

  230. Consensus & leader election
    Leader
    Leader Follower
    Client

    View Slide

  231. Consensus & leader election
    Follower
    Leader Follower
    Client

    View Slide

  232. Consensus & leader election
    Follower
    Leader Follower
    Client

    View Slide

  233. Consensus & leader election
    Follower
    Leader Follower
    Client

    View Slide

  234. Consensus & leader election
    Follower
    Leader Follower
    Client

    View Slide

  235. Consensus & leader election
    Follower
    Leader Follower
    Client

    View Slide

  236. Consensus & leader election
    Follower
    Leader Follower
    Client

    View Slide

  237. Consensus & leader election
    Follower
    Leader Follower
    Client

    View Slide

  238. Consensus & leader election
    Follower
    Leader Follower
    Client

    View Slide

  239. Consensus & leader election
    Follower
    Leader Follower
    Client

    View Slide

  240. Consensus & leader election
    Follower
    Leader Follower
    Client

    View Slide

  241. Consensus & leader election
    Follower
    Leader Follower
    Client

    View Slide

  242. Consensus & leader election
    Follower
    Leader Follower
    Client

    View Slide

  243. Consensus & leader election
    Follower
    Leader
    Follower
    Client

    View Slide

  244. Logs
    Linearized Writes
    Replicated across all nodes
    RocksDB

    View Slide

  245. Testing
    Property Tests
    Jepsen

    View Slide

  246. What can we
    do now?

    View Slide

  247. KV Store
    Service discovery
    Distributed lock manager
    Database transactions
    Configuration management

    View Slide

  248. Toniq

    View Slide

  249. https://github.com/toniqsystems/raft
    https://github.com/toniqsystems/raft_demo
    https://Toniq.sh
    https://speakerdeck/keathley
    Links

    View Slide

  250. Todo
    More Testing
    Dynamic Node Configurations
    LMDB Storage Adapter

    View Slide

  251. Now we can build
    applications and manage
    state safely without
    needing to leave elixir.

    View Slide

  252. Thanks
    Chris Keathley / @ChrisKeathley / [email protected]

    View Slide