[RailsWorld 2023] Untangling cables & demystifying twisted transistors

Transcript

1. Vladimir Dementyev
   Evil Martians
   Untangling Cables &
   Demystifying Twisted Transistors
   2023
   1. It's Rails
   2. Freak on a Cable
   3. Y'all Want a Stream
   4. Make Me ping
   5. Wire Tongue
   6. Threaded Transistor
   7. Throw me & Forget
   8. Turbo Streams Everywhere
   9. (Not) Alone I Reconnect
   Bonus: See You on the Other Cable
   Lyrics by Vladimir "Palkan" Dementyev
   Produced by Evil Martians, inc.
   

   View Slide

2. 1. It's Rails
   

   View Slide

3. Rails is a
   software
   development
   philosophy
   3 palkan_tula
   palkan
   

   View Slide

4. Conceptual
   Compression
   4 palkan_tula
   palkan
   

   View Slide

5. Unboxing time!
   5 palkan_tula
   palkan
   

   View Slide

6. Unboxing time!
   5 palkan_tula
   palkan
   

   View Slide

7. 2. Freak on a Cable
   

   View Slide

8. View Slide

9. github.com/palkan
   

   View Slide

10. github.com/palkan
    evilmartians.com
    

    View Slide

11. github.com/palkan
    evilmartians.com
    anycable.io
    

    View Slide

12. The Book on
    abstraction
    layers from and
    for Rails
    8 palkan_tula
    palkan
    

    View Slide

13. 3. Y'all Want a Stream
    

    View Slide

14. palkan_tula
    palkan
    Channels Layer
    10
    class ChatChannel < ApplicationCable::Channel
    def subscribed
    @room = Room.find_by(id: params[:id])
    return reject unless @room
    stream_for @room
    end
    def speak(data)
    broadcast_to(@room,
    event: "newMessage", user_id: user.id, body: data.fetch("body")
    )
    end
    end
    

    View Slide

15. 11 palkan_tula
    palkan
    class ChatChannel < ApplicationCable::Channel
    def subscribed
    @room = Room.find_by(id: params[:id])
    return reject unless @room
    stream_for @room
    end
    def speak(data)
    broadcast_to(@room,
    event: "newMessage", user_id: user.id,
    body: data.fetch("body")
    )
    end
    end
    import { createConsumer } from "@rails/actioncable"
    let cable = createConsumer()
    let sub = cable.subscriptions.create(
    {channel: "ChatChannel", id: 2023},
    {
    received({user_id, body}) {
    console.log(`Message from ${user_id}:${body}`);
    }
    }
    )
    sub.perform("speak", {body: "Hoi!"})
    

    View Slide

16. 12
    

    View Slide

17. palkan_tula
    palkan
    13
    Protocol
    Web
    Server
    Action Cable
    Server
    I/O
    loop
    Rails
    Executor
    Connection
    Channel
    Pub/Sub
    Subscriber
    Map
    Client
    Socket
    Consumer
    Web
    Socket
    Sub
    Pubsub
    Adapter
    Monitor
    Channel Channel
    Worker Pool
    Internal Pool
    Sub Sub
    Map of
    Action Cable
    HTTP
    rack.hijack
    #connect
    #subscribed
    { command: "subscribe" }
    OPEN
    { command: "message" }
    #receive
    CLOSE
    #disconnect
    #unsubscribed
    #stream_from
    #subscribe
    #unsubscribe
    broadcast
    #invoke_callback
    #send
    

    View Slide

18. 4. Make Me Ping
    

    View Slide

19. palkan_tula
    palkan
    @rails/actioncable
    15
    import { createConsumer } from "@rails/actioncable"
    let cable = createConsumer()
    cable.subscriptions.create(
    {channel: "ChatChannel", id: 2023},
    {
    received({user_id, body}) {
    console.log(`Message from ${user_id}:${body}`);
    }
    }
    )
    

    View Slide

20. palkan_tula
    palkan
    @rails/actioncable
    16
    import { createConsumer } from "@rails/actioncable"
    let cable = createConsumer()
    cable.subscriptions.create(
    {channel: "ChatChannel", id: 2023},
    {
    received({user_id, body}) {
    console.log(`Message from ${user_id}:${body}`);
    }
    }
    )
    ~ WebSocket connection
    Abstraction to subscribe for
    and receive updates
    

    View Slide

21. palkan_tula
    palkan
    One consumer per session (e.g., a
    browser tab)
    17
    @rails/actioncable
    

    View Slide

22. palkan_tula
    palkan
    github.com/le0pard/cable-shared-worker
    18
    

    View Slide

23. palkan_tula
    palkan
    One consumer per session (e.g., a
    browser tab)
    Many subscriptions per consumer
    (multiplexing)
    19
    @rails/actioncable
    

    View Slide

24. palkan_tula
    palkan
    One consumer per session (e.g., a
    browser tab)
    Many subscriptions per consumer
    (multiplexing) — how many?
    20
    @rails/actioncable
    

    View Slide

25. We may* have
    as many
    subscriptions
    as canals in
    Amsterdam
    21 palkan_tula
    palkan
    * Doesn't mean we should though
    Consumer
    Monitor
    Sub Sub Sub
    Web
    Socket
    Sub Sub
    

    View Slide

26. palkan_tula
    palkan
    @rails/actioncable
    22
    import { createConsumer } from "@rails/actioncable"
    let cable = createConsumer()
    let a = cable.subscriptions.create("UserChannel") // SUBSCRIBE
    let b = cable.subscriptions.create("UserChannel") //
    a.unsubscribe() //
    b.unsubscribe() // UNSUBSCRIBE
    

    View Slide

27. palkan_tula
    palkan
    One consumer per session (e.g., a
    browser tab)
    Many subscriptions per consumer
    Monitor to keep consumer connected
    23
    @rails/actioncable
    

    View Slide

28. palkan_tula
    palkan
    Keep alive
    24
    Consumer
    Monitor
    Web
    Socket
    stale timer
    

    View Slide

29. palkan_tula
    palkan
    Keep alive
    24
    Consumer
    Monitor
    Web
    Socket
    ping
    stale timer
    

    View Slide

30. palkan_tula
    palkan
    Keep alive
    24
    Consumer
    Monitor
    Web
    Socket
    stale timer reconnect
    

    View Slide

31. palkan_tula
    palkan
    Action Cable Pings
    Help to identify broken TCP connections
    (from both sides, but not always)
    25
    

    View Slide

32. palkan_tula
    palkan
    Action Cable Pings
    Help to identify broken TCP connections
    (from both sides, but not always)
    Hardly configurable (and may be too
    frequent to cause battery drain for mobile
    devices)
    25
    

    View Slide

33. 26 palkan_tula
    palkan
    # config/application.rb
    # Re-define internal constant
    module ActionCable::Server::Connections
    BEAT_INTERVAL = 10
    end
    import { ConnetionMonitor } from "@rails/actioncable"
    // Adjust stale timeout (2xBEAT_INTERVAL)
    ConnectionMonitor.staleThreshold = 20
    

    View Slide

34. palkan_tula
    palkan
    More clients
    github.com/anycable/anycable-client
    27
    

    View Slide

35. palkan_tula
    palkan
    More clients
    github.com/anycable/anycable-client
    
    27
    

    View Slide

36. 5. Wire Tongue
    

    View Slide

37. Protocol is a
    language in
    which the
    server and
    client speak to
    each other
    29 palkan_tula
    palkan
    Protocol
    Web Server
    Consumer
    Web
    Socket
    HTTP
    

    View Slide

38. palkan_tula
    palkan
    docs.anycable.io
    30
    

    View Slide

39. 31 palkan_tula
    palkan
    {
    "command":"subscribe",
    "identifier":"{\"channel\" \"UserChannel\"}"
    }
    {
    "command":"message",
    "identifier":"{\"channel\":\"UserChannel\"}",
    "data":"{\"action\":\"speak\",\"text\":\"hoi\"}"
    }
    {
    "command":"unsubscribe",
    "identifier":"{\"channel\":\"UserChannel\"}"
    }
    {"type":"welcome"}
    {
    "type":"confirm_subscription",
    "identifier":"{\"channel\":\"UserChannel\"}"
    }
    {
    "type":"reject_subscription",
    "identifier":"{\"channel\":\"UserChannel\"}"
    }
    {
    "identifier":"{\"channel\":\"UserChannel\"}",
    "message":"{\"text\":\"hoi\"}"
    }
    {"type":"ping"}
    {
    "type":"disconnect",
    "reason":"unauthorized",
    "reconnect":false
    }
    No type!
    

    View Slide

40. palkan_tula
    palkan
    Protocol is incomplete
    No IDs (session, message)
    No perform ACKs
    No error handling
    32
    

    View Slide

41. palkan_tula
    palkan
    33
    let cable = createConsumer()
    for(let i=0; i<10; i++) {
    let sub = cable.subscriptions.create("UserChannel")
    sub.unsubscribe()
    }
    // Will I be subscribed or not?
    cable.subscriptions.create("UserChannel")
    

    View Slide

42. palkan_tula
    palkan
    github.com/palkan/wsdirector
    - loop:
    multiplier: 10
    actions:
    - send:
    data: &sub
    command: "subscribe"
    identifier: "{\"channel\":\"BenchmarkChannel\"}"
    - send:
    data:
    command: "unsubscribe"
    identifier: "{\"channel\":\"BenchmarkChannel\"}"
    - send:
    <<: *sub
    - send:
    data:
    command: "message"
    identifier: "{\"channel\":\"BenchmarkChannel\"}"
    data: "{\"action\":\"echo\",\"test\":42}"
    - receive:
    data:
    identifier: "{\"channel\":\"BenchmarkChannel\"}"
    message: {action: "echo", test: 42}
    34
    

    View Slide

43. 35
    

    View Slide

44. 36
    

    View Slide

45. 36
    RuntimeError - Unable to find subscription with identifier: {"channel":"BenchmarkChannel"}
    RuntimeError - Already subscribed to {"channel":"BenchmarkChannel"}
    

    View Slide

46. palkan_tula
    palkan
    Solution?
    37
    

    View Slide

47. 38
    (First rows only note) Some random Rails application
    

    View Slide

48. palkan_tula
    palkan
    39
    let cable = createConsumer()
    for(let i=0; i<10; i++) {
    let sub = cable.subscriptions.create("UserChannel")
    sub.unsubscribe()
    }
    // Will I be subscribed or not? It depends. Why so?
    cable.subscriptions.create("UserChannel")
    

    View Slide

49. 6. Threaded Transistor
    

    View Slide

50. palkan_tula
    palkan
    41
    Web
    Server
    Web
    Socket
    HTTP
    Action Cable
    Executor
    

    View Slide

51. palkan_tula
    palkan
    41
    Web
    Server
    Action Cable
    Server
    Web
    Socket
    HTTP
    rack.hijack
    Action Cable
    Executor
    

    View Slide

52. palkan_tula
    palkan
    41
    Web
    Server
    Action Cable
    Server
    I/O
    loop
    Client
    Socket
    Web
    Socket
    HTTP
    rack.hijack
    Action Cable
    Executor
    

    View Slide

53. palkan_tula
    palkan
    41
    Web
    Server
    Action Cable
    Server
    I/O
    loop
    Client
    Socket
    Web
    Socket
    Worker Pool
    HTTP
    rack.hijack
    OPEN
    Action Cable
    Executor
    

    View Slide

54. palkan_tula
    palkan
    41
    Web
    Server
    Action Cable
    Server
    I/O
    loop
    Rails
    Executor
    Client
    Socket
    Web
    Socket
    Worker Pool
    HTTP
    rack.hijack
    OPEN
    Action Cable
    Executor
    

    View Slide

55. palkan_tula
    palkan
    41
    Web
    Server
    Action Cable
    Server
    I/O
    loop
    Rails
    Executor
    Connection
    Client
    Socket
    Web
    Socket
    Worker Pool
    HTTP
    rack.hijack
    #connect
    OPEN
    Action Cable
    Executor
    

    View Slide

56. palkan_tula
    palkan
    41
    Web
    Server
    Action Cable
    Server
    I/O
    loop
    Rails
    Executor
    Connection
    Channel
    Client
    Socket
    Web
    Socket
    Worker Pool
    HTTP
    rack.hijack
    #connect
    {command: "subscribe"}
    OPEN
    #subscribed
    Action Cable
    Executor
    

    View Slide

57. palkan_tula
    palkan
    41
    Web
    Server
    Action Cable
    Server
    I/O
    loop
    Rails
    Executor
    Connection
    Channel
    Client
    Socket
    Web
    Socket
    Worker Pool
    HTTP
    rack.hijack
    #connect
    {command: "subscribe"}
    OPEN
    {command: "message"}
    #receive
    #subscribed
    Action Cable
    Executor
    

    View Slide

58. palkan_tula
    palkan
    41
    Web
    Server
    Action Cable
    Server
    I/O
    loop
    Rails
    Executor
    Connection
    Channel
    Client
    Socket
    Web
    Socket
    Worker Pool
    HTTP
    rack.hijack
    #connect
    #unsubscribed
    {command: "subscribe"}
    OPEN
    {command: "message"}
    #receive
    CLOSE
    #disconnect
    #subscribed
    Action Cable
    Executor
    

    View Slide

59. palkan_tula
    palkan
    Action Cable Server
    Thread pool executor (4 threads by default)
    Must be taken into account for shared
    resources (e.g., database pool)
    Uses Rails executor to cleanup after work
    42
    

    View Slide

60. 43 palkan_tula
    palkan
    class Connection
    identified_by :user
    def connect
    self.user = find_verified_user
    Current.account = user.account
    end
    end
    

    View Slide

61. 44 palkan_tula
    palkan
    class Connection
    identified_by :user
    def connect
    self.user = find_verified_user
    # BAD — will be reset at the end
    # of the #handle_open call
    Current.account = user.account
    end
    end
    

    View Slide

62. class Connection
    identified_by :user
    before_command do
    # GOOD — set for every command
    Current.account = user.account
    end
    def connect
    self.user = find_verified_user
    end
    end
    44 palkan_tula
    palkan
    class Connection
    identified_by :user
    def connect
    self.user = find_verified_user
    # BAD — will be reset at the end
    # of the #handle_open call
    Current.account = user.account
    end
    end
    

    View Slide

63. palkan_tula
    palkan
    Units of Work
    Lifecycle callbacks (connect, disconnect)
    45
    

    View Slide

64. palkan_tula
    palkan
    Units of Work
    Lifecycle callbacks (connect, disconnect)
    Incoming commands
    45
    

    View Slide

65. palkan_tula
    palkan
    Units of Work
    Lifecycle callbacks (connect, disconnect)
    Incoming commands
    Outgoing messages
    45
    

    View Slide

66. 7. Throw me & Forget
    

    View Slide

67. palkan_tula
    palkan
    47
    Rails
    Executor
    Connection
    Pub/Sub
    Subscriber
    Map Client
    Socket
    Pubsub
    Adapter
    Channel
    Worker Pool
    Internal Pool
    Pub/Sub
    

    View Slide

68. palkan_tula
    palkan
    47
    Rails
    Executor
    Connection
    Pub/Sub
    Subscriber
    Map Client
    Socket
    Pubsub
    Adapter
    Channel
    Worker Pool
    Internal Pool
    broadcast message
    Pub/Sub
    

    View Slide

69. palkan_tula
    palkan
    47
    Rails
    Executor
    Connection
    Pub/Sub
    Subscriber
    Map Client
    Socket
    Pubsub
    Adapter
    Channel
    Worker Pool
    Internal Pool
    broadcast message
    #broadcast
    Pub/Sub
    

    View Slide

70. palkan_tula
    palkan
    47
    Rails
    Executor
    Connection
    Pub/Sub
    Subscriber
    Map Client
    Socket
    Pubsub
    Adapter
    Channel
    Worker Pool
    Internal Pool
    broadcast message
    #invoke_callback
    #broadcast
    Pub/Sub
    

    View Slide

71. palkan_tula
    palkan
    47
    Rails
    Executor
    Connection
    Pub/Sub
    Subscriber
    Map Client
    Socket
    Pubsub
    Adapter
    Channel
    Worker Pool
    Internal Pool
    broadcast message
    #invoke_callback
    #broadcast
    #stream_from callback
    is executed
    Pub/Sub
    

    View Slide

72. palkan_tula
    palkan
    47
    Rails
    Executor
    Connection
    Pub/Sub
    Subscriber
    Map Client
    Socket
    Pubsub
    Adapter
    Channel
    Worker Pool
    Internal Pool
    broadcast message
    #invoke_callback
    #send
    #broadcast
    #transmit
    #stream_from callback
    is executed
    Pub/Sub
    

    View Slide

73. palkan_tula
    palkan
    47
    Rails
    Executor
    Connection
    Pub/Sub
    Subscriber
    Map Client
    Socket
    Pubsub
    Adapter
    Channel
    Worker Pool
    Internal Pool
    Pub/Sub
    at-most once
    

    View Slide

74. 48
    The pitfalls of realtime-ification, RailsConf, 2022
    

    View Slide

75. palkan_tula
    palkan
    Yet Another Pitfall
    49
    100.times {
    ActionCable.server.broadcast "test", {text: "Count: #{_1}"}
    }
    # Will a client receive 1,2,3,4,...,99 in order?
    

    View Slide

76. 50
    

    View Slide

77. 51
    

    View Slide

78. palkan_tula
    palkan
    And One More
    52
    def subscribed
    stream_from "test"
    # Q: Will this client receive the message?
    ActionCable.server.broadcast "test", {text: "Welkom!"}
    end
    

    View Slide

79. Are you tired of
    threads?
    53 palkan_tula
    palkan
    I am
    

    View Slide

80. 8. Turbo Streams Everywhere
    

    View Slide

81. palkan_tula
    palkan
    Turbo Streams z
    Automatic subscriptions via HTML
    ()
    Built-in Turbo::StreamsChannel
    Zero application code to connect cables
    55
    

    View Slide

82. palkan_tula
    palkan
    Turbo Streams
    56
    www.my-hot.app
    

    View Slide

83. palkan_tula
    palkan
    Turbo Streams
    54
    www.my-hot.app
    
    SUBSCRIBE
    Connection
    Channel
    

    View Slide

84. palkan_tula
    palkan
    DOM
    update
    Turbo Streams
    54
    www.my-hot.app
    
    HTML
    Action Cable broadcast
    from anywhere
    Connection
    Channel
    

    View Slide

85. palkan_tula
    palkan
    UNSUBSCRIBE
    Turbo Streams
    54
    www.my-hot.app
    Connection
    

    View Slide

86. palkan_tula
    palkan
    Turbo Streams
    Prone to subscribe/unsubscribe race
    conditions
    Managed in a decentralized way (via
    HTML partials)
    57
    

    View Slide

87. 58
    

    View Slide

88. 59
    anycable.substack.com
    

    View Slide

89. 9. (Not) Alone I ReConnect
    

    View Slide

90. palkan_tula
    palkan
    Server Restart
    N clients re-connect
    N clients re-subscribe each to M channels each
    Threaded executor's work queue is:
    (N+1)*M
    61
    

    View Slide

91. palkan_tula
    palkan
    Server Restart
    Threaded executor's work queue is:
    (N+1)*M
    Server can not keep up with confirmations,
    clients re-issue subscribe commands—!
    62
    

    View Slide

92. palkan_tula
    palkan
    Server Restart
    Threaded executor's work queue is:
    (N+1)*M
    Server can not keep up with confirmations,
    clients re-issue subscribe commands—!
    62
    Thundering
    Herd
    

    View Slide

93. palkan_tula
    palkan
    Server Restart
    Threaded executor's work queue is:
    (N+1)*M
    Server can not keep up with confirmations,
    clients re-issue subscribe commands—!
    62
    Connection
    Avalanche
    

    View Slide

94. Connection
    avalanches lead
    to significant
    load spikes
    63 palkan_tula
    palkan
    

    View Slide

95. How to protect
    Rails from
    connection
    avalanches?
    64 palkan_tula
    palkan
    

    View Slide

96. palkan_tula
    palkan
    Avalanche Protection
    Reconnect with backoff & jitter
    65
    

    View Slide

97. 66
    

    View Slide

98. palkan_tula
    palkan
    Avalanche Protection
    Reconnect with backoff & jitter
    Merge subscriptions
    67
    

    View Slide

99. palkan_tula
    palkan
    Avalanche Protection
    Reconnect with backoff & jitter
    Merge subscriptions
    Avoid slow calls in #subscribed
    67
    

    View Slide

100. palkan_tula
     palkan
     Avalanche Protection
     Reconnect with backoff & jitter
     Merge subscriptions
     Avoid slow calls in #subscribed
     Serialize subscribe commands
     67
     

     View Slide

101. palkan_tula
     palkan
     68
     import { createCable } from "@anycable/web"
     let cable = createCable({
     concurrentSubscribes: false
     })
     

     View Slide

102. palkan_tula
     palkan
     Isolate WebSocket and regular HTTP
     clusters
     69
     Avalanche Protection
     

     View Slide

103. palkan_tula
     palkan
     Isolate WebSocket and regular HTTP
     clusters
     AnyCable: disconnect-less deploys
     69
     Avalanche Protection
     

     View Slide

104. palkan_tula
     palkan
     Isolate WebSocket and regular HTTP
     clusters
     AnyCable: disconnect-less deploys
     AnyCable: resumable sessions
     69
     Avalanche Protection
     

     View Slide

105. See you on the other Cable
     

     View Slide

106. palkan_tula
     palkan
     71
     Map of
     AnyCable
     Protocol+
     AnyCable-Go
     I/O
     poll
     Rails
     Executor
     Connection
     Channel
     Cache
     Session
     Cable
     Web
     Socket
     Channel
     Monitor
     Goroutine Pools
     Sub
     HTTP
     #connect
     #subscribed
     #disconnect
     #unsubscribed
     broadcast
     Encoder
     Transport
     Hub
     Sub
     Channel
     Channel
     Logger
     gRPC Pool
     RPC
     Controller
     Read Chan
     Write Chan
     Broadcaster
     Broker
     Hub
     Pub/Sub
     Shard
     Shard
     subscribe
     OPEN
     DATA
     CLOSE
     

     View Slide

107. Thank You
     Slides: evilmartians.com/events
     Twitter: @palkan_tula, @evilmartians,
     @any_cable
     

     View Slide