Game of Loom 2: life and dead(lock) of a virtual thread

Transcript

1. Game of Loom 2:
   life and dead(lock) of
   a virtual thread
   by Mario Fusco
   @mariofusco
   

   View Slide

2. Where did we leave? A recap on virtual threads …
   ❖ Virtual threads make a better use of OS threads getting rid of the
   1:1 relationship with operating system threads
   ➢ Many virtual threads can be multiplexed on the same platform
   thread (carrier)
   

   View Slide

3. Where did we leave? A recap on virtual threads …
   ❖ Virtual threads make a better use of OS threads getting rid of the
   1:1 relationship with operating system threads
   ➢ Many virtual threads can be multiplexed on the same platform
   thread (carrier)
   ❖ When a virtual thread calls a blocking operation the JDK performs a
   nonblocking OS call and automatically suspends the virtual
   thread until the operation finishes
   ➢ Virtual threads perform blocking calls without consuming
   resources, thus allowing to write asynchronous code in a
   way that looks synchronous
   ➢ What virtual threads are really good for is … waiting
   

   View Slide

4. Where did we leave? A recap on virtual threads …
   ❖ Virtual threads make a better use of OS threads getting rid of the
   1:1 relationship with operating system threads
   ➢ Many virtual threads can be multiplexed on the same platform
   thread (carrier)
   ❖ When a virtual thread calls a blocking operation the JDK performs a
   nonblocking OS call and automatically suspends the virtual
   thread until the operation finishes
   ➢ Virtual threads perform blocking calls without consuming
   resources, thus allowing to write asynchronous code in a
   way that looks synchronous
   ➢ What virtual threads are really good for is … waiting
   ❖ Virtual threads are lightweight
   ➢ 200-300B metadata
   ➢ Pay-as-you-go stack (allocated on heap)
   ➢ Some ns (or below 1µs) for context switch (in user space)
   

   View Slide

5. View Slide

6. Should we really use virtual threads *EVERYWHERE*?
   

   View Slide

7. Should we really use virtual threads *EVERYWHERE*?
   ❖ Not ideal for CPU-bound task
   ➢ Lack of fairness due to the fact that virtual
   threads are not pre-emptive
   ➢ Overall performance concerns caused by
   overhead of virtual threads scheduler
   

   View Slide

8. Should we really use virtual threads *EVERYWHERE*?
   ❖ Not ideal for CPU-bound task
   ➢ Lack of fairness due to the fact that virtual
   threads are not pre-emptive
   ➢ Overall performance concerns caused by
   overhead of virtual threads scheduler
   ❖ Not virtual-threads friendly frameworks & tools
   ➢ Use of synchronize can lead to pinning
   ➢ Use of ThreadLocal
   

   View Slide

9. Should we really use virtual threads *EVERYWHERE*?
   ❖ Not ideal for CPU-bound task
   ➢ Lack of fairness due to the fact that virtual
   threads are not pre-emptive
   ➢ Overall performance concerns caused by
   overhead of virtual threads scheduler
   ❖ Not virtual-threads friendly frameworks & tools
   ➢ Use of synchronize can lead to pinning
   ➢ Use of ThreadLocal
   ❖ Poorly customizable
   ➢ Carrier native thread pool is not pluggable
   (but for a very good reason)
   

   View Slide

10. Lack of fairness in CPU bound tasks
    Virtual Threads are not pre-emptive, they cannot be descheduled while running heavy
    calculations without ever calling a JDK’s blocking methods, so they are not a good fit
    for CPU-bound tasks when fairness is key
    Interrupts the
    processing of a
    task and
    transfers the
    CPU to another
    process
    Non-preemptive
    operation usually
    proceeds towards
    completion
    uninterrupted
    

    View Slide

11. Lack of fairness in CPU bound tasks
    Virtual Threads are not pre-emptive, they cannot be descheduled while running heavy
    calculations without ever calling a JDK’s blocking methods, so they are not a good fit
    for CPU-bound tasks when fairness is key
    

    View Slide

12. The cost of a continuation
    In Project Loom, the word continuation means
    delimited continuation, also sometimes called a
    coroutine. It can be thought of as sequential
    code that may suspend or yield execution at
    some point by itself and can be resumed by a
    caller.
    When a virtual thread hits a blocking call its
    continuation is suspended and the JVM
    unmounts the thread from its carrier …
    

    View Slide

13. The cost of a continuation
    In Project Loom, the word continuation means
    delimited continuation, also sometimes called a
    coroutine. It can be thought of as sequential
    code that may suspend or yield execution at
    some point by itself and can be resumed by a
    caller.
    When a virtual thread hits a blocking call its
    continuation is suspended and the JVM
    unmounts the thread from its carrier …
    … but this also has a cost!
    

    View Slide

14. Native Threads Execution
    

    View Slide

15. Virtual Threads Execution - The cost of a continuation
    

    View Slide

16. Virtual Threads Execution - The cost of a continuation
    

    View Slide

17. ThreadLocals as thread scoped variables
    enum RightsLevel { ADMIN, GUEST }
    record Principal(RightsLevel rights) {
    public boolean canOpen() {
    return rights == RightsLevel.ADMIN;
    }
    }
    record Request(boolean authorized) { }
    class Server {
    final static ThreadLocal PRINCIPAL =
    new ThreadLocal<>();
    String serve(Request request) {
    var level = request.authorized() ?
    RightsLevel.ADMIN : RightsLevel.GUEST;
    PRINCIPAL.set( new Principal(level) );
    return Application.handle( request );
    }
    }
    

    View Slide

18. ThreadLocals as thread scoped variables
    static class DBConnection {
    static DBConnection open() {
    var principal = Server.PRINCIPAL.get();
    if (!principal.canOpen())
    throw new IllegalArgumentException();
    return new DBConnection();
    }
    String doQuery() {
    return "Result";
    }
    }
    static class Application {
    public static String handle(Request request) {
    return DBConnection.open().doQuery();
    }
    }
    enum RightsLevel { ADMIN, GUEST }
    record Principal(RightsLevel rights) {
    public boolean canOpen() {
    return rights == RightsLevel.ADMIN;
    }
    }
    record Request(boolean authorized) { }
    class Server {
    final static ThreadLocal PRINCIPAL =
    new ThreadLocal<>();
    String serve(Request request) {
    var level = request.authorized() ?
    RightsLevel.ADMIN : RightsLevel.GUEST;
    PRINCIPAL.set( new Principal(level) );
    return Application.handle( request );
    }
    }
    

    View Slide

19. ThreadLocals as thread scoped variables
    A ThreadLocal is a construct that
    allows us to store data accessible
    only by a specific thread.
    static class DBConnection {
    static DBConnection open() {
    var principal = Server.PRINCIPAL.get();
    if (!principal.canOpen())
    throw new IllegalArgumentException();
    return new DBConnection();
    }
    String doQuery() {
    return "Result";
    }
    }
    static class Application {
    public static String handle(Request request) {
    return DBConnection.open().doQuery();
    }
    }
    enum RightsLevel { ADMIN, GUEST }
    record Principal(RightsLevel rights) {
    public boolean canOpen() {
    return rights == RightsLevel.ADMIN;
    }
    }
    record Request(boolean authorized) { }
    class Server {
    final static ThreadLocal PRINCIPAL =
    new ThreadLocal<>();
    String serve(Request request) {
    var level = request.authorized() ?
    RightsLevel.ADMIN : RightsLevel.GUEST;
    PRINCIPAL.set( new Principal(level) );
    return Application.handle( request );
    }
    }
    

    View Slide

20. ThreadLocals as (virtual) thread scoped variables
    public static void main(String[] args) {
    var server = new Server();
    var authorized = Thread.ofVirtual().name("Authorized").unstarted(() -> callServer(server, true));
    var notAuthorized = Thread.ofVirtual().name("NOT Authorized").unstarted(() -> callServer(server, false));
    authorized.start();
    notAuthorized.start();
    try { Thread.sleep(1000L); } catch (InterruptedException e) { throw new RuntimeException(e); }
    }
    private static void callServer(Server server, boolean auth) {
    var result = server.serve(new Request(auth));
    System.out.println( "thread " + Thread.currentThread().getName() + " got result " + result );
    }
    

    View Slide

21. ThreadLocals as (virtual) thread scoped variables
    public static void main(String[] args) {
    var server = new Server();
    var authorized = Thread.ofVirtual().name("Authorized").unstarted(() -> callServer(server, true));
    var notAuthorized = Thread.ofVirtual().name("NOT Authorized").unstarted(() -> callServer(server, false));
    authorized.start();
    notAuthorized.start();
    try { Thread.sleep(1000L); } catch (InterruptedException e) { throw new RuntimeException(e); }
    }
    private static void callServer(Server server, boolean auth) {
    var result = server.serve(new Request(auth));
    System.out.println( "thread " + Thread.currentThread().getName() + " got result " + result );
    }
    Exception in thread "NOT Authorized" java.lang.IllegalArgumentException
    at org.mfusco.loom.experiments.threadlocal.ThreadLocalMain$DBConnection.open(ThreadLocalMain.java:48)
    at org.mfusco.loom.experiments.threadlocal.ThreadLocalMain$Application.handle(ThreadLocalMain.java:41)
    thread Authorized got result Result
    

    View Slide

22. ThreadLocals as (virtual) thread scoped variables
    So? Everything is fine, right?
    What is the problem to use
    ThreadLocal with virtual threads?
    

    View Slide

23. ThreadLocals as (virtual) thread scoped variables
    So? Everything is fine, right?
    What is the problem to use
    ThreadLocal with virtual threads?
    ❖ That we can have a huge number of virtual threads, and each
    virtual thread will have its own ThreadLocal
    ➢ The memory footprint of the application may quickly become
    very high
    ➢ ThreadLocal will be useless in a one-thread-per-request
    scenario since data won’t be shared between different requests
    

    View Slide

24. ScopedValues to the rescue
    class Server {
    final static ThreadLocal PRINCIPAL = new ThreadLocal<>();
    String serve(Request request) {
    var level = request.authorized() ? RightsLevel.ADMIN : RightsLevel.GUEST;
    PRINCIPAL.set( new Principal(level) );
    return Application.handle( request );
    }
    }
    

    View Slide

25. ScopedValues to the rescue (still preview in JDK 21)
    class Server {
    final static ThreadLocal PRINCIPAL = new ThreadLocal<>();
    String serve(Request request) {
    var level = request.authorized() ? RightsLevel.ADMIN : RightsLevel.GUEST;
    PRINCIPAL.set( new Principal(level) );
    return Application.handle( request );
    }
    }
    class Server {
    final static ScopedValue PRINCIPAL =
    ScopedValue.newInstance();
    String serve(Request request) {
    var level = request.authorized() ?
    RightsLevel.ADMIN : RightsLevel.GUEST;
    try {
    return ScopedValue.where(PRINCIPAL, new Principal(level))
    .call(() -> Application.handle(request));
    } catch (Exception e) { throw new RuntimeException(e); }
    }
    }
    

    View Slide

26. ThreadLocals as Cache / Objects Pool ???
    Do ScopedValues cover all the possible
    typical usage patterns of ThreadLocals?
    

    View Slide

27. ThreadLocals as Cache / Objects Pool ???
    Do ScopedValues cover all the possible
    typical usage patterns of ThreadLocals?
    ❖ It’s very common to use a ThreadLocal
    as a thread-safe object pool
    ➢ Easy to implement and use
    ➢ Does not require an explicit lifecycle: it is not necessary to
    put the pooled resource back into the pool when finished
    

    View Slide

28. ThreadLocals as Cache / Objects Pool ???
    Do ScopedValues cover all the possible
    typical usage patterns of ThreadLocals?
    ❖ It’s very common to use a ThreadLocal
    as a thread-safe object pool
    ➢ Easy to implement and use
    ➢ Does not require an explicit lifecycle: it is not necessary to
    put the pooled resource back into the pool when finished
    ➢ ScopedValues don’t help in this scenario
    ➢ This pattern plays particularly bad with virtual threads
    

    View Slide

29. ThreadLocals as Cache / Objects Pool ???
    Do ScopedValues cover all the possible
    typical usage patterns of ThreadLocals?
    ❖ It’s very common to use a ThreadLocal
    as a thread-safe object pool
    ➢ Easy to implement and use
    ➢ Doesn’t require an explicit lifecycle: it is not necessary to
    put the pooled resource back into the pool when finished
    ➢ ScopedValues don’t help in this scenario
    ➢ This pattern plays particularly bad with virtual threads
    

    View Slide

30. ThreadLocals as Cache / Objects Pool into the wild …
    

    View Slide

31. … and the fix 🎉
    

    View Slide

32. Plugging a different thread pool as carriers
    Can we outperform the default Fork/Join
    pool based carrier thread pool?
    

    View Slide

33. Plugging a different thread pool as carriers
    Can we outperform the default Fork/Join
    pool based carrier thread pool?
    Let’s put at work Game of Life again!
    

    View Slide

34. Plugging a different thread pool as carriers
    Can we outperform the default Fork/Join
    pool based carrier thread pool?
    Let’s put at work Game of Life again!
    

    View Slide

35. Plugging a different thread pool as carriers
    Can we outperform the default Fork/Join
    pool based carrier thread pool?
    Benchmark (executionStrategy) Mode Cnt Score Error Units
    GameOfLifeBenchmark.benchmark Native thrpt 40 40.317 ± 0.353 ops/s
    GameOfLifeBenchmark.benchmark ForkJoinVirtual thrpt 40 244.639 ± 1.419 ops/s
    GameOfLifeBenchmark.benchmark FixedCarrierPoolVirtual thrpt 40 129.525 ± 5.346 ops/s
    GameOfLifeBenchmark.benchmark PinnedCarrierVirtual thrpt 40 59.071 ± 1.532 ops/s
    Let’s put at work Game of Life again!
    

    View Slide

36. So why we may want to plug a different carrier?
    The Quarkus use case - Reactive ≊ Virtual Threads
    

    View Slide

37. So why we may want to plug a different carrier?
    The Quarkus use case - Reactive ≊ Virtual Threads
    ≊
    

    View Slide

38. So why we may want to plug a different carrier?
    The Quarkus use case - The Event Loop as carrier
    @RunOnVirtualThread
    

    View Slide

39. So why we may want to plug a different carrier?
    The Quarkus use case - The Event Loop as carrier
    →
    @RunOnVirtualThread
    

    View Slide

40. Why not using the event-loop as carrier thread???
    ❖ No (more) API to do that in Loom - API removed in fall 2021
    

    View Slide

41. Why not using the event-loop as carrier thread???
    ❖ No (more) API to do that in Loom - API removed in fall 2021
    ❖ It lead to…. deadlocks!
    

    View Slide

42. What’s going on ???
    Main EventLoop/Carrier
    VirtualThread
    newThread
    execute
    lockAcquired.join() lockAcquired.join()
    

    View Slide

43. What’s going on ???
    Main EventLoop/Carrier
    VirtualThread
    newThread
    execute
    lockAcquired.join() lockAcquired.join()
    lock()
    lockAcquired.complete()
    

    View Slide

44. What’s going on ???
    Main EventLoop/Carrier
    VirtualThread
    newThread
    execute
    lockAcquired.join() lockAcquired.join()
    lock()
    lockAcquired.complete()
    lock() lock()
    busy wait for both carrier
    and vThread to ask the lock
    

    View Slide

45. What’s going on ???
    Main EventLoop/Carrier
    VirtualThread
    newThread
    execute
    lockAcquired.join() lockAcquired.join()
    lock()
    lockAcquired.complete()
    lock() lock()
    busy wait for both carrier
    and vThread to ask the lock
    unlock()
    wait for vThread to
    acquire the lock
    cannot make any
    progress and acquire
    the lock since its own
    carrier is blocked
    

    View Slide

46. Conclusions
    ❖ Virtual threads are not faster threads ⇒ they don’t magically execute
    more instructions per second than native ones do.
    ❖ Virtual threads are not drop in replacement for native threads ⇒ they
    are a different (new) tool and you have to know their features and
    characteristics in order to use them appropriately.
    ❖ Virtual threads are generally not a good fit for CPU bound tasks ⇒ lack
    of fairness and the performance cost of coroutines can be problematic.
    ❖ What virtual threads are really good for is waiting ⇒ their goal is
    maximizing the utilization of external resources and then improving
    throughput, without affecting readability.
    

    View Slide

47. Quarkus Virtual Threads
    - REST / HTTP
    - Hibernate, Bean Validation,
    Transactions
    - Kafka, AMQP, JMS
    - gRPC
    - Scheduled tasks
    - Event Bus
    - …
    Learn More
    

    View Slide