I am an Outreachy intern for Servo and I have been working on implementing AudioWorklet. I am wrapping up my internship with Servo and this blog post will detail the results of my work, the state of Worklets in Servo, and what's left of it. Towards the end, I will also talk about my career prospects and post-internship plans, and how I could hopefully continue my work on Servo. It's been a long journey from feeling absolutely overwhelmed trying to understand the Worklet specifications, the thread pool, and Rust in general, to finally starting to find some semblance of comfort working on Servo. Servo is the most complex and most rewarding software project I have ever worked on. I am grateful to my Outreachy, and my Outreachy mentors Eri and Mukilan for answering a lot of my questions and providing so much support throughout the internship.

Background

The state of Worklets in Servo before and what we needed to change

When I started my internship, Servo had an existing Worklet implementation and it was deeply tied with the Paint Worklet.

The script loading mechanism for the Worklet's addModule method was incorrect* as it used classic scripts (think a script tag without a type attribute) instead of JavaScript Modules (think about <script type="module">) as the addModule specification says that addModule should use the fetch a worklet script graph algorithm.

The Worklet used a three thread pool, and the usage of the three worklets were hardcoded in the Worklet implementation. The three thread architecture was specifically designed for Paint Worklets.

To try and understand how the existing Worklet Thread Pool worked, I created this diagram:

"Diagram of three threads used in the Worklet"

The architecture works well for the Paint Worklet because the Paint Worklet does not continuously run in the background, the thread can be destroyed and recreated. This design was simply not plausible for Audio Worklets as the Audio Worklet requires a single stateful thread to be able to continuously process Audio in the background.

The Worklet Thread Pool was also being initalized by the script thread while the Thread Pool was not using any values associated with the Script thread. The initialization of the Worklet Thread Pool becomes more relevant to the Audio Worklet because exactly one Audio Worklet is possessed by a Base Audio Context (see Audio Worklet specification), we want the Audio Worklet Thread Pool to be able to exist independently of the script thread. And that's what I did in this PR.

Fetching Modules instead of Classic Scripts

This section talks about the different changes you might see in my pull request that fixes the Worklet implementation to use modules instead of classic scripts.

Fetching modules take place on the main script thread, quoting a note on the Worklet specification:

when addModule() is called, user agents can fetch the module graph on the main thread, and send the fetched source text

The event loop and the worklets exist on separate thread, and to communicate between separate threads in Servo, a ScriptEventLoopSender is used.

The existing Worklet implementation used an evaluate_js, changing the Worklet to use a fetch module implementation required adding a microtask queue on the WorkletGlobalScope.**

A crucial part of managing the microtasks queued on the WorkletGlobalScope is the TaskManager. The TaskManager stores the Script Event Loop Sender, which is used by the Script thread to send control messages to a Worklet, the TaskManager also stores the a closing of type Arc<AtomicBool> so that the script thread knows when to stop sending control messages.

There are additional changes in global_scope.rs that are used to make sure that the script thread is able to get the TaskManager present on the Worklet and call the perform_a_microtask_checkpoint defined on the Worklet.

Waking Up Threads

After implementing the fetch modules implementations on Worklet, we realized that threads were never receiving the Control messages. If you look back at the Worklet Thread Pool diagram, you might notice that the Data Messages take a priority as they execute the main Worklet tasks. The main event loop of the worklet starts by waiting for Data messages.

This turned out to be the issue that was blocking the control messages sent by the script thread after it was done fetching the modules. However, it was not as simple as waking up the thread we were sending the control message to, note how the hot backup thread is expected to take over the primary worklet thread.

This meant that the senders and receiver pairs for Worklet Data change as the threads take different roles. This meant that we could just not wake one thread up as there is a possibility that it switches role after waking up and we continue being blocked on the data message. To fix the issue, we wake up all the threads when sending a control message to the main script thread. The decision did introduce a small inefficiency but the alternative would have been much more complex to implement.

Changing the Worklet Thread Pool to a trait

Pull Request: https://github.com/servo/servo/pull/47064

As talked about earlier, the Worklet Thread Pool in Servo started as a struct that was hard coded for the Paint Worklet.

The WorkletThreadPool should be an abstraction that implements the addModule method, and at the same time we should be able to use the Worklet Thread Pool for both the Paint Worklet (stateless, three-thread worklet thread pool) and Audio Worklet (stateful, single-thread worklet thread pool).

We decided to turn the WorkletThreadPool into a trait. This allows us to turn the existing Worklet Thread Pool implementation into a Stateless Worklet Thread Pool that will implement the base WorkletThreadPool trait and we will have the right abstraction to create a stateful Worklet Thread Pool.

The WorkletThreadPool was wrapped with OnceCell to lazily initialize the Thread Pool. The initialization took place when addModule was called. While this worked for Paint Worklet. Each AudioContext needs it's own Worklet thread, so the Worklet Thread Pool should logically live on the AudioContext, not the ScriptThread.

And since we are switching the WorkletThreadPool to be a trait. addModule won't know which Thread Pool to construct. In order to preserve the laziness of the initialization of the thread pool we decided to switch to a LazyCell.

Instead of initializing the thread pool inside addModule, A Worklet would take a Boxed closure which would contain WorkletThreadPool::spawn(worklet_global_scope_init). The worklet_global_scope_init would be specific to the Window and is used to create the global_scope on WorkletGlobalScope.

To get the necessary fields for global_scope the From<&Window> is implemented for WorkletGlobalScope.

Incorrectness of the Worklet implementation

It was not possible to implement everything perfectly due to limited time of my internship as well as efforts required to get a lot of different things working together.

  1. WorkletGlobalScope is currently defined as a subclass of GlobalScope in Servo. According to the specification, WorkletGlobalScope should be a standalone class on its own.

    [Exposed=Worklet, SecureContext]
interface WorkletGlobalScope {};

    It was simply convenient to use the methods inherited from the GlobalScope, adding the infrastructure to Worklet to call those methods will take times and efforts.

  2. The addModule implementation does not correctly propagate the rethrow error in the step 6.4.2.1.1.2. We decided to skip the change as it would have made the scope of the pull request larger, we would need to read the specification closely, understand the behavior of other web browsers, and possible reopen this accidentally closed CSS houdini issue which indicates that the specification might be buggy.

  3. The addModule implementation does not have a module responses map. The module responses map allows the fetch a worklet script graph to cache the fetch responses. (step 3 in the fetch a worklet script graph spec). The addModule makes multiple fetch requests that can possibly be cached.

AudioWorklet – what's left of it

In the current state of Servo, here is a high level overview of steps left for implementing AudioWorklets

  1. Create a StatefulWorklet with a single thread.
  2. Define the AudioWorklet related bindings in Servo.
  3. Modify the servo/media crate and create an AudioNode to interface with the AudioWorklet
  4. Similar to PaintWorkletGlobalScope, invoke_a_paint_callback, we need to implement a method on the AudioWorkletGlobalScope that executes the JavaScript from the Worklet.
  5. Connect the AudioNode and AudioWorklet with the Worklet Executor to trigger the callback mentioned in the last step.

It's hard to say for sure what kind of challenges might come implementing the steps above.

How to help Servo and me continue work on the AudioWorklet

If you have enjoyed reading the article so far, I think donating to Servo is a great way to support the project and help the open web.

I really enjoyed the work I did in the last 3 months, including writing blog posts, learning about Servo and Rust. As my internship comes to an end, I have decided that I will continue my work on the AudioWorklet in my own time. If you have experience working with Rust, Servo, or GStreamer, it's likely you can help me!

I plan to start create thread(s) in Servo's zulip so that other more experienced folks in the community can help me continue the AudioWorklet, since my internship is ending and I will be working as an independent contributor, my contributions and all the help will be voluntary.

Servo and organizations like NLNet do fund individuals to work on certain features, but that's more down the line for me as I am early in my Rust journey and need a lot more guidance and help to contribute to Servo. I personally aspire to one day be able to work more complex features and being able to guide other people. I got to start somewhere :)

Learnings

The biggest lesson I learnt during the contribution period was that open source communities can be kinder and more helpful than what I initially expected.

Throughout the internship I have learnt a lot about channels, multi threaded programs, the web platform, Servo internals etc. I feel much more confident in contributing to larger projects now.

A lot of challenges came as we made changes to worklets, and a part of my work was to investigate and figure the challenges out and discuss them with my mentors.

For example, recently, we were talking about how in some of my ongoing work, WorkletThread::perform_a_worklet_task and a run_task method we created were duplicating the same functionality. I ended up spending a lot of time trying to understand how we could just use the WorkletExecutor to perform both the tasks.

I ended up creating this whiteboard on logseq: A huge diagram showing how different Worklet related methods Servo call each other

At the end, we decided that now is not the right time to refactor the WorkletExecutor.

Career Prospects and post-internship plans.

Any discussions about my career prospects without talking about AI in the current job market would be incomplete.

Servo is a unique project from multiple standpoints. Servo does not allow AI contributions, while the broader industry has rushed to adopt AI. Servo is one of the major Web Rendering Engine written in Rust, and considering there aren't other browsers that use a lot of Rust, Servo stands out as a unique project. The unique circumstances of complexities and novelties of Servo are a good enough reason to not allow AI contributions.

I also believe that not using AI has helped me improve my programming and communication skills. Zig, the programming language for example, does not allow AI contributions because they see Zig as an educational project.

A lot of engineers I respect use AI, but I think a wide consensus among them is to thoroughly review the code that the AI generates. I believe that writing code the hard way has helped me become better at reviewing code. I think it's really easy to become passive about reviewing code if the code feels challenging, and writing code by hand and reading the documentation of each library or crate you use is a really fun way to understand everything very well and become a better reviewer.

I do not have a job lined up and I am currently looking for employment. If you can refer me to a company, or have recommendations of companies to apply to, I would highly appreciate if you contact me, you can find me with the handle @niyabits almost everywhere on the internet and my email is niyabits@disroot.org

Helping me find employment is another great way to ensure that I can smoothly continue voluntarily contributing to Servo and other open source projects without worrying about my finances. I recently moved to Bengaluru and I am open to relocation, remote and in-person positions.

You can also find me on Fediverse.

Thanks for reading!

*The incorrectness can be caused by the fact that specification was updated and they started requiring Worklets to use JavaScript modules.

**In the fetch a worklet script graph implementation, when the loading promise here is resolved, it queues a task to run on_complete and we need a microtask queue for these tasks. A clone of the on_complete closure is called inside the created task in the method.

LinkedIn: https://www.linkedin.com/in/niyabits/