A little bit about compute in Varnish Cache and some Deno JS benchmarks

Hey all. I recently wrote about TinyKVM, a sandbox with native performance. This time I want to write about how you can try it out as a compute framework in Varnish Cache. I’ve also been very anxious (to say the least) about whether or not my theories hold up in practice. Is TinyKVM really the fastest way to sandbox compute workloads? What about per-request isolation? I’ve invited Laurence Rowe to write with me about his adventure in embedding the Rust-based Deno JavaScript runtime in TinyKVM.

I wrote previously that TinyKVM runs regular Linux ELFs and I use this to test my programs on the terminal like a regular program before I load them in Varnish. When running TinyKVM embedded in Varnish, there is a small API to facilitate receiving a HTTP request and writing back a response. Which is really the crux of the whole thing: I specifically designed it to be just a dumb “here’s a request” and “where’s the response?” type of thing:

In the example GBC program we call a function in the storage VM from a client request with the gamepad inputs, which then contributes to the next frame. The data we return from storage is an encoded PNG of the current frame.

One thing I noticed is that keys tend to be held for a very long time (compared to a single frame) until they change. So I also made it predict the next frame by assuming the same input. So:

The last point is the important one: We can schedule something to happen in-between storage requests. A function call which storage will schedule outside of request VM accesses. This reduces latency because most of the work is in encoding the PNG. You can find the source code here.

There’s more stuff, of course. Like the ability to hotswap out the GBC program with an updated version while keeping the state. We can make the Gameboy green without restarting the game! But I have to stop now or it will never end.

Have a look at the program examples repository. C/C++ has the most complete API with languages that understand C headers coming in very close, like Zig. Zig is really up there, but I do think that people who write Zig would appreciate a more Zig-idiomatic API despite it understanding C headers. Right? I asked for an opinion on my Rust API and it got slaughtered. It’s clearly more C-like than I had realized. People have also asked for Python examples, and I will not work on that until after loading dynamically linked executables is supported. It’s coming, though!

I will add that if you feel the API for a language is lacking, make an issue and let me know. My time isn’t infinite, but I’ll do my best.

Otherwise, there’s example programs for many things. There’s WebP and AVIF transcoders. Zstandard and gzip compressors. The usual things.

I invited Laurence Rowe to write this bit. He does an excellent job running realistic benchmarks, comparing them to existing solutions.

With most web UIs now written in JavaScript, server rendering is necessary to provide the best experience for initial page loads. Ideally such UI code would be run with per-request isolation to avoid the possibility of leaking data between requests — all too easy if a variable declaration is placed at the wrong level — but current options are prohibitively slow.

Existing approaches for per-request isolation use V8 isolates or process forking but these incur several milliseconds of additional latency. WebAssembly can provide microsecond latency for per-request isolation but is incompatible with JIT, so slower execution outweighs the lower startup latency when executing more complicated JS such as rendering a page with React.

Seeing TinyKVM’s performance numbers was really exciting, it clearly works extremely well for many types of programs so how would it fare with JavaScript? V8 alone lacks much of the web platform support necessary to easily run much real world code. Deno provides a mature, fully featured JS runtime built on top of V8 which is largely compatible with the web platform and Node.

The current proof of concept implementation achieves per-request isolation with ~0.4ms of additional latency when rendering a complex page with React taken from a real site. Median rendering times on my system are 0.57ms under stock Deno without per-request isolation and with GC running on background threads. 0.72ms running single threaded under TinyKVM without per-request isolation and 0.95ms with per-request isolation.

This appears to be the fastest option currently for running substantial JavaScript programs under per-request isolation by an order of magnitude! Executables with smaller memory footprints have even lower latencies on TinyKVM and we will continue looking for further optimizations.

The current proof of concept builds a static executable that runs deno_runtime in single threaded mode and provides just enough integration with the Varnish TinyKVM API to run some benchmarks.

Support was added to TinyKVM to run Rust’s crt-static support used when statically linking with glibc, avoiding the complexity of building Deno under musl. And a new wait_for_requests_paused API was added to allow JS to synchronously call into the Varnish API as a host function, avoiding an extra trip around the event loop that the callback API would require.

It’s not yet clear how Deno under TinyKVM will look longer term. It’s possible it could become a Deno extension if Deno adds an option to run single threaded and TinyKVM gains the ability to run dynamically linked executables. But there are many V8 build options to explore which might make a custom build worthwhile.

Zlib-ng provides at least a 33% performance boost over the zlib embedded in Varnish for these relatively small payloads. libdeflate claims to be significantly faster than alternatives, and really delivers on that promise.

Otherwise, benchmarking can be puzzling sometimes:

This is a benchmark of per-request isolation of a small Deno JS program first without hugepages and then with them enabled. Not for the whole main memory, and not for all of the working memory in request VMs. Just enough pages each to cover read-only segments in the main VM and the hot path in request VMs. One could speculate that hugepages reduces the number of IPIs required per request in addition to the reduced pagetable walking. Either way, it’s a 12% performance boost from a run-time setting, as the program is unmodified. I think that’s a source of my continued bewilderment when doing all this: We’re gaining a lot of performance without recompiling programs just by enabling a setting in TinyKVM.

Fun fact: Laurence’s Deno JS UI renderer.js is at 252k lines of JavaScript!?

We can see that TinyKVM provides high performance sandboxing, not just in raw compute but also in per-request isolation:

It can perform per-request isolation of a tiny program in on average 14us in a HTTP benchmark on my machine. That’s 14 micros end-to-end!

The TinyKVM compute framework in Varnish aims to trivialize data processing. Being directly embedded in Varnish it provides access to the cache and the ability to directly cache data. I hope that some of what the framework can do has been made clear.

It’s weight off my shoulders to know that there isn’t anything fundamentally badly scaling in TinyKVM. It’s also weird to think about that I made a whole new reset mechanism because of the Deno runtime. But I will say that I’m glad that I did. The new reset mechanism proves quite a bit faster for certain programs! Is it really a magnitude faster than other solutions for per-request isolation!?