r/rust • • Jan 24 '19

Embedding WebAssembly in your Rust application

https://medium.com/wasmer/executing-webassembly-in-your-rust-application-d5cd32e8ce46
166 Upvotes

47 comments sorted by

60

u/matthieum [he/him] Jan 24 '19

For a very long time, LUA has been the script language used in games because it was easily embeddable.

I wonder if WASM could change this statu-quo.

24

u/[deleted] Jan 24 '19

That would be awesome! I think the main advantage of lua is that it's easy to write, but wasm can compile from a ton of different languages...

I'd be super interested in seeing a game that used wasm instead of lua/python!

1

u/Crandom Jan 25 '19

I wonder if you could compile Lua to wasm, then just use wasm for everything.

2

u/[deleted] Jan 25 '19

We've gotten Lua working with our emscripten layer running on the old Wasmer runtime.

1

u/Crandom Jan 25 '19

What did you use for GC?

2

u/[deleted] Jan 25 '19

Lua has its own GC, so it just used that, but running inside the wasm linear memory.

3

u/redalastor Jan 25 '19

I wonder if you could compile Lua to wasm

The biggest obstacle to compile things to wasm is that it doesn't have a GC yet so you have to provide one. For webpages it prohibitive because by the time you have downloaded the GC the plain JS would have finished running.

For a game, that might not be a concern but the lack of a GC means that people are less interested to use wasm as a compile target.

It's been actively worked on though and eventually it will be a target that's as attractive to Lua and Rust.

1

u/brokenAmmonite Jan 25 '19

I wonder if that would beat the standard lua interpreter performance-wise. It might if your wasm code was jitted.

You wouldn't beat luajit, of course, but nothing beats luajit so that's not worth remarking on.

14

u/[deleted] Jan 24 '19

Yeah this is totally going to happen and it’ll be glorious. Unity (and others) will probably support it at some point

16

u/CryZe92 Jan 24 '19

I actually use wasm for loading custom user scripts into my project. And so far it seems to be a good choice. In fact I experimented with embedding a JavaScript runtime into the wasm files to support JavaScript files as well and that seems to work perfectly fine. So in theory wasm allows users to write their scripts in tons of different languages by just compiling their runtimes into the wasm as well.

9

u/zackel_flac Jan 25 '19 edited Jan 25 '19

Lua is also easy to read and easy to script. WASM is ought to be a very low level language, like assembly is. No doubt its usage will increase and we might see more WASM embedded in binaries, but at the code level, I doubt it. Would not be surprised to see some Lua -> wasm transcompilers though!

0

u/[deleted] Jan 25 '19

[deleted]

0

u/[deleted] Jan 25 '19

[deleted]

7

u/fioralbe Jan 25 '19

I would add that (at least in games) the advantage of using Lua is that you get a command console for almost free.

5

u/the_gnarts Jan 25 '19

For a very long time, LUA has been the script language used in games because it was easily embeddable.

The Lua runtime is around 24k lines of code. What’s the smallest WASM runtime out there including a language frontend?

5

u/Holy_City Jan 25 '19

I don't really care about the LoC of a runtime as much as the binary size and overhead.

11

u/the_gnarts Jan 25 '19 edited Jan 25 '19

I don't really care about the LoC of a runtime as much as the binary size and overhead.

Sizes with GCC 8.2.1 / x86_64:

sizes for -O3

| lib                 | unstripped | stripped
| src/liblua.so.5.3.5 |     293688 |   265704
| src/liblua.a        |     502028 |   258580

sizes for -Os

| lib                 | unstripped | stripped
| src/liblua.so.5.3.5 |     219752 |   187896
| src/liblua.a        |     375820 |   177116

The values are for a regular build with without the compat symbols for older Lua versions.

For size-efficiency you probably want to compare the shared version at -Os. It links against nothing except the C library (1960064 B vs. 1709504 B stripped for glibc 2.27 on Fedora).

With LTO, a bare liblua.a is 2223188 B at -O3 and 2096532 B at -Os, a minimal statically linked self-contained executable results in a 256496 B and 182408 B binary, respectively.

LTO appears to be the most size-efficient method. At -Os it even beats lhf’s amalgamated version (http://lua-users.org/lists/lua-l/2015-01/msg00334.html) which produces a 183760 B binary. Though this may be still be an option depending on your platform’s compiler.

For a meaningful comparison with WASM we’ll need the size of a size-optimized shared object for the dynamic linking case, and that of a linked binary for the static linking case. Feature wise, the result should include a frontend for a scripting language and its standard library so it can run programs an average user of the language would write.

As far as overhead is concerned, that will depend too much on the workload you envision to address it here.

-2

u/eugene2k Jan 25 '19

Not until they put a GC in it.

6

u/JZypo Jan 25 '19

I see that you can call an external function belonging to the host code. This function is okay for return value strings. Is there a way to send strings to the wasm module?

4

u/[deleted] Jan 25 '19 edited Jan 25 '19

You can, but it's not particularly simple.

From within the wasm module:

  1. Allocate space for the string.

  2. Tell the host where that memory is.

From the host:

  1. Write your string into the memory that the wasm module allocated for your string.

In the future, we'll make it easy to use apis similar to wasm-bindgen to make things like what you want to do here much easier!

10

u/[deleted] Jan 24 '19

If you're interested in working on stuff like this and are as excited about Rust and WebAssembly as we are at Wasmer, shoot me an email at [email protected]! We're located in San Francisco, but are open to remote work as well.

28

u/pure_x01 Jan 24 '19

Cool stuff. Webassembly will change the software industry as we know it. Reuse software between languages and platforms like never before .

26

u/bluejekyll hickory-dns · trust-dns Jan 25 '19

Maybe. It’s worth noting that many people saw the JVM in the same way: Java, JRuby, Scala, Kotlin, Clojure, etc.

Right now WASM looks like it’s in a better position because it’s more open, but I think it’s worth pondering why the JVM isn’t already this.

15

u/DroidLogician sqlx · clickhouse-rs · mime_guess · rust Jan 25 '19 edited Jan 25 '19

The JVM isn't really all that flexible of a runtime. The JIT is tuned for Java, the GC is tuned for Java, the instruction set was designed for implementing Java. All the other languages/recompilers had to be built around these limitations, and additionally had to be designed for Java interop to have any chance at success. Arguably only Scala, Kotlin, Clojure and Groovy* have had any real success, in my opinion mainly because they sought to improve on Java rather than try to crowbar an existing language into the JVM.

WASM, as far as I can tell, was designed as a virtual machine and flexible compilation target first and foremost, and it's building on old experiments like NaCl and asm.js, both of which had to compile to Javascript and so are rather similar to the situation with the JVM.

It also has an LLVM backend which means almost any language with an LLVM frontend can already compile to WASM. The alternative languages to Java all had to write their own compilers from scratch (though I'm sure they reused some stuff from OpenJDK).

The ABI for interop is also a lot simpler than on the JVM. It doesn't require knowledge of objects or interfaces or anything, just functions (which almost every language has in some form or another).

5

u/[deleted] Jan 25 '19

The JVM isn't really all that flexible of a runtime.

There is also CIL/CLR. I guess it started as the JVM for C#, but pretty soon they wanted to support other languages with it. The mono implementation can be embedded on the common platforms.

2

u/DroidLogician sqlx · clickhouse-rs · mime_guess · rust Jan 25 '19

Except I can't think of any new language being developed primarily for the CLI/CLR that isn't published by Microsoft. According to Wikipedia there's been a number of attempts, but most of them look to have stagnated or been abandoned.

1

u/[deleted] Jan 26 '19

There are plenty of old languages that target it though. I believe there are more targeting it compared to wasm.

0

u/[deleted] Jan 25 '19

[deleted]

1

u/DroidLogician sqlx · clickhouse-rs · mime_guess · rust Jan 25 '19

Forgot about that one, though the only high-profile use I can think of is Gradle.

10

u/fioralbe Jan 25 '19

Well there is a very big difference, nobody planned to compile C to the JVM

3

u/rebootyourbrainstem Jan 25 '19

I think it'll help a lot that Webassembly started with a simpler runtime model (flat memory) and went from there. Also that it's an open standard, and is much better integrated with the Web.

3

u/pure_x01 Jan 25 '19

I think the openess is key here. Wasm is also backed by many bug companies. There was a totally different climate when the jvm entered the world.

18

u/Crandom Jan 24 '19

This seems like a pretty cool way to enable plugins in a rust application in a technically language agnostic way.

7

u/CryZe92 Jan 25 '19

It seems like this has some problems compiling on Windows. Cranelift has problems with mingw's python and then wasmer itself references a module that doesn't exist.

2

u/Devnought Jan 25 '19

I fixed the module problem on my local version of `wasmer-runtime-core`, but the memory module that was missing previously throws a bunch of errors and cannot compile.

1

u/CryZe92 Jan 25 '19

Yeah I just ran into that too. Have you fixed any of them or should I attempt to do it myself?

1

u/Devnought Jan 25 '19 edited Jan 25 '19

The &str to String errors were simple enough.

There's a reference to the crate errno which is a unix compilation dependency.

I'm not sure if the single libc references is intended to be there, or if it should be using something in winapi.

6 items in the use winapi::um::memoryapi::{ .. } statement don't exist in that module.

I'd say go for it for trying to fix it, but I myself am not confident enough in what this module is doing to take a stab at it myself :)

10

u/ssokolow Jan 24 '19

I think you've just given me a way to move the runtime-loadable plugin system from the Python side of my rust-cpython project to the Rust side. :)

6

u/dpc_pw Jan 25 '19

Perfect! I was just looking for how exactly would I go about using WebAssembly for plugins in my experimental (Vim/Kakoune-like) text editor: https://github.com/dpc/breeze

1

u/Javyre Jan 25 '19

Neovim already has the |-cursor and I think vim and emacs can be configured to. I would consider that a pretty standard feature at this point.

1

u/dpc_pw Jan 25 '19

Not the same. You get | but under the hood it works like a block.

1

u/Javyre Jan 25 '19

Can you further explain the difference?

(Also, I'm sure it could be implemented as some plugin. Although by that logic, kakoune could've been implemented as a plugin for emacs)

1

u/dpc_pw Jan 25 '19

There is no "character under the cursor" which is how most CLI text editors work.

5

u/maninalift Jan 24 '19

This looks like exactly what I've been looking for for a couple of projects, if as I understand it I can specify at run time when loading the WASM what imports are available to it and thus how it can interact with the world.

My next question is can I achieve the same thing with Rust compiled to wasm? Ie can I safely load isolated modules interacting over a specified API into wasm from wasm. My guess is no - at least not for now but I'm interested whether it is plausible that this could be achieved.

5

u/zokier Jan 24 '19

My next question is can I achieve the same thing with Rust compiled to wasm? Ie can I safely load isolated modules interacting over a specified API into wasm from wasm

Last time I checked WASM does not support JIT, which limits your ability to do such wasmception, at least with Wasmer. But I guess you could use WASM interpreter, wasmi looks like it might work?

3

u/TUSF Jan 25 '19

WASM does not support JIT,

Neither did JavaScript, until everyone started doing it. JIT is all a matter of the interpreter/compiler/etc…

3

u/radix Jan 24 '19

Yay, I've been waiting for something like this!

1

u/matklad rust-analyzer Jan 25 '19

This is awesome! One step closer to my dream of XXI-century Emacs in Rust, with WASM runtime, sane scripting language and real GUI :-)

Thanks!

2

u/unrealhoang Jan 25 '19

There’s remacs. Elisp is quite a sane language imho

-1

u/Schmeckinger Jan 25 '19

Would it be possible to put wat support behind a feature gate?