Sort out host environment assumptions for wasm-unknown-unknown

[aturon]

At the moment, the wasm32-unknown-unknown target assuming nothing about its host environment. That means that std, as it stands, cannot even print to stdout.

There's some ongoing debate on a Rust PR about whether and how to approach this issue. I wanted to open an issue here just to get more visibility -- but please comment on the linked PR.

[aturon]

cc @lukewagner, I bet you have thoughts.

[aturon]

In a nutshell, the PR proposes to add a general "syscall" function which is assumed to always be imported by the wasm module. That's then leveraged to provide functionality like printing and getting the current time.

[aturon]

I posted a summary comment which is probably a good way to jump in.

[Diggsey]

@aturon Great job summarising that mega thread, here's a few more things related to specifically to this issue:

In WebAssembly, all imports must be fulfilled by the host. It's not possible to have optional imports. If we assume that at some point in the future we will care about backwards compatibility, then we're only left with four choices:

1. Use a single import, and implement something resembling the system call interface I designed in the above PR.
2. Use multiple imports, and just never add any new ones after stabilisation.
3. Require hosts to dynamically generate all required imports.
4. Use a system-call like interface internally to libstd, and define wasm imports in a separate crate, similarly to how we can link in different allocators. Periodically we release a new major version of the "imports" crate which contains any new functions we wanted to import. Old programs can continue linking to older versions of the imports crate.

For 1) some people complained about using numerical identifiers for system calls. I don't personally think this is an issue, but it's worth noting that you could store and export a mapping of indexes to names as part of the compiled wasm file if you really wanted.

For 2) the downsides are self-evident.

For 3) it places the burden of backwards compatibility on users: we will never be able to provide this generator ourselves, because this target is intended to work anywhere in any host language, and so we must carefully specify exactly what algorithm should be used to generate imports from a list of names. This generator will be complex to implement as we must use name mangling to encode the type signatures of functions that we import. Furthermore, dynamically generating imports may not be easy or even possible for all hosts.

For 4) from the point of view of libstd itself, this is technically the same solution as 1). The benefit is that it provides a more normal interface to the host language, at the cost of increased work to maintain this additional "imports" crate.

[Diggsey]

Another thing to add is that imports may be fulfilled either via the wasm imports section, or by linking to libraries which export those functions. This means that even if we add imports to libstd, it's possible to generate a wasm file without imports by implementing them in rust.

[fitzgen]

It is already very difficult to create small .wasm files, even with wasm32-unknown-unknown, wasm-gc, wasm-opt, and wasm-snip. I am very wary of baking in more code that I do not use and which will likely add more call graph edges to panicking and formatting infrastructure code that I have to go to trouble to remove. This sounds like a different target, if it is making assumptions about capabilities that exist in the host. For the purposes of minimizing code size, having zero assumptions is great.

[Diggsey]

@fitzgen if you really want to go bare metal, wouldn't it make more sense to go #![no_std]? Libstd is supposed to be precisely the set of things that do require capabilities from the host/os/whatever.

[rpjohnst]

The very idea of an -unknown target is that you don't know what your environment provides. The idea of a uniform way for std to talk to the environment is good; putting it in -unknown is bad.

It belongs in another target like -stdweb or -web or -node depending on how that question gets resolved- just like libc-based targets have -msvc, -gnu, and -musl. (...and -unknown for bare metal!)

[Diggsey]

@rpjohnst
It doesn't make sense to put it in -stdweb or -web or -node because none of this stuff is specific to one of those targets - it works on any target, hence the -unknown.

I don't think it makes sense on any target to have an implementation of libstd that does nothing - certainly that's not a precedent that has been set with any other targets - we don't implement a stubbed out libstd for bare-metal x86 targets.

Either we should say that if you decide to use libstd on the -unknown-unknown target, then you do have some expectations of your environment, or we should make it a #![no_std]-only target, and add a new target, let's say wasm32-unknown-unknown-rust.

Personally I don't see much value in splitting out the two targets, since one is never going to support libstd, and the other will always behave identically to the first in #![no_std] contexts, but I don't particularly care either way.

[rpjohnst]

It doesn't make sense to put it in -stdweb or -web or -node because none of this stuff is specific to one of those targets - it works on any target, hence the -unknown.

That's fine, call it -rust or -std. That's not the issue.

Either we should say that if you decide to use libstd on the -unknown-unknown target, then you do have some expectations of your environment, or we should make it a #![no_std]-only target, and add a new target, let's say wasm32-unknown-unknown-rust.

I'm talking about the former. If you use std on an -unknown target, it should fail to build unless the the crate graph provides what it needs- not lie about successfully linking when it actually produced an incomplete binary.

This is how everything else works. If you try to build a #![no_std] executable without defining the necessary lang items, the build fails. If you try to build a -sys crate without providing the library, the build fails.

When you treat that incomplete binary as a successful build, you are placing requirements on the environment (just like, say, an installed libc) and must use a target that expresses that.

the other will always behave identically to the first in #![no_std] contexts

I don't think this is quite enough. If you want to use std without the standard syscall interface, you should be able to use the -unknown target for that. #![no_std] disabling it is insufficient.

This all ignores the portability lint, too. We definitely want to disable parts of std when running on -web, and conflating that with -unknown makes that harder.

[Diggsey]

I don't think this is quite enough. If you want to use std without the standard syscall interface, you should be able to use the -unknown target for that. #![no_std] disabling it is insufficient.

That doesn't make sense: how can using #![no_std] not be sufficient? What do you expect using libstd to do if it doesn't have access to any system calls?

This all ignores the portability lint, too. We definitely want to disable parts of std when running on -web, and conflating that with -unknown makes that harder.

You keep bringing up web but this has nothing to do with web, nothing in the referenced PR or anything I've said is specific to web. I don't expect this syscall interface would even need to be used for web once there is a separate web target.

What this does conflate is a pluggable libstd with no libstd. The rationale being that you can choose between them with the #![no_std] attribute.

In case it's not clear: the whole point of the system call interface is that we don't provide the implementation. The implementation is provided for the specific host, or could even be implemented from rust.

[rpjohnst]

What do you expect using libstd to do if it doesn't have access to any system calls?

In one sense, std can do plenty without any system calls whatsoever- remember it can run whole test suites as long as you don't care about printing the results.

In the sense I meant, you could have your own host interface (e.g. for fitting into some existing Javascript library) and you get std running the same way you get #![no_core] binaries or -sys libraries going- by providing it with the necessary functionality before the compiler produces its final output.

I guess you could say I'm arguing for these syscalls to be exposed in Rust for crates to fill in if necessary, rather than forcing them to be wasm imports.

this has nothing to do with web

You misunderstand me. The web is just an example. I'm not arguing for -unknown not to support std at all. I'm arguing for it not to default to producing wasm modules with dependencies that may not be possible to meet, given that the host is unknown. If you want to use std on -unknown, you declare the imports yourself and hook them up to std yourself (where "yourself" may mean just linking a crate of course).

The reason I brought up the portability lint is that some platforms (including the web, but think of another one if you like) will be missing functionality like, say, the file system. If -unknown starts generating imports for everything you use, and then fails at runtime, that throws out any advantage the portability lint would bring.

The implementation is provided for the specific host, or could even be implemented from rust.

How could it be provided from Rust if the syscalls are all wasm imports?

[Diggsey]

I guess you could say I'm arguing for these syscalls to be exposed in Rust for crates to fill in if necessary, rather than forcing them to be wasm imports.

How could it be provided from Rust if the syscalls are all wasm imports?

Ah, this discussion makes a lot more sense now: there's no difference between wasm imports and C/Rust imports! When you generate a wasm file, any unresolved extern symbols get added as imports - that means as long as you can satisfy any wasm import by statically linking a C or Rust file the exports the required symbol.