Sending large object silently hangs

[tkf]

Evaluating

using ElectronDisplay
using VegaLite
n = 10^5
NamedTuple{(:x, :y)}.(tuple.(randn(n), randn(n))) |>
@vlplot(:point, x=:x, y=:y)

opens an Electron window but it does not show anything. Also, the REPL hangs and Ctrl-C yields the error below.

Since my implementation of load(win::Window, html::AbstractString) is rather ad-hoc, I'm not super surprised:

Electron.jl/src/Electron.jl

Lines 277 to 278 in bc0de51

	load(win::Window, html::AbstractString) =
	load(win, URI("data:text/html;charset=utf-8," * escape(html)))

But what is the best way to load (large) dynamic HTML content? I can't find anything better in webContents API. The most ad-hoc solution seems to be to use a temporary file.

---

Full stacktrace

^Cfatal: error thrown and no exception handler available.
InterruptException()
jl_mutex_unlock at /home/takafumi/repos/watch/julia/src/locks.h:156 [inlined]
jl_task_get_next at /home/takafumi/repos/watch/julia/src/partr.c:450
poptaskref at ./task.jl:660
wait at ./task.jl:667
task_done_hook at ./task.jl:401
jl_apply at /home/takafumi/repos/watch/julia/src/julia.h:1640 [inlined]
jl_finish_task at /home/takafumi/repos/watch/julia/src/task.c:197
start_task at /home/takafumi/repos/watch/julia/src/task.c:697
unknown function (ip: 0xffffffffffffffff)
Error showing value of type VegaLite.VLSpec{:plot}:
ERROR: Unexpected end of input
Line: 0
Around: ......
           ^

Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] _error(::String, ::JSON.Parser.MemoryParserState) at /home/takafumi/.julia/packages/JSON/d89fA/src/Parser.jl:142
 [3] byteat at /home/takafumi/.julia/packages/JSON/d89fA/src/Parser.jl:51 [inlined]
 [4] parse_value(::JSON.Parser.ParserContext{Dict{String,Any},Int64,true,nothing}, ::JSON.Parser.MemoryParserState) at /home/takafumi/.julia/packages/JSON/d89fA/src/Parser.jl:162
 [5] parse(::String; dicttype::Type, inttype::Type{Int64}, allownan::Bool, null::Nothing) at /home/takafumi/.julia/packages/JSON/d89fA/src/Parser.jl:463
 [6] parse at /home/takafumi/.julia/packages/JSON/d89fA/src/Parser.jl:461 [inlined]
 [7] req_response(::Electron._Application{Electron.Window}, ::Dict{String,Any}) at /home/takafumi/.julia/packages/Electron/wL84D/src/Electron.jl:227
 [8] load at /home/takafumi/.julia/packages/Electron/wL84D/src/Electron.jl:268 [inlined]
 [9] load(::Electron.Window, ::String) at /home/takafumi/.julia/packages/Electron/wL84D/src/Electron.jl:277
 [10] displayhtml(::ElectronDisplay.ElectronDisplayType, ::String; options::Dict{String,Dict{String,Bool}}) at /home/takafumi/.julia/packages/ElectronDisplay/efZyl/src/ElectronDisplay.jl:77
 [11] _display_vegalite(::ElectronDisplay.ElectronDisplayType, ::String, ::String, ::VegaLite.VLSpec{:plot}) at /home/takafumi/.julia/packages/ElectronDisplay/efZyl/src/vega.jl:47
 [12] display(::ElectronDisplay.ElectronDisplayType, ::MIME{Symbol("application/vnd.vegalite.v3+json")}, ::VegaLite.VLSpec{:plot}) at /home/takafumi/.julia/packages/ElectronDisplay/efZyl/src/vega.jl:97
 [13] display(::ElectronDisplay.ElectronDisplayType, ::VegaLite.VLSpec{:plot}) at /home/takafumi/.julia/packages/ElectronDisplay/efZyl/src/ElectronDisplay.jl:259
 [14] display(::Any) at ./multimedia.jl:323
 [15] #invokelatest#1 at ./essentials.jl:709 [inlined]
 [16] invokelatest at ./essentials.jl:708 [inlined]
 [17] print_response(::IO, ::Any, ::Bool, ::Bool, ::Any) at /home/takafumi/repos/watch/julia/usr/share/julia/stdlib/v1.4/REPL/src/REPL.jl:156
 [18] print_response(::REPL.AbstractREPL, ::Any, ::Bool, ::Bool) at /home/takafumi/repos/watch/julia/usr/share/julia/stdlib/v1.4/REPL/src/REPL.jl:141
 [19] (::REPL.var"#do_respond#38"{Bool,REPL.var"#48#57"{REPL.LineEditREPL,REPL.REPLHistoryProvider},REPL.LineEditREPL,REPL.LineEdit.Prompt})(::Any, ::Any, ::Any) at /home/takafumi/repos/watch/julia/usr/share/julia/stdlib/v1.4/REPL/src/REPL.jl:719
 [20] #invokelatest#1 at ./essentials.jl:709 [inlined]
 [21] invokelatest at ./essentials.jl:708 [inlined]
 [22] run_interface(::REPL.Terminals.TextTerminal, ::REPL.LineEdit.ModalInterface, ::REPL.LineEdit.MIState) at /home/takafumi/repos/watch/julia/usr/share/julia/stdlib/v1.4/REPL/src/LineEdit.jl:2306
 [23] run_frontend(::REPL.LineEditREPL, ::REPL.REPLBackendRef) at /home/takafumi/repos/watch/julia/usr/share/julia/stdlib/v1.4/REPL/src/REPL.jl:1045
 [24] run_repl(::REPL.AbstractREPL, ::Any) at /home/takafumi/repos/watch/julia/usr/share/julia/stdlib/v1.4/REPL/src/REPL.jl:201
 [25] (::Base.var"#772#774"{Bool,Bool,Bool,Bool})(::Module) at ./client.jl:391
 [26] #invokelatest#1 at ./essentials.jl:709 [inlined]
 [27] invokelatest at ./essentials.jl:708 [inlined]
 [28] run_main_repl(::Bool, ::Bool, ::Bool, ::Bool, ::Bool) at ./client.jl:375
 [29] exec_options(::Base.JLOptions) at ./client.jl:313
 [30] _start() at ./client.jl:492

(v1.4) pkg> st ElectronDisplay
Status `~/.julia/environments/v1.4/Project.toml`
  [a1bb12fb] Electron v1.0.0
  [d872a56f] ElectronDisplay v0.8.1
  [82899510] IteratorInterfaceExtensions v1.0.0
  [5e66a065] TableShowUtils v0.2.5
  [3783bdb8] TableTraits v1.0.0

[davidanthoff]

Yeah, this is a tricky one I've been thinking about a fair bit :) I think at the end of the day we want to move to a situation where we don't generate these very large HTML content things. The main problem for that really seems to be VegaLite, and there I think our long term strategy might be something like this:

1. Don't put the data inline into the spec
2. Variant a: Send the tabular data in the form of arrow IPC messages via the existing communication channels to the window process, and then use the vega data api to load this arrow data into vega.
3. Variant b: Create a shared memory section in the julia process, serialize the tabular data as arrow data into that shared memory section, then somehow access that same shared memory from the electron window process
4. Variant c: Create a shared library version of electron, host it inside the julia process, and then again go via arrow as an in-memory format.
5. Variant d: Save the data as a csv file to disc, and load the data from the electron window directly.

This is obviously all pretty speculative... The one thing I'm really sure about is that embedding the data in the vega-lite spec is not a good idea.

2d feels the most hacky option to me, but probably the easiest to implement. But of course we first need to sort out the problem that our version of vega right now doesn't include the file loading capability :) queryverse/VegaLite.jl#183

[PaulSoderlind]

this caught me too - in a simple application where I display a too large html string with tabular data.

[sprig]

I ran into this as well.

AFAICT from testing, loading html directly as string is limited to 1mb. Im not sure if this is a limitation of data uri spec, of chromium/electron, or in the configuration of how this package is used.

As a workaround, the following seems to work:

run(window, “document.documentElement.innerhtml=`”*html_string*”`”)

idk if it is a good idea to replace load logic with that generally since changing it this way may potentially leave some dangling resources (?), but for the initial Window creation it seems mostly safe. Perhaps as a replace method?

EDIT: probably escaping or encoding+decoding would be needed generally.