DOM freezes when storing very large attachments

[nolanlawson]

Steps to repro: store a huge attachment (>500KB), watch the DOM stutter. Animated GIFs are good for this.

Looks like we should probably store binaries in chunks rather than all as one big blob.

[nolanlawson]

This applies to both WebSQL and IndexedDB

[calvinmetcalf]

you could store them in a content addressable manner, aka have an attachment store where instead of storing the attachment you store a list of hashes, and those hashes corisponde to chunks stored in a differnt table which could have a hash, a binary blob, and a number representing how many things reference it (e.g. if you try to put a block but there is already one there you just increment the counter, same for deletes, just decrease it unless its at one)

[nolanlawson]

Interesting design, but is there really a lot of benefit to hashing and deduping the chunks? Do you imagine with e.g. PNG data that the chunks would be repeated a lot?

My initial idea was to simply break the attachments into chunks of e.g. 100 KB each, and then if an attachment was e.g. 300KB, it would have 3 rows in Web SQL or be three separate objects in IndexedDB. Simple.

To be fair, though, I haven't yet deduced if this is even a WebSQL- or IndexedDB-level problem, or instead related to how I'm using Blobs or maybe the MD5 digest algorithm (maybe we ought to reduce the chunk size there). I'll test some more with the Squarespace app and hopefully have some results by the end of the weekend.

[calvinmetcalf]

you'd likely get some savings if you were storing similar pngs (e.g. map
tiles) in which case there could be some substantial savings

On Fri, Oct 17, 2014 at 9:24 PM, Nolan Lawson notifications@github.com
wrote:

Interesting design, but is there really a lot of benefit to hashing and
deduping the chunks? Do you imagine with e.g. PNG data that the chunks
would be repeated a lot?

My initial idea was to simply break the attachments into chunks of e.g.
100 KB each, and then if an attachment was e.g. 300KB, it would have 3 rows
in Web SQL or be three separate objects in IndexedDB. Simple.

To be fair, though, I haven't yet deduced if this is even a WebSQL- or
IndexedDB-level problem, or instead related to how I'm using Blobs or maybe
the MD5 digest algorithm (maybe we ought to reduce the chunk size there).
I'll test some more with the Squarespace app and hopefully have some
results by the end of the weekend.

—
Reply to this email directly or view it on GitHub
#2890 (comment).

-Calvin W. Metcalf

[nolanlawson]

can see where you're coming from, but that sounds a bit application-specific

[daleharvey]

I suspect this is likely something wrong we are doing on our end, all the
blob processing to ensure its the correct type or some such

In Chrome blobs do not perform great, leveldb is just generally bad at
anything above 2MB from the recommendations the Chrome devs gave me, in
Firefox blobs even in indexeddb are just stored on plain disk, returning
them to the user should be little more than a filehandle.

I feel like this is an issue we should be looking to report and fix the
underlying storage issues than come up with fancy and complicated storage
techniques

On 18 October 2014 05:44, Nolan Lawson notifications@github.com wrote:

can see where you're coming from, but that sounds a bit
application-specific

—
Reply to this email directly or view it on GitHub
#2890 (comment).

[nolanlawson]

Excellent point, yes, especially when it comes to IndexedDB, where we should prefer to return a Blob directly from the storage engine instead of creating our own. I imagine there are big perf benefits there.

However, this might still make sense as a WebSQL-only fix, especially since WebSQL is pretty dominant on mobile devices, which are exactly where resources are most constrained, and we have to stringify the blob before storing it anyway.

[nolanlawson]

Never mind, I figured out what it really is. I profiled this thing (save locally for best results), and it turns out it's because the parseHexString() function is expensive and is synchronous. Working on a fix.

[nolanlawson]

So for the record, IndexedDB actually isn't as bad as I thought, unless you're in Chrome <38 or Android, probably due to lack of blob support: http://bl.ocks.org/nolanlawson/raw/08d5e7ddb2c854a0b6a9/