(#2890) - WIP- incremental hex parse calculation

[nolanlawson]

@calvinmetcalf I think your expertise would be very useful here.

Background: we need to pull hex binaries out of WebSQL using the SQLite HEX() function because WebSQL has a fun bug. For the record, I recently filed the bug on Chromium and Safari.

This is what's causing the DOM to freeze. The proof is in this bl.ocks.org. You can run the Chrome profiler and see that parseHexString() is blocking.

I've coded up this fix, which blocks the DOM a lot less, but apparently the function is still blocking quite a bit according to the Chrome profiler. And the GIFs are still jerky. @calvinmetcalf I'm not sure what else I could be doing; I even did a jsperf to verify my hex parser. Any ideas?

[calvinmetcalf]

process.nextTick is not what you want you want
global.setImmediate||global.setTimeout.
On Oct 19, 2014 12:23 PM, "Nolan Lawson" [email protected] wrote:

@calvinmetcalf https://github.com/calvinmetcalf I think your expertise
would be very useful here.

Background: we need to pull hex binaries out of WebSQL using the SQLite
HEX() function because WebSQL has a fun bug. For the record, I recently
filed the bug on Chromium
https://code.google.com/p/chromium/issues/detail?id=422690 and (Safari)[
https://bugs.webkit.org/show_bug.cgi?id=137637].

This is what's causing the DOM to freeze. The proof is in this bl.ocks.org
http://bl.ocks.org/nolanlawson/raw/73f15f3fe612b8770e79/. You can run
the Chrome profiler and see that parseHexString() is blocking.

I've coded up this fix, which blocks the DOM a lot less, but apparently
the function is still blocking quite a bit according to the Chrome
profiler. And the GIFs are still jerky. @calvinmetcalf
https://github.com/calvinmetcalf I'm not sure what else I could be
doing; I even did a jsperf http://jsperf.com/parse-hex to verify my hex

parser. Any ideas?

You can merge this Pull Request by running

git pull https://github.com/pouchdb/pouchdb 2890-2

Or view, comment on, or merge it at:

#2899
Commit Summary

• (DOM freezes when storing very large attachments #2890) - incremental hex parse calculation

File Changes

• M lib/adapters/websql.js
  https://github.com/pouchdb/pouchdb/pull/2899/files#diff-0 (44)
• A lib/deps/parse-hex.js
  https://github.com/pouchdb/pouchdb/pull/2899/files#diff-1 (78)

Patch Links:

• https://github.com/pouchdb/pouchdb/pull/2899.patch
• https://github.com/pouchdb/pouchdb/pull/2899.diff

—
Reply to this email directly or view it on GitHub
#2899.

[nolanlawson]

Tried that, didn't improve the performance at all.

[calvinmetcalf]

Your also going to want to drastically lower the block size to like 16
bytes
On Oct 19, 2014 1:01 PM, "Nolan Lawson" [email protected] wrote:

Tried that, didn't improve the performance at all.

—
Reply to this email directly or view it on GitHub
#2899 (comment).

[nolanlawson]

I figured out a much better solution. If we just transform the \u0000 before storing and then put them back in after retrieval, we don't need hex() at all.

[nolanlawson]

this will require a migration, tho

[nolanlawson]

Here's proof that the no-hex technique works very well: http://bl.ocks.org/nolanlawson/raw/11b1af329e1f8f27fa13/

[nolanlawson]

Closing in favor of #2900. It's way better than my meager improvements, although they are still nice to have.