Filter addFiles between hooks #370
Conversation
* Make addFiles call files-added hook with a file list filtered by every individual file-added like in v2 and old synchronous v3 functions * Cleanup: removed the intermediary state objects and array * Cleanup: rename flowfiles array to flowFiles
ilessiivi
force-pushed
the
alwaysFilterAddFiles
branch
from
92e9218 to
827ecf4
Compare
ilessiivi
force-pushed
the
alwaysFilterAddFiles
branch
from
827ecf4 to
a44217c
Compare
| var flowfiles = await Promise.all(states); | ||
| for (let ff of flowfiles) { | ||
| await this.hook('file-added', ff, event); | ||
| if(flowFile && flowFile.file) { |
There was a problem hiding this comment.
formatting is off, space should be placed after an if
There was a problem hiding this comment.
I've a problem with this. When multiple files are added this is expected to be a (mostly) independent process for each of them.
Accumulating the (async) bootstrap (and possibly async initFileFn) then waiting for all of them at once is superior as it allows for parallel non-blocking initialization processes to occurs (FlowFile.bootstrap()).
Then, only file-added hooks were run sequentially (awaiting in a loop for each of them).
With this change, we'd now wait twice in a loop. It's inefficient and contrary to parallelization abilities offered by Promises.
(While talking about this, would you mind having a look at #368?)
| it('should validate file-added filtering before files-added', async function() { | ||
| var valid = false; | ||
| flow.on('file-added', (flowFile) => { | ||
| if(flowFile.name === 'f2') { |
There was a problem hiding this comment.
formatting is off, space should be placed after an if
3 participants
files-addedhook with a file list filtered by every individualfile-addedhookfiles-addedhook won't receive files thatfile-addedhas deleted