Delays while making requests over a slow network

[bradfol]

Operating System and Hardware: Raspberry Pi 2 B running Jessie
NodeJS Version: 4.2.6
Serialport version: 3.1.1
EDIT: Also reproduced on ASUS 64-bit desktop running Debian 8.4.0
NodeJS version: 4.4.4
Serialport version: 3.1.1

Problem description

Simultaneous network requests alongside serial port writes can cause large delays to the serial port communication. When the Raspberry Pi is on a high latency connection, the serial port communication can become significantly delayed, usually multiple seconds between writes or data events. All the code should be performed async, so it is unclear why there would be a delay.

Steps and/or code to reproduce the issue

Here is a small test case to reproduce the issue. Running this on a Raspberry Pi connected to a high latency network will cause significant delays. (There is an Arduino on the other side of the serial port connection that sends back responses.)

We were experiencing this issue as part of a larger system, but have isolated it down to node-serialport while there are network requests being made. Simply unplugging the network connection (ethernet cable or Wifi adapter, we have tested both) will cause the serial port communications to significantly speed up (dozens of read/writes per second).

Further reference

I have run across a couple other examples of people experiencing similar issues:

• Serialport together with Socket.IO is blocking event loop? #165
• http://stackoverflow.com/questions/22263094/node-js-node-serialport-has-significant-write-delays

Any suggestions on how to further isolate or debug this issue would be greatly appreciated.

[reconbot]

This is a weird one. Does it reproduce on other systems or just the pi?

[bradfol]

@reconbot Agreed, it is very unusual.

I took some time today to get Debian 8.4.0 set up on a 64-bit ASUS desktop, and was able to reproduce the issue on that machine. (We have also reproduced across multiple Raspberry Pis.)

So it appears that it is not isolated to the Pi hardware.

[reconbot]

I had a great suggestion that this is due to the event queue being backed up. The network events might be queued behind serialport events and vice versa. Our write currently calls the write callback a large amount of times for writes. I've released beta serialport@3.2.0-beta1 which fixes this issue. Give it a try!

[reconbot]

Thanks to @stevekinney for this idea!

[bradfol]

@reconbot Thanks for your and @stevekinney 's help! Unfortunately the issue still reproduces on both the Raspberry Pi and the desktop machine with serialport@3.2.0-beta1.

I would be happy to perform any debugging tasks you can suggest.

[reconbot]

@bradfol I'll see if I can get more attention for this issue, we're not doing anything that should be effected by network traffic.

[soldair]

im very curious about this. =)
@bradfol its probably not related. But just wondering what device do you have plugged into serial?

[bradfol]

@reconbot Thanks for the help. This is an important issue for us, so please let me know anything that can be done to provide additional debug information.

@soldair It is an Atmel micro controller (very similar to Arduino) running an echo program. It is isolated to just the USB connection (does not have any other connectivity).

[reconbot]

@bradfol as a work around; Can you do network operations and then serialport operations? eg not at the same time

[soldair]

i should have time to test later today. I'm sure this effects serial bridging on pinoccios also and i'd like to see them still work =)

[rvagg]

Maybe try something like UV_THREADPOOL_SIZE=16 (put this before you run the application, e.g. UV_THREADPOOL_SIZE=16 node foo.js, the number can go up to 128). Just a hunch but it's possible that serialport's use of the worker threads is somehow blocking something else that needs doing. Because there are only 4 and they are shared across the various functions that are blocking within Node (and any other addon that wants to use them), it's possible to block them all up. We've seen evidence of this with LevelDB and I've been inclined to suggest we move that to its own threadpool so it doesn't disrupt other uses of the standard worker threads.

Of course ... networking is (mostly) done non-blocking in the main thread in Node so it's not clear that this is even related, it's mainly file IO that's done with the worker threads but there are some other minor things.

Also @bradfol, can you see how long the port.write() takes to come back (add a callback and count ms from call to callback)? That would probably be an interesting datapoint.

[reconbot]

@rvagg is being in our own threadpool something that as a library I can do or provide options to do?

[bradfol]

@rvagg Thank you for the suggestions. I'm traveling for the next two weeks, but I will definitely test this out as soon as I get back to the office.