Enhance LatencyModel to handle message ingress dynamic data latency

[stastnypremysl]

Bug Report

In the given folder, there is a strategy, their output and the catalog.

Expected Behavior

Ticks are delayed according to base_latency_nanos. (ts_event - ts_init = base_latency_nanos)

Actual Behavior

Tick are not delayed. (ts_event - ts_init = 0)

Steps to Reproduce the Problem

1. Download the data
2. Extract the zip file and change extracted catalog path in strategy.py to the catalog
3. ./strategy.py >strategy.out 2>&1
4. less strategy.out

Specifications

• OS platform: Gentoo Linux (native build, amd64)
• Python version: Python 3.12.9 (native build)
• nautilus_trader version: 1.218.0 (revision c35d944; make build-wheel-debug)

Additional notes

• The simulation is running on aggregated trade ticks from Binance data collection
• The instrument information was downloaded using the NT Binance client

Further proposal

I propose to add separated latency config for ticks, as they might be received slower than orders are submitted.

[cjdsellers]

Hi @stastnypremysl

Data latency is represented as the difference between ts_event the matching engine event time, and ts_init, which is when the Nautilus system created / received the data (as described in the docs here).

There are hooks to alter this, for example here with the wranglers. In all cases some pre-processing is intended and required to alter data latency.

The LatencyModel is a different concept that relates to trading commands send to, and received from, a venue. If you look at the code there is nowhere the LatencyModel interacts with data, so therefore the system is operating to spec and this is not a bug.

I'll change this to an enhancement request for dynamic data latency, which is an interesting concept, but I'd say would be well down on the list of priorities for now when considering implementation effort vs reward/value.

[stastnypremysl]

Hi, @cjdsellers.

Thank you for the answer.

The events received from a venue should be also delayed? I am not sure, they are currently delayed after the order a filled.

May I also propose to add a variable, how much event is delayed after the order is filled?

[stastnypremysl]

I would like to further add, that I think, this feature is in my opinion important as it might help easily find bugs in user's strategy, which will reveal later at live trading.

[cjdsellers]

Hi @stastnypremysl

The latency model doesn't currently handle execution message ingress, only egress. I agree it would be more symmetrical if we also modeled the message ingress so that the full round trip is at least 2 * base_latency.

This would certainly be a useful enhancement for realism and modeling race conditions. It's just a question of how/where to implement, probably an outbound queue at the SimulatedExchange similar to what is currently _message_queue.

One potential challenge is that we would have to model two different order states:

• local state
• venue state

Currently these are both coincident and centralized at the Cache. It would be more "realistic" to maintain order state separately at the venue - but this does increase complexity and moving parts as well.

[stastnypremysl]

I think, it would be great, if it would be maintained separately. If half solution is implemented, it will be probably sooner or later changed to the better one, and it might be a work wasted, if better solution is known at the beginning.

If/when you get to the implementation, I would further recommend to consider following thing in the design, which might be needed by the users in the future:
a) d-random latency and TCP packet loss simulation (with tick and event racing implemented)
b) d-random connection loss during trading, request_data,.. (implementing this into backtests would also test a NT and its design a lot)
c) d-random event and tick reordering/dropout when applicable (this would not only test strategies, but further test NT, how is prepared for such situations, when venue starts malfunctioning)
d) d-random various CPU freezes (many users will run in the cloud, or other things causing this may casually happen,...)
e) d-random venue freezes (sometimes venue is rolling out upgrades, some of them are running in the cloud,...)

Various statistical distributions might be needed.

[stastnypremysl]

I would like to further add to the considerations, that original behavior might be still needed. (due to performance reasons during the optimization)