Bitmex live and TimeFrame.Ticks erros

[dedeco]

When I tried to run live using Bitmex exchange I got some errors:

data = bt.feeds.CCXT(exchange='bitmex', symbol='BTC/USD', timeframe=bt.TimeFrame.Ticks, compression=1)

Error (1):

***** DATA NOTIF: LIVE
Traceback (most recent call last):
  File "run_backtrader_ccxt.py", line 43, in <module>
    cerebro.run()
  File "/Users/dedeco/Envs/backenv2/lib/python3.6/site-packages/backtrader/cerebro.py", line 1127, in run
    runstrat = self.runstrategies(iterstrat)
  File "/Users/dedeco/Envs/backenv2/lib/python3.6/site-packages/backtrader/cerebro.py", line 1298, in runstrategies
    self._runnext(runstrats)
  File "/Users/dedeco/Envs/backenv2/lib/python3.6/site-packages/backtrader/cerebro.py", line 1542, in _runnext
    drets.append(d.next(ticks=False))
  File "/Users/dedeco/Envs/backenv2/lib/python3.6/site-packages/backtrader/feed.py", line 404, in next
    ret = self.load()
  File "/Users/dedeco/Envs/backenv2/lib/python3.6/site-packages/backtrader/feed.py", line 476, in load
    _loadret = self._load()
  File "/Users/dedeco/Envs/backenv2/lib/python3.6/site-packages/backtrader/feeds/ccxt.py", line 89, in _load
    return self._load_ticks()
  File "/Users/dedeco/Envs/backenv2/lib/python3.6/site-packages/backtrader/feeds/ccxt.py", line 146, in _load_ticks
    trade_id = int(trade['id'])
ValueError: invalid literal for int() with base 10: 'e6476f26-f09a-6972-bf1e-bf123e00e160'

The id from bitmex is not int, so I fixed the problem the lines 145 - 154 from ccxt.py (feeds) changing to int and add some comparation by timestamp:

        for trade in trades:
            trade_id = trade['id']

            ts  = datetime.strptime(trade['info']['timestamp'], '%Y-%m-%dT%H:%M:%S.%fZ').timestamp()

            if ts > self._last_ts:
            #if trade_id > self._last_id:
                trade_time = datetime.strptime(trade['datetime'], '%Y-%m-%dT%H:%M:%S.%fZ')
                self._data.append((trade_time, float(trade['price']), float(trade['amount'])))
                self._last_id = trade_id

[dedeco]

I would like to up a PR if possible.

[bartosh]

@dedeco thank you for the report. It looks like this is caused by this PR

@JacobHanouna willing to fix this?

[JaCoderX]

@dedeco your solution is based on '_last_ts' which get updated in '_fetch_ohlcv'.
but not all exchanges have ohlcv so this path in the code doesn't always get called.

@dedeco have a look at the PR solution . it is a small fix to make '_last_id' get updated. (_last_id was initialize to '' which isn't equal to None).
and the assumption was trade_id is an int.

try to make a solution that doesn't use trade_id as an int

[dedeco]

Hi Guys.

I'm newbie in backtest platforms and crypto and share market, sorry if I had suggested a crazy thing.

Let me explain again with more details:

• When I use exchange= 'bitmex' and symbol= 'BTC/USD' the error is caused because on line 146 (in ccxt.py) try coerce a string UUID to INT. In bitmex data the id is some string UUID, eg: e6476f26-f09a-6972-bf1e-bf123e00e160. So is not possible coerce to INT, then the error ValueError.

trade_id = int(trade['id'])

• I have changed this line above just because is not possible compare a increase value of id, because is not INT. So I decided to compare by timestamp.

if trade_id > self._last_id:

Here is the example trades (object) in line 145 (for trade in trades:):

[{'id': 'e4db2886-ae1d-4191-c516-566fb703365c', 'info': {'timestamp': '2017-01-01T00:00:36.806Z', 'symbol': 'XBTUSD', 'side': 'Buy', 'size': 500, 'price': 968.49, 'tickDirection': 'PlusTick', 'trdMatchID': 'e4db2886-ae1d-4191-c516-566fb703365c', 'grossValue': 51627000, 'homeNotional': 0.51627, 'foreignNotional': 500}, 'timestamp': 1483228836806, 'datetime': '2017-01-01T00:00:36.806Z', 'symbol': 'BTC/USD', 'order': None, 'type': None, 'side': 'buy', 'price': 968.49, 'amount': 500}, {'id': 'b496b1fd-6e6e-7cf8-c4bf-f73994a37b42', 'info': {'timestamp': '2017-01-01T00:00:36.806Z', 'symbol': 'XBTUSD', 'side': 'Buy', 'size': 97, 'price': 968.5, 'tickDirection': 'PlusTick', 'trdMatchID': 'b496b1fd-6e6e-7cf8-c4bf-f73994a37b42', 'grossValue': 10015444, 'homeNotional': 0.10015444, 'foreignNotional': 97}, 'timestamp': 1483228836806, 'datetime': '2017-01-01T00:00:36.806Z', 'symbol': 'BTC/USD', 'order': None, 'type': None, 'side': 'buy', 'price': 968.5, 'amount': 97}, {'id': '1cc07aba-44e0-825c-03e2-837a88b0ea23', 'info': {'timestamp': '2017-01-01T00:00:36.806Z', 'symbol': 'XBTUSD', 'side': 'Buy', 'size': 100, 'price': 968.52, 'tickDirection': 'PlusTick', 'trdMatchID': '1cc07aba-44e0-825c-03e2-837a88b0ea23', 'grossValue': 10325000, 'homeNotional': 0.10325, 'foreignNotional': 100}, 'timestamp': 1483228836806, 'datetime': '2017-01-01T00:00:36.806Z', 'symbol': 'BTC/USD', 'order': None, 'type': None, 'side': 'buy', 'price': 968.52, 'amount': 100}, {'id': '83bdd6ed-4a1b-2957-2df7-fb558e931ea5', 'info': {'timestamp': '2017-01-01T00:00:36.806Z', 'symbol': 'XBTUSD', 'side': 'Buy', 'size': 1303, 'price': 968.74, 'tickDirection': 'PlusTick', 'trdMatchID': '83bdd6ed-4a1b-2957-2df7-fb558e931ea5', 'grossValue': 134504781, 'homeNotional': 1.34504781, 'foreignNotional': 1303}, 'timestamp': 1483228836806, 'datetime': '2017-01-01T00:00:36.806Z', 'symbol': 'BTC/USD', 'order': None, 'type': None, 'side': 'buy', 'price': 968.74, 'amount': 1303}, {'id': '939bd9b9-0315-1465-ab8d-0a1e350d1f2b', 'info': {'timestamp': '2017-01-01T00:00:39.408Z', 'symbol': 'XBTUSD', 'side': 'Buy', 'size': 999, 'price': 968.75, 'tickDirection': 'PlusTick', 'trdMatchID': '939bd9b9-0315-1465-ab8d-0a1e350d1f2b', 'grossValue': 103122774, 'homeNotional': 1.03122774, 'foreignNotional': 999}, 'timestamp': 1483228839408, 'datetime': '2017-01-01T00:00:39.408Z', 'symbol': 'BTC/USD', 'order': None, 'type': None, 'side': 'buy', 'price': 968.75, 'amount': 999}]

Now I realized that my ideia is not good because I had update the last_timestamp too. So, Can now understand the issue?

[JaCoderX]

@bartosh if we can't rely on trade_id to understand the order of ticks, is there a reason why we shouldn't change it to timestamp like we do on _fetch_ohlcv?

[bartosh]

@JacobHanouna

if we can't rely on trade_id to understand the order of ticks

Why we can't? Please, elaborate.

is there a reason why we shouldn't change it to timestamp like we do on _fetch_ohlcv

Yes, there is a reason. Unlike in _fetch_ohlcv where timestamp is unique for the bar here we can have multiple trades at the same time. Here is a quote from the example posted by @dedeco above to demonstrate this:

{'id': 'e4db2886-ae1d-4191-c516-566fb703365c', 'info': {'timestamp': '2017-01-01T00:00:36.806Z', 'symbol': 'XBTUSD', 'side': 'Buy', 'size': 500, 'price': 968.49, 'tickDirection': 'PlusTick', 'trdMatchID': 'e4db2886-ae1d-4191-c516-566fb703365c', 'grossValue': 51627000, 'homeNotional': 0.51627, 'foreignNotional': 500}, 'timestamp': 1483228836806, 'datetime': '2017-01-01T00:00:36.806Z', 'symbol': 'BTC/USD', 'order': None, 'type': None, 'side': 'buy', 'price': 968.49, 'amount': 500}, 

{'id': 'b496b1fd-6e6e-7cf8-c4bf-f73994a37b42', 'info': {'timestamp': '2017-01-01T00:00:36.806Z', 'symbol': 'XBTUSD', 'side': 'Buy', 'size': 97, 'price': 968.5, 'tickDirection': 'PlusTick', 'trdMatchID': 'b496b1fd-6e6e-7cf8-c4bf-f73994a37b42', 'grossValue': 10015444, 'homeNotional': 0.10015444, 'foreignNotional': 97}, 'timestamp': 1483228836806, 'datetime': '2017-01-01T00:00:36.806Z', 'symbol': 'BTC/USD', 'order': None, 'type': None, 'side': 'buy', 'price': 968.5, 'amount': 97},

[bartosh]

@JacobHanouna

@dedeco have a look at the PR solution . it is a small fix to make '_last_id' get updated. (_last_id was initialize to '' which isn't equal to None).
and the assumption was trade_id is an int.

I still don't understand why your PR changed type of the _last_id. I think it should be reverted and replaced by this tiny fix:

-       self._last_id = '' # last processed trade id for ohlcv
+       self._last_id = None # last processed trade id for ohlcv

[JaCoderX]

Can we check for '>' using this string id?
'id': 'e4db2886-ae1d-4191-c516-566fb703365c'

If so it is a easy fix like you said

[bartosh]

@dedeco please check if current ccxt branch fixes this issue. Thanks.

[bartosh]

Closing after 4 days of waiting for feedback.
@dedeco Feel free to reopen it if the fix doesn't work for you.