[Feature Request] CCXTStore support for ccxt 'rateLimit'

[JaCoderX]

Every crypto exchange have a limit for how many request the user is allowed to do in a given time.
this can be controled in ccxt project under the 'rateLimit' param.

The problem is that we instantiate the exchange in at least 2 places:
Feed and Broker (and we can create multiple feeds as well)

the solution is to warp ccxt under a singleton class per exchange
(for the case we want to work with feeds/brokers from multiple brokers)

note:
when a broker instance is needed the feed's ccxt instance will probably need to get replaces so private and public method can be accessed

[bartosh]

The problem is that we instantiate the exchange in at least 2 places:
Feed and Broker (and we can create multiple feeds as well)

yes, and we can use different rateLimit and any other parameters for those. What's is the problem then?

the solution is to warp ccxt under a singleton class per exchange

That would make it impossible to use different exchange configuration for different feeds I guess.

[JaCoderX]

yes, and we can use different rateLimit and any other parameters for those. What's is the problem then?

ccxt internally make use of rateLimit param. this is part of the ccxt mechanism to guard the user form over making requests. it will come to more importance if you have for example 10 different feeds from the same exchange

That would make it impossible to use different exchange configuration for different feeds I guess.

I need to investigate more about this, but in general the idea is to wrap ccxt in a small class and that class should be a singleton. then calling the new class from the feed or broker

[bartosh]

ccxt internally make use of rateLimit param. this is part of the ccxt mechanism to guard the user form over making requests. it will come to more importance if you have for example 10 different feeds from the same exchange.

I'm still not getting what's wrong with passing rateLimit (or any other parameters) this way:

bt.feeds.CCXT(exchange='bitfinex', symbol='EOS/BTC', config={'rateLimit': 2000})

or this way:

broker = bt.brokers.CCXTBroker(exchange='gemini', currency='USD', config={'rateLimit': 2000})

[kikingalgo]

@bartosh
it is the feed that takes the rateLimit - i use it as following
feed_config = {'rateLimit': 1000, 'enableRateLimit': True, 'symbol' : 'BTC/USD',}

data = bt.feeds.CCXT(exchange='bitmex', symbol=symbol, name="btc_usd_min",
timeframe=bt.TimeFrame.Minutes, compression=1,backfill_start=True,fromdate=hist_start_date,config=feed_config)

[bartosh]

yes, and what's wrong with it? Why do we need to change this?

[kikingalgo]

i am not sure it is an issue as described - even if we access the exchange from the feed and from the broker - still we are safley under the rate limit - (2 fetches per minute for feed, another 5-6 requests for - balance, orders, position, create, cancel ...

i would like to understand more ....

[JaCoderX]

let's say bitmex only allow 60 requests per minute.
if you exceed it you can't make any request for an hour.

now ccxt can manage how many request you are really doing and can limit the request rate so you don't exceed the exchange limit.

but because we have more then one instance of ccxt exchange, each one doesn't exceed the limit but combine they can

[JaCoderX]

kikingalgo think of a scenario where you want to work with many feeds for the same exchange.
for example 10 different currencies and now the broker would need to make also more requests for the position, order book, orders... on each currency.

you can easily exceed the limit

[bartosh]

@JacobHanouna thanks for the explanations. Now it's clear for me I think.

[kikingalgo]

@JacobHanouna now it make sense ... !
thanks

[sirhc78]

Hi There,
Regarding exchange limit, Bitmex has this rule : "Requests to our REST API are rate limited to 300 requests per 5 minutes. This counter refills continuously. If you are not logged in, your ratelimit is 150/5minutes."
How do you set your API-Key to Ccxt.feeds ?

I am trying to understand why I can't get any data from this exchange with simple code :

`import ccxt
import backtrader as bt

class SmaCross(bt.SignalStrategy):
def init(self):
sma1, sma2 = bt.ind.SMA(period=10), bt.ind.SMA(period=30)
crossover = bt.ind.CrossOver(sma1, sma2)
self.signal_add(bt.SIGNAL_LONG, crossover)

cerebro = bt.Cerebro()
cerebro.addstrategy(SmaCross)

data = bt.feeds.CCXT(exchange="bitmex", symbol="BTC/USD",fromdate=datetime(2018, 8,20),todate=datetime(2018, 8, 31), timeframe=bt.TimeFrame.Minutes)

cerebro.adddata(data)
cerebro.run()
cerebro.plot()`

Code is stuck here :
File "D:\Dev2\Anaconda3\lib\site-packages\backtrader\stores\ccxtstore.py", line 95, in retry_method
time.sleep(self.exchange.rateLimit / 1000)
KeyboardInterrupt

What I am doing wrong ?

[kikingalgo]

@sirhc78 try adding compression = 1 (for 1 minute) in the feed

data = bt.feeds.CCXT(exchange="bitmex", symbol="BTC/USD",fromdate=datetime(2018, 8,20),todate=datetime(2018, 8, 31), timeframe=bt.TimeFrame.Minutes,compression = 1)

[sirhc78]

Thank you for your answer.
I get the same behavior.
How to control that I am downloading few datas from bitmex ?

[bartosh]

@sirhc78

How do you set your API-Key to Ccxt.feeds ?

Would something like this work for you?

feedconf = {'apiKey': '<put your api key here>', 'secret': '<put your secret here>'}

data = bt.feeds.CCXT(..., config=feedconf)

[sirhc78]

Yes. It is available for the brokerage part. Does it exist for feeds ?