After upgraded to 1.11.8, new bottleneck appears

[Caoxuyang]

I'm running:

• Which Parity version?: 1.11.8 (1.11.5 before that)
• Which operating system?: Linux
• How installed?: from source
• Are you fully synchronized?: no
• Which network are you connected to?: private chain
• Did you try to restart the node?: yes

I developed a benchmark tools to test TPS performance on Parity private chain.
Before I upgrade to 1.11.8, I was using 1.11.5. And my benchmark tests went our fine.
Tps can go above 3000.

My test goes like this:

I usually start 20 processes and send totallly 200000 signed txs to the network and calculate TPS from the blocks. The network consists of one transaction node to receive RPC requests and two mining nodes.
All the nodes are running in separate VMs.
My gasLimit is big enough for more than 50000 transactions to be mined in a single block.

But in 1.11.8, I run the exactly same test mentioned above.
Tps is limited by the mining-node. (tx-node has the same behavior, cpu, memory, throughput ...)

In each block, at most 4096 txs are mined. (One transaction uses 21000 gas.)
And I will paste my configure parameters for parity:

--gasprice 0 --tx-queue-size 400000 --tx-queue-per-sender=25000 --jsonrpc-threads=8 --jsonrpc-server-threads=8
And my gasLimit is big enough, well, my previous tests proved that.

I looked through all the releases from 1.11.5 to 1.11.8, but I didn't get any idea.
Any help will be great, thanks in advance.

[Tbaut]

This looks like an interesting project, thanks for building it.
What is your actual question?

[Caoxuyang]

@Tbaut Question is everything works fine with parity v 1.11.5. (TPS was around 3000.)
But after I upgrade my parity to 1.11.8, the performance of mining node seems to appear a new bottleneck.
In each block, there are at most 4096 txs mined. (TPS is only around 800 now)
I wonder if you guys added some new tags to restrict the performance?

[debris]

@tomusdrw were there any changes to tx queue that could affect this?

[tomusdrw]

Yes: https://github.com/paritytech/parity-ethereum/blob/master/ethcore/sync/src/chain/mod.rs#L155

We only propagate up to 4096 transactions from the queue, so the mining nodes just don't have more transactions. I suppose we should make this number dynamic (dependent on the block gas limit) and propagate up to BLOCKS * block_gas_limit / 21_000 transactions from the pool, with BLOCKS = 4 | 5.

Also it would be good to implement additional propagation strategy, where each new transaction (or at least every local transaction) in the pool is propagated right away (1tx per packet) - currently propagation is triggered every second afair.

[Caoxuyang]

@tomusdrw Thx! So for now in parity 1.11.8, this parameter is kind of hard coded, right? Maybe provide a tag to allow developers to tune this value will be great.

[tomusdrw]

@Caoxuyang yes, although have a look at the submitted PR (#9265) it should solve the issue and bring the perf back. Let me know the tests results :)