If all mining stop for few hours, network peers will colapse?

[rrguardo]

I was experiment with an independent private DASH copy network(fork), and note that after ~12 hour not mining, all the nodes get stuck synchronizing.

Is DASH daemon not prepare to work under that non realistic conditions?

Maybe I miss something, I am new with the related tech.

[codablock]

Can you show some logs of the stuck nodes?

[rrguardo]

I remember that qt wallet show all the time stuck, that was in sync process since the last generated block.
An for that reason, I am unable to continue mining.

Give me some hours to provide logs.

[codablock]

Did it show that block sync was stuck or MN sync? Was the wallet freshly started or did it run for some time and then suddenly showed a stuck sync? Did you continue mining when you've seen that and if yes, did the Qt wallet get synced?

[rrguardo]

I was mining fine for ~2 days, all the network was fine until I stop mining for ~12 hours or a little more.
After 12 hours without mining network looks stuck, daemons get forever in sync status like assuming that should exist blocks from past hours not synchronized.

In my setup network don't have yet masternodes online, but it worked in that way until mining stop.
I tried using no qt daemon and deleting all file minus wallet and new nodes are stuck specting sync blocks that don't exist.

In some hours will provide logs and more info.

[rrguardo]

Logs before the problem, at one node

2018-01-30 12:27:47 CInstantSend::CheckAndRemove -- Lock Candidates: 0, Votes 0
2018-01-30 12:28:47 CMasternodeMan::CheckAndRemove
2018-01-30 12:28:47 CMasternodeMan::CheckAndRemove -- Masternodes: 0, peers who asked us for Masternode list: 0, peers we asked for Masternode list: 0, entries in Masternode list we asked for: 0, nDsqCount: 0
2018-01-30 12:28:47 CMasternodePayments::CheckAndRemove -- Votes: 0, Blocks: 0
2018-01-30 12:28:47 CInstantSend::CheckAndRemove -- Lock Candidates: 0, Votes 0
2018-01-30 12:29:19 UpdateTip: new best=00000018303613506594fd2e2e5997f821eb392c05f4400d756be8fc920a03d0 height=1383 log2_work=34.030093 tx=1388 date=2018-01-30 12:29:18 progress=0.999992 cache=0.1MiB(341txo)
2018-01-30 12:29:19 ProcessNewBlock : ACCEPTED
2018-01-30 12:29:47 CMasternodeMan::CheckAndRemove
2018-01-30 12:29:47 CMasternodeMan::CheckAndRemove -- Masternodes: 0, peers who asked us for Masternode list: 0, peers we asked for Masternode list: 0, entries in Masternode list we asked for: 0, nDsqCount: 0
2018-01-30 12:29:47 CMasternodePayments::CheckAndRemove -- Votes: 0, Blocks: 0
2018-01-30 12:29:47 CInstantSend::CheckAndRemove -- Lock Candidates: 0, Votes 0
2018-01-30 12:29:47 CGovernanceManager::UpdateCachesAndClean -- Governance Objects: 0 (Proposals: 0, Triggers: 0, Watchdogs: 0/0, Other: 0; Erased: 0), Votes: 0
2018-01-30 12:30:47 CMasternodeMan::CheckAndRemove
2018-01-30 12:30:47 CMasternodeMan::CheckAndRemove -- Masternodes: 0, peers who asked us for Masternode list: 0, peers we asked for Masternode list: 0, entries in Masternode list we asked for: 0, nDsqCount: 0
2018-01-30 12:30:47 CMasternodePayments::CheckAndRemove -- Votes: 0, Blocks: 0
2018-01-30 12:30:47 CInstantSend::CheckAndRemove -- Lock Candidates: 0, Votes 0
2018-01-30 12:31:47 CMasternodeMan::CheckAndRemove

[rrguardo]

In the hours with zero mining we see this logs at one node:

2018-01-31 18:36:08 CMasternodeMan::CheckAndRemove -- Masternodes: 0, peers who asked us for Masternode list: 0, peers we asked for Masternode list: 0, entries in Masternode list we asked for: 0, nDsqCount: 0
2018-01-31 18:36:08 CMasternodePayments::CheckAndRemove -- Votes: 0, Blocks: 0
2018-01-31 18:36:08 CInstantSend::CheckAndRemove -- Lock Candidates: 0, Votes 0
2018-01-31 18:37:08 CMasternodeMan::CheckAndRemove
2018-01-31 18:37:08 CMasternodeMan::CheckAndRemove -- Masternodes: 0, peers who asked us for Masternode list: 0, peers we asked for Masternode list: 0, entries in Masternode list we asked for: 0, nDsqCount: 0
2018-01-31 18:37:08 CMasternodePayments::CheckAndRemove -- Votes: 0, Blocks: 0
2018-01-31 18:37:08 CInstantSend::CheckAndRemove -- Lock Candidates: 0, Votes 0
2018-01-31 18:38:08 CMasternodeMan::CheckAndRemove
2018-01-31 18:38:08 CMasternodeMan::CheckAndRemove -- Masternodes: 0, peers who asked us for Masternode list: 0, peers we asked for Masternode list: 0, entries in Masternode list we asked for: 0, nDsqCount: 0
2018-01-31 18:38:08 CMasternodePayments::CheckAndRemove -- Votes: 0, Blocks: 0
2018-01-31 18:38:08 CInstantSend::CheckAndRemove -- Lock Candidates: 0, Votes 0
2018-01-31 18:39:08 CMasternodeMan::CheckAndRemove
2018-01-31 18:39:08 CMasternodeMan::CheckAndRemove -- Masternodes: 0, peers who asked us for Masternode list: 0, peers we asked for Masternode list: 0, entries in Masternode list we asked for: 0, nDsqCount: 0
2018-01-31 18:39:08 CMasternodePayments::CheckAndRemove -- Votes: 0, Blocks: 0
2018-01-31 18:39:08 CInstantSend::CheckAndRemove -- Lock Candidates: 0, Votes 0

[rrguardo]

After some hours without mining the logs change to this at nodes

2018-01-31 21:54:20 ThreadSocketHandler -- removing node: peer=428 addr=ip2:34706 nRefCount=1 fNetworkNode=0 fInbound=1 fMasternode=0
2018-01-31 21:54:20 receive version message: /Dash Core:0.12.2.3/: version 70208, blocks=1401, us=ip:port, peer=429
2018-01-31 21:54:20 AdvertiseLocal: advertising address ip:port
2018-01-31 21:54:20 ThreadSocketHandler -- removing node: peer=429 addr=ip2:34708 nRefCount=1 fNetworkNode=0 fInbound=1 fMasternode=0
2018-01-31 21:54:21 receive version message: /Dash Core:0.12.2.3/: version 70208, blocks=1401, us=ip:port, peer=430
2018-01-31 21:54:21 AdvertiseLocal: advertising address ip:port
2018-01-31 21:54:21 ThreadSocketHandler -- removing node: peer=430 addr=ip2:34710 nRefCount=1 fNetworkNode=0 fInbound=1 fMasternode=0
2018-01-31 21:54:22 receive version message: /Dash Core:0.12.2.3/: version 70208, blocks=1401, us=ip:port, peer=431
2018-01-31 21:54:22 AdvertiseLocal: advertising address ip:port
2018-01-31 21:54:22 ThreadSocketHandler -- removing node: peer=431 addr=ip2:34712 nRefCount=1 fNetworkNode=0 fInbound=1 fMasternode=0
2018-01-31 21:54:22 receive version message: /Dash Core:0.12.2.3/: version 70208, blocks=1401, us=ip:port, peer=432
2018-01-31 21:54:22 AdvertiseLocal: advertising address ip:port
2018-01-31 21:54:22 ThreadSocketHandler -- removing node: peer=432 addr=ip2:34714 nRefCount=1 fNetworkNode=0 fInbound=1 fMasternode=0
2018-01-31 21:54:23 receive version message: /Dash Core:0.12.2.3/: version 70208, blocks=1401, us=ip:port, peer=433
2018-01-31 21:54:23 AdvertiseLocal: advertising address ip:port
2018-01-31 21:54:23 ThreadSocketHandler -- removing node: peer=433 addr=ip2:34716 nRefCount=1 fNetworkNode=0 fInbound=1 fMasternode=0
2018-01-31 21:54:23 receive version message: /Dash Core:0.12.2.3/: version 70208, blocks=1401, us=ip:port, peer=434
2018-01-31 21:54:23 AdvertiseLocal: advertising address ip:port
2018-01-31 21:54:23 ThreadSocketHandler -- removing node: peer=434 addr=ip2:34718 nRefCount=1 fNetworkNode=0 fInbound=1 fMasternode=0
2018-01-31 21:54:24 receive version message: /Dash Core:0.12.2.3/: version 70208, blocks=1401, us=ip:port, peer=435
2018-01-31 21:54:24 AdvertiseLocal: advertising address ip:port
2018-01-31 21:54:24 ThreadSocketHandler -- removing node: peer=435 addr=ip2:34720 nRefCount=1 fNetworkNode=0 fInbound=1 fMasternode=0
2018-01-31 21:54:24 receive version message: /Dash Core:0.12.2.3/: version 70208, blocks=1401, us=ip:port, peer=436
2018-01-31 21:54:25 AdvertiseLocal: advertising address ip:port
2018-01-31 21:54:25 ThreadSocketHandler -- removing node: peer=436 addr=ip2:34722 nRefCount=1 fNetworkNode=0 fInbound=1 fMasternode=0
2018-01-31 21:54:25 receive version message: /Dash Core:0.12.2.3/: version 70208, blocks=1401, us=ip:port, peer=437
2018-01-31 21:54:25 AdvertiseLocal: advertising address ip:port
2018-01-31 21:54:25 ThreadSocketHandler -- removing node: peer=437 addr=ip2:34724 nRefCount=1 fNetworkNode=0 fInbound=1 fMasternode=0
2018-01-31 21:54:26 receive version message: /Dash Core:0.12.2.3/: version 70208, blocks=1401, us=ip:port, peer=438
2018-01-31 21:54:26 AdvertiseLocal: advertising address ip:port
2018-01-31 21:54:26 ThreadSocketHandler -- removing node: peer=438 addr=ip2:34726 nRefCount=1 fNetworkNode=0 fInbound=1 fMasternode=0
2018-01-31 21:54:26 receive version message: /Dash Core:0.12.2.3/: version 70208, blocks=1401, us=ip:port, peer=439
2018-01-31 21:54:26 AdvertiseLocal: advertising address ip:port
2018-01-31 21:54:26 ThreadSocketHandler -- removing node: peer=439 addr=ip2:34728 nRefCount=1 fNetworkNode=0 fInbound=1 fMasternode=0
2018-01-31 21:54:27 receive version message: /Dash Core:0.12.2.3/: version 70208, blocks=1401, us=ip:port, peer=440
2018-01-31 21:54:27 AdvertiseLocal: advertising address ip:port
2018-01-31 21:54:27 ThreadSocketHandler -- removing node: peer=440 addr=ip2:34730 nRefCount=1 fNetworkNode=0 fInbound=1 fMasternode=0
2018-01-31 21:54:27 receive version message: /Dash Core:0.12.2.3/: version 70208, blocks=1401, us=ip:port, peer=441
2018-01-31 21:54:27 AdvertiseLocal: advertising address ip:port
2018-01-31 21:54:28 ThreadSocketHandler -- removing node: peer=441 addr=ip2:34732 nRefCount=1 fNetworkNode=0 fInbound=1 fMasternode=0
2018-01-31 21:54:28 receive version message: /Dash Core:0.12.2.3/: version 70208, blocks=1401, us=ip:port, peer=442

real ip port replaced

[rrguardo]

Network looks stuck at block=1401
Maybe was something that only affect my custom network, I change some parameters at chainparams file.

[rrguardo]

New nodes in the network get stuck in a loop with this log repeated >>

2018-02-02 01:15:23 trying connection 219.25.22.23:9999 lastseen=0.1hrs
2018-02-02 01:15:24 Added connection peer=461
2018-02-02 01:15:24 sending version (107 bytes) peer=461
2018-02-02 01:15:24 send version message: version 70208, blocks=1, us=[::]:0, peer=461
2018-02-02 01:15:24 received: version (107 bytes) peer=461
2018-02-02 01:15:24 sending verack (0 bytes) peer=461
2018-02-02 01:15:24 sending getaddr (0 bytes) peer=461
2018-02-02 01:15:24 receive version message: /Dash Core:0.12.2.3/: version 70208, blocks=1401, us=145.102.197.218:34030, peer=461
2018-02-02 01:15:24 received: verack (0 bytes) peer=461
2018-02-02 01:15:24 sending sendheaders (0 bytes) peer=461
2018-02-02 01:15:24 sending ping (8 bytes) peer=461
2018-02-02 01:15:24 initial getheaders (1400) to peer=461 (startheight:1401)
2018-02-02 01:15:24 sending getheaders (741 bytes) peer=461
2018-02-02 01:15:24 received: sendheaders (0 bytes) peer=461
2018-02-02 01:15:24 received: ping (8 bytes) peer=461
2018-02-02 01:15:24 sending pong (8 bytes) peer=461
2018-02-02 01:15:24 received: addr (31 bytes) peer=461
2018-02-02 01:15:24 received: pong (8 bytes) peer=461
2018-02-02 01:15:24 received: headers (82 bytes) peer=461
2018-02-02 01:15:24 ERROR: detected bad peer for initial headers sync, disconnecting 461
2018-02-02 01:15:24 ProcessMessages(headers, 82 bytes) FAILED peer=461
2018-02-02 01:15:24 ThreadSocketHandler -- removing node: peer=461 addr=219.25.22.23:9999 nRefCount=1 fNetworkNode=1 fInbound=0 fMasternode=0
2018-02-02 01:15:24 disconnecting peer=461
2018-02-02 01:15:24 trying connection 219.25.22.23:9999 lastseen=0.1hrs

[rrguardo]

Main reason looks >>
ERROR: detected bad peer for initial headers sync, disconnecting

[rrguardo]

header is detected as invalid from all peers, will be reviewing the related code

[rrguardo]

Will try now, set one node without internet and put date before the problem(~block 1401 date) at the related computer.
If all work fine again, just will reset all other nodes to sync from the one fixed.
Additionally will try to never stop mining, previous steps are possible only because my fork network is not alive yet.

[codablock]

Hmm not sure which behavior you expected, but what you observe seems to be normal. The "ERROR: detected bad peer for initial headers sync, disconnecting" you see means that it connected to a peer which did not provide recent enough HEADERs to be considered as up-to-date, which is to be expected after some time if there is no mining peer in the network.

It should however heal itself when a peer starts to do mining again. Were you able to restart mining after the errors appeared? Or is it exactly the mining node that struggles now?

[rrguardo]

I am not able to continue mining, mining node get stuck connecting and disconnecting from all accessible peers.

I was able to start mining in one node after setup old date on node server.
Is possible that mining node have some error or problem, will be reviewing, thanks

[rrguardo]

My network is working again following this steps:

• setup mining node time to 5 hours before last block time
• after generate a new block set time +5 hours until reach current date time
• reset all main nodes to get synchronized from zero using only the updated miner node or updated nodes in the synchronization.
• setup some main nodes as miner nodes to avoid repeat the problem.

I imagine that should exist other more easy way to do the same thing, unknown from me.