EthereumJ fails on import

[holiman]

Hive sometimes wants to deliver blocks non-contiguously, which EthereumJ doesn't seem to like:

08:03:14.135 INFO [general]  Config ( no  ): config passed via constructor
Performing action: importBlocks
08:03:14.212 INFO [general]  Overriding config file with CLI options: {peer.discovery.enabled=false}
08:03:14.222 INFO [general]  Starting EthereumJ...
08:03:16.005 INFO [general]  Running frontier.json,  core version: 1.9.0-SNAPSHOT
08:03:16.005 INFO [general]  git.hash: [433ae2f]
08:03:16.005 INFO [general]  build.time: [GMT]-[10/Aug/2018]-[09:05]
08:03:16.006 INFO [general]  Created database version file
08:03:16.007 INFO [general]  capability eth version: [62, 63]
08:03:16.007 INFO [general]  capability shh version: [3]
08:03:16.007 INFO [general]  capability bzz version: [0]
08:03:16.667 ERROR [general]  Uncaught exception
org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'defaultConfig': Unsatisfied dependency expressed through field 'config'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'systemProperties' defined in org.ethereum.config.CommonConfig: Initialization of bean failed; nested exception is java.lang.RuntimeException: Block numbers should increase
	at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:569)
	at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:88)
	at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:349)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1219)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:543)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482)
	at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:751)
	at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:861)
	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:541)
	at org.springframework.context.annotation.AnnotationConfigApplicationContext.<init>(AnnotationConfigApplicationContext.java:84)
	at org.ethereum.facade.EthereumFactory.createEthereum(EthereumFactory.java:64)
	at org.ethereum.facade.EthereumFactory.createEthereum(EthereumFactory.java:45)
	at org.ethereum.facade.EthereumFactory.createEthereum(EthereumFactory.java:41)
	at org.ethereum.Start.main(Start.java:53)
	at com.ethercamp.harmony.Application.main(Application.java:58)
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'systemProperties' defined in org.ethereum.config.CommonConfig: Initialization of bean failed; nested exception is java.lang.RuntimeException: Block numbers should increase
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:553)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482)
	at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
	at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:207)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1128)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1056)
	at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:566)
	... 18 common frames omitted
Caused by: java.lang.RuntimeException: Block numbers should increase
	at org.ethereum.config.net.BaseNetConfig.add(BaseNetConfig.java:37)
	at org.ethereum.config.net.JsonNetConfig.<init>(JsonNetConfig.java:114)
	at org.ethereum.config.SystemProperties.getBlockchainConfig(SystemProperties.java:330)
	at org.ethereum.config.Initializer.initConfig(Initializer.java:66)
	at org.ethereum.config.Initializer.postProcessBeforeInitialization(Initializer.java:82)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:408)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1575)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:545)

[holiman]

For some reason, this hangs the client totally, thus halting Hive, which waits for the process to end. Not sure why that happens.

[holiman]

ping @mkalinin

[holiman]

When I run it locally, however, I do not hit the issue that it hangs -- it fails cleanly (as it should). Need to keep investigating what causes it to hang on hive

[holiman]

Ah, it may be these ones that hang:

22:05:44.361 INFO [ethash]  Kept caches: cnt: 1 epochs: 0...0
22:05:44.512 INFO [general]  Bind address wasn't set, Punching to identify it...
22:05:44.565 INFO [general]  UDP local bound to: 172.18.0.5
22:05:44.781 INFO [sync]  Sync Manager: OFF
Loading blocks: /blocks/0001.rlp, format: rlp
22:05:54.619 INFO [net]  TCP: Speed in/out 0b / 0b(sec), packets in/out 0/0, total in/out: 0b / 0b
22:05:54.619 INFO [net]  UDP: Speed in/out 0b / 0b(sec), packets in/out 0/0, total in/out: 0b / 0b
22:05:54.766 INFO [sync]  Sync state: Off, last block #0, best known #0
22:06:02.433 ERROR [VM]  
 !!! StackOverflowError: update your java run command with -Xss2M (-Xss4M for tests) !!!

java.lang.StackOverflowError: null
	at org.ethereum.util.ALock.lock(ALock.java:39)
	at org.ethereum.datasource.WriteCache.get(WriteCache.java:202)
	at org.ethereum.datasource.WriteCache.get(WriteCache.java:205)
	at org.ethereum.datasource.WriteCache.get(WriteCache.java:205)
	at org.ethereum.datasource.WriteCache.get(WriteCache.java:205)
	at org.ethereum.datasource.WriteCache.get(WriteCache.java:205)
	at org.ethereum.datasource.WriteCache.get(WriteCache.java:205)
	at org.ethereum.datasource.WriteCache.get(WriteCache.java:205)
	at org.ethereum.datasource.WriteCache.get(WriteCache.java:205)
	at org.ethereum.datasource.WriteCache.get(WriteCache.java:205)
	at org.ethereum.datasource.WriteCache.get(WriteCache.java:205)
	at org.ethereum.datasource.WriteCache.get(WriteCache.java:205)

[holiman]

Even weirder, when I force a full rebuild of my local hive client, the build fails:

Download
https://jcenter.bintray.com/org/apache/tomcat/tomcat-juli/8.5.5/tomcat-juli-8.5.5.jar
Download
https://jcenter.bintray.com/org/apache/httpcomponents/httpcore-nio/4.4.5/httpcore-nio-4.4.5.jar
Download
https://jcenter.bintray.com/xml-apis/xml-apis/1.4.01/xml-apis-1.4.01.jar
Download
https://jcenter.bintray.com/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.jar
/ethereum-harmony/src/main/java/com/ethercamp/harmony/jsonrpc/EthJsonRpcImpl.java:474:
error: DataWord(byte[]) has private access in DataWord
                getStorageValue(addressAsByteArray, new
DataWord(hexToByteArray(storageIdx)));
                                                    ^
/ethereum-harmony/src/main/java/com/ethercamp/harmony/jsonrpc/EthJsonRpcImpl.java:1045:
error: DataWord(byte[]) has private access in DataWord
                    logFilter.withTopic(new
DataWord(hexToByteArray((String) topic)).getData());
                                        ^
/ethereum-harmony/src/main/java/com/ethercamp/harmony/jsonrpc/EthJsonRpcImpl.java:1049:
error: DataWord(byte[]) has private access in DataWord
                        t.add(new DataWord(hexToByteArray(s)).getData());
                              ^
Note:
/ethereum-harmony/src/main/java/com/ethercamp/harmony/service/WalletService.java
uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
3 errors
:compileJava FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':compileJava'.
> Compilation failed; see the compiler error output for details.

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or
--debug option to get more log output. Run with --scan to get full insights.

[mkalinin]

hm, lots of different errors cause by recent updates in develop branch. Thanks, Martin, let us work on that.

[holiman]

So, when it hangs, this is the last it prints:

	at org.ethereum.vm.VM.step(VM.java:1252)
	at org.ethereum.vm.VM.play(VM.java:1323)
	at org.ethereum.vm.program.Program.callToAddress(Program.java:607)
	at org.ethereum.vm.VM.step(VM.java:1252)
	at org.ethereum.vm.VM.play(VM.java:1323)
12:34:16.215 INFO [facade]  Shutting down Ethereum instance...
12:34:16.215 INFO [general]  close: stopping peer discovery ...
12:34:16.216 INFO [discover]  Closing UDPListener...
12:34:16.216 INFO [discover]  Closing PeerConnectionTester...
 * Done * 

[holiman]

(that is during import)..
One suspiscion I have, is ....

• EtheruemJ opens RPC ports during import
• Thus Hive believes the client to be up, while it's in fact still at import stage
• And then EthereumJ fails, and the script exits

But I'm not sure why hive doesn't return control to the simulator (which would detect the failure to respond) or kill the container after the shell-script exits. @karalabe any ideas ?

[holiman]

I'm going to have to disable EtherumJ, because it blocks all others. Please ping me when it's time to enable it again!

[mkalinin]

ok, sure!

[eugene-shevchenko]

@holiman could you share any use case when ethereumj hangs? And what branch do you use?

[holiman]

Sure, if you

1. go to http://hivetests.ethstats.net/ ,
2. then filter for ethereumj,
3. and click load on the most recent one,
4. then order by Exec time
5. you'll see a lot that took over 500K seconds (before I killed them)

All of those, for example JUMPDEST_Attack_d0g0v0_Homestead