AccessControlException when restoring backup from azure repository

[okke-formsma]

Elasticsearch version
Version: 6.2.1, Build: 7299dc3/2018-02-07T19:34:26.990113Z, JVM: 1.8.0_161

Plugins installed: [repository-azure]

JVM version (java -version):
java version "1.8.0_161"
Java(TM) SE Runtime Environment (build 1.8.0_161-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.161-b12, mixed mode)

OS version (uname -a if on a Unix-like system):
Linux elastic6-dev 4.13.0-32-generic #35-Ubuntu SMP Thu Jan 25 09:13:46 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Description of the problem including expected versus actual behavior:
I'm restoring a backup from elasticsearch 5 (or 6) to an elasticsearch 6 machine using the azure-repository plugin. All index definitions seem to be restored properly, but many shards fail (>50%). Some shards are successfully recovered.

This seems similar to a previous issue; #25931

Provide logs (if relevant):

[2018-02-13T14:04:54,847][WARN ][o.e.c.a.s.ShardStateAction] [ela6dev1] [xxx-2017-05][1] received shard failed for shard id [[xxx-2017-05][1]], allocation id [B_vVqxR5SGKfRoSLOZAW9Q], primary term [0], message [failed recovery], failure [RecoveryFailedException[[xxx-2017-05][1]: Recovery failed on {ela6dev1}{6LKL387lRjuE9mDdWWoanQ}{Ip9YutruRCyl7vjIv7M3WA}{10.2.30.4}{10.2.30.4:9300}]; nested: IndexShardRecoveryException[failed recovery]; nested: IndexShardRestoreFailedException[restore failed]; nested: IndexShardRestoreFailedException[failed to restore snapshot [2018-02-03/Pj8IMfHASyOCkl9rv9xO3Q]]; nested: RuntimeException[java.security.AccessControlException: access denied ("java.net.SocketPermission" "xxxelasticbackup.blob.core.windows.net:443" "connect,resolve")]; nested: AccessControlException[access denied ("java.net.SocketPermission" "xxxelasticbackup.blob.core.windows.net:443" "connect,resolve")]; ]
org.elasticsearch.indices.recovery.RecoveryFailedException: [xxx-2017-05][1]: Recovery failed on {ela6dev1}{6LKL387lRjuE9mDdWWoanQ}{Ip9YutruRCyl7vjIv7M3WA}{10.2.30.4}{10.2.30.4:9300}
        at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$7(IndexShard.java:2065) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:566) [elasticsearch-6.2.1.jar:6.2.1]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery
        at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:342) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.index.shard.StoreRecovery.recoverFromRepository(StoreRecovery.java:275) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.index.shard.IndexShard.restoreFromRepository(IndexShard.java:1613) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$7(IndexShard.java:2061) ~[elasticsearch-6.2.1.jar:6.2.1]
        ... 4 more
Caused by: org.elasticsearch.index.snapshots.IndexShardRestoreFailedException: restore failed
        at org.elasticsearch.index.shard.StoreRecovery.restore(StoreRecovery.java:463) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromRepository$5(StoreRecovery.java:277) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:300) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.index.shard.StoreRecovery.recoverFromRepository(StoreRecovery.java:275) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.index.shard.IndexShard.restoreFromRepository(IndexShard.java:1613) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$7(IndexShard.java:2061) ~[elasticsearch-6.2.1.jar:6.2.1]
        ... 4 more
Caused by: org.elasticsearch.index.snapshots.IndexShardRestoreFailedException: failed to restore snapshot [2018-02-03/Pj8IMfHASyOCkl9rv9xO3Q]
        at org.elasticsearch.repositories.blobstore.BlobStoreRepository.restoreShard(BlobStoreRepository.java:827) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.index.shard.StoreRecovery.restore(StoreRecovery.java:448) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromRepository$5(StoreRecovery.java:277) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:300) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.index.shard.StoreRecovery.recoverFromRepository(StoreRecovery.java:275) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.index.shard.IndexShard.restoreFromRepository(IndexShard.java:1613) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$7(IndexShard.java:2061) ~[elasticsearch-6.2.1.jar:6.2.1]
        ... 4 more
Caused by: java.lang.RuntimeException: java.security.AccessControlException: access denied ("java.net.SocketPermission" "xxxelasticbackup.blob.core.windows.net:443" "connect,resolve")
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1506) ~[?:?]
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492) ~[?:?]
        at sun.net.www.protocol.http.HttpURLConnection.getHeaderField(HttpURLConnection.java:3036) ~[?:?]
        at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:489) ~[?:1.8.0_161]
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:347) ~[?:?]
        at com.microsoft.azure.storage.StorageException.translateException(StorageException.java:76) ~[?:?]
        at com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:199) ~[?:?]
        at com.microsoft.azure.storage.blob.CloudBlob.downloadRangeInternal(CloudBlob.java:1492) ~[?:?]
        at com.microsoft.azure.storage.blob.BlobInputStream.dispatchRead(BlobInputStream.java:255) ~[?:?]
        at com.microsoft.azure.storage.blob.BlobInputStream.readInternal(BlobInputStream.java:448) ~[?:?]
        at com.microsoft.azure.storage.blob.BlobInputStream.read(BlobInputStream.java:420) ~[?:?]
        at org.elasticsearch.index.snapshots.blobstore.SlicedInputStream.read(SlicedInputStream.java:92) ~[elasticsearch-6.2.1.jar:6.2.1]
        at java.io.FilterInputStream.read(FilterInputStream.java:133) ~[?:1.8.0_161]
        at org.elasticsearch.index.snapshots.blobstore.RateLimitingInputStream.read(RateLimitingInputStream.java:69) ~[elasticsearch-6.2.1.jar:6.2.1]
        at java.io.FilterInputStream.read(FilterInputStream.java:107) ~[?:1.8.0_161]
        at org.elasticsearch.repositories.blobstore.BlobStoreRepository$RestoreContext.restoreFile(BlobStoreRepository.java:1572) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.repositories.blobstore.BlobStoreRepository$RestoreContext.restore(BlobStoreRepository.java:1515) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.repositories.blobstore.BlobStoreRepository.restoreShard(BlobStoreRepository.java:825) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.index.shard.StoreRecovery.restore(StoreRecovery.java:448) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromRepository$5(StoreRecovery.java:277) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:300) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.index.shard.StoreRecovery.recoverFromRepository(StoreRecovery.java:275) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.index.shard.IndexShard.restoreFromRepository(IndexShard.java:1613) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$7(IndexShard.java:2061) ~[elasticsearch-6.2.1.jar:6.2.1]
        ... 4 more
Caused by: java.security.AccessControlException: access denied ("java.net.SocketPermission" "xxxelasticbackup.blob.core.windows.net:443" "connect,resolve")
        at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472) ~[?:1.8.0_161]
        at java.security.AccessController.checkPermission(AccessController.java:884) ~[?:1.8.0_161]
        at java.lang.SecurityManager.checkPermission(SecurityManager.java:549) ~[?:1.8.0_161]
        at java.lang.SecurityManager.checkConnect(SecurityManager.java:1051) ~[?:1.8.0_161]
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:541) ~[?:?]
        at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:264) ~[?:?]
        at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367) ~[?:?]
        at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191) ~[?:?]
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1156) ~[?:?]
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1050) ~[?:?]
        at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177) ~[?:?]
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1564) ~[?:?]
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492) ~[?:?]
        at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480) ~[?:1.8.0_161]
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:347) ~[?:?]
        at com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:119) ~[?:?]
        at com.microsoft.azure.storage.blob.CloudBlob.downloadRangeInternal(CloudBlob.java:1492) ~[?:?]
        at com.microsoft.azure.storage.blob.BlobInputStream.dispatchRead(BlobInputStream.java:255) ~[?:?]
        at com.microsoft.azure.storage.blob.BlobInputStream.readInternal(BlobInputStream.java:448) ~[?:?]
        at com.microsoft.azure.storage.blob.BlobInputStream.read(BlobInputStream.java:420) ~[?:?]
        at org.elasticsearch.index.snapshots.blobstore.SlicedInputStream.read(SlicedInputStream.java:92) ~[elasticsearch-6.2.1.jar:6.2.1]
        at java.io.FilterInputStream.read(FilterInputStream.java:133) ~[?:1.8.0_161]
        at org.elasticsearch.index.snapshots.blobstore.RateLimitingInputStream.read(RateLimitingInputStream.java:69) ~[elasticsearch-6.2.1.jar:6.2.1]
        at java.io.FilterInputStream.read(FilterInputStream.java:107) ~[?:1.8.0_161]
        at org.elasticsearch.repositories.blobstore.BlobStoreRepository$RestoreContext.restoreFile(BlobStoreRepository.java:1572) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.repositories.blobstore.BlobStoreRepository$RestoreContext.restore(BlobStoreRepository.java:1515) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.repositories.blobstore.BlobStoreRepository.restoreShard(BlobStoreRepository.java:825) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.index.shard.StoreRecovery.restore(StoreRecovery.java:448) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromRepository$5(StoreRecovery.java:277) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:300) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.index.shard.StoreRecovery.recoverFromRepository(StoreRecovery.java:275) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.index.shard.IndexShard.restoreFromRepository(IndexShard.java:1613) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$7(IndexShard.java:2061) ~[elasticsearch-6.2.1.jar:6.2.1]
        ... 4 more
[2018-02-13T14:04:54,920][WARN ][o.e.c.m.MetaDataMappingService] [ela6dev1] [xxx-2017-03] re-syncing mappings with cluster state because of types [[xxx]]

[okke-formsma]

Today I tested the ability to restore backups made by an elastic6 cluster to an elastic6 cluster, and those fail with the same exception.

This is blocking our elastic6 migration. Does anyone have some insight in what is going wrong?

[Tim-Brooks]

@okke-formsma thank your for opening this issue. It identified a real issue with the azure repository plugin.

A fix has been merged and will be included in 7.0, 6.3, and 6.2.3.

[okke-formsma]

@tbrooks8 thank you for the fix. Any word on the release date of 6.2.3?

[jasontedor]

Any word on the release date of 6.2.3?

Sorry, we do not provide release dates.