Provide an official ARM docker image and/or provide documentation for building on ARM

[gizmochief7]

Would it be possible to get an official Docker image of Envoy on ARM or at least some documentation on how to build it?
My team and I have tried off and on to build Envoy on ARM using the Bazel build process without any luck. Its challenging to say the least. I just saw a PR #1795 by @costinm dated a couple weeks ago based on the issue #1781 that talked about building it on ARM successfully after a few fixes. This is great news but we aren't sure how to duplicate it. @costinm mentioned that he used a custom CMAKE and a pi-cross-compiler to make it work. Documentation on this would be very helpful. Also, if there exist an image I could use for testing, then that would be very helpful too. Thanks again for such an awesome product!

[vielmetti]

@gizmochief7 it's been a little while, have you made progress? It sounds like you're building for 32-bit ARMv7; my particular interest is in arm64 builds.

[gizmochief7]

@vielmetti It has been and I almost forgot about posting this. We did not have any further luck with this attempt and without any community help we decided to move past Envoy. We tried out another reverse proxy/side car project for micro services and it just worked out of the box on both 32-bit and 64-bit. So we haven't wasted any more time here. Sorry I couldn't be more help.

[costinm]

Sorry, didn't see this bug: my cmake files are here: https://github.com/costinm/istio-build (tested on pi and android), I'm using them with repo to get the deps ( start from https://github.com/costinm/istio-repo ) There is another PR needed to compile head ( another option with 64bit int ) - will send it when I get back from vacation. I've been running envoy on my home pi (for a personal server) for few weeks, it's pretty stable and light, but I haven't made any progress in automating the build, CircleCI doesn't seem to like my build.

…

On Mon, Nov 20, 2017 at 8:44 PM, gizmochief7 ***@***.***> wrote: @vielmetti <https://github.com/vielmetti> It has been and I almost forgot about posting this. We did not have any further luck with this attempt and without any community help we decided to move past Envoy. We tried out another reverse proxy/side car project for micro services and it just worked out of the box on both 32-bit and 64-bit. So we haven't wasted any more time here. Sorry I couldn't be more help. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1861 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAFI6jvr_r5AMUuCa4cAUnkmziR4N3x5ks5s4lUkgaJpZM4P5-S8> .

[rshriram]

fwiw, I tried to build bazel on my pi (yes, it took several hours), and then built envoy on pi for armv7l 64bit. While envoy started up and was able to talk to http services, it had difficulty talking to certain https services (esp twitter). There seemed to be some boringssl issue that prevented ssl connections from working. My hunch was that it didn't compile properly. At that point, I gave up, because I had spent 4 solid days fighting bazel on pi.

[costinm]

I think the cross-compilation option is faster and less painful - and easy to expand to other platforms. Boringssl and most of the dependencies have native support for cmake (or can be easily adapted) - one of the problems with building with 'bazel' on pi is that deps are not really built with bazel, so even if bazel can cross-compile c++ code, it would still not help us, most of the code comes from the libraries compiled with autoconf. Of course autoconf also supports cross - but then I also wanted to run it on android, and cmake is supported very nicely by the NDK. It requires some periodic tweaking - adding the new envoy files and fixing ocasional 32-bit regressions (so far only the flags package seems to be a problem - not clear why), but it's a pretty fast build. But I appreciate the pain and effort in straight-compiling and fighting bazel :-)

…

On Tue, Nov 21, 2017 at 6:35 AM, Shriram Rajagopalan < ***@***.***> wrote: fwiw, I tried to build bazel on my pi (yes, it took several hours), and then built envoy on pi for armv7l 64bit. While envoy started up and was able to talk to http services, it had difficulty talking to certain https services (esp twitter). There seemed to be some boringssl issue that prevented ssl connections from working. My hunch was that it didn't compile properly. At that point, I gave up, because I had spent 4 solid days fighting bazel on pi. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1861 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAFI6mUOLKwygzX6Xwx9LXYCxrEc7_oEks5s4t-fgaJpZM4P5-S8> .

[costinm]

https://github.com/costinm/istio-build/releases if you want to try some (older) binaries, matching ~ Istio 0.2

…

On Tue, Nov 21, 2017 at 9:23 AM, Costin Manolache ***@***.***> wrote: I think the cross-compilation option is faster and less painful - and easy to expand to other platforms. Boringssl and most of the dependencies have native support for cmake (or can be easily adapted) - one of the problems with building with 'bazel' on pi is that deps are not really built with bazel, so even if bazel can cross-compile c++ code, it would still not help us, most of the code comes from the libraries compiled with autoconf. Of course autoconf also supports cross - but then I also wanted to run it on android, and cmake is supported very nicely by the NDK. It requires some periodic tweaking - adding the new envoy files and fixing ocasional 32-bit regressions (so far only the flags package seems to be a problem - not clear why), but it's a pretty fast build. But I appreciate the pain and effort in straight-compiling and fighting bazel :-) On Tue, Nov 21, 2017 at 6:35 AM, Shriram Rajagopalan < ***@***.***> wrote: > fwiw, I tried to build bazel on my pi (yes, it took several hours), and > then built envoy on pi for armv7l 64bit. While envoy started up and was > able to talk to http services, it had difficulty talking to certain https > services (esp twitter). There seemed to be some boringssl issue that > prevented ssl connections from working. My hunch was that it didn't compile > properly. At that point, I gave up, because I had spent 4 solid days > fighting bazel on pi. > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <#1861 (comment)>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/AAFI6mUOLKwygzX6Xwx9LXYCxrEc7_oEks5s4t-fgaJpZM4P5-S8> > . >

[moderation]

I've secured a Pine64 Rock64 SOC which is ARM8 / aarch64 and has 4GB of RAM. I've successfully
compiled Bazel 0.13.0. I'm not having any luck compiling Envoy.

I've leveraged @clnperez way of removing the Lua JIT.

I can't get her trick to use the locally installed Go to work go_register_toolchains(go_version="host"). When I use this Bazel uses a symbolic link to the installed Go tools (~/Library/go/pkg/tool/linux_arm64) and tries to compile Gazelle and protoc-gen-go and fails because link can't see gcc on $PATH. Installing these tools on the host don't fix the issue. I expect this is something to do with breaking the hermetic seal by using go_version="host".

I've modified source/extensions/extensions_build_config.bzl to only include a minimum number of extensions.

I'm using bazel --bazelrc=/dev/null build -s -c opt //source/exe:envoy-static.stripped --define google_grpc=disabled --define signal_trace=disabled --define hot_restart=disabled --verbose_failures as the build command with the following WORKSPACE.

workspace(name = "envoy")

load("//bazel:repositories.bzl", "envoy_dependencies")
load("//bazel:cc_configure.bzl", "cc_configure")

envoy_dependencies(
   skip_targets=['luajit']
)
cc_configure()

load("@envoy_api//bazel:repositories.bzl", "api_dependencies")
api_dependencies()

load("@io_bazel_rules_go//go:def.bzl", "go_download_sdk", "go_rules_dependencies", "go_register_toolchains")
load("@com_lyft_protoc_gen_validate//bazel:go_proto_library.bzl", "go_proto_repositories")

go_proto_repositories(shared=0)
go_rules_dependencies()

go_download_sdk(name = "go_sdk", sdks = {"linux_arm64":  ("go1.10.2.linux-arm64.tar.gz", "d6af66c71b12d63c754d5bf49c3007dc1c9821eb1a945118bfd5a539a327c4c8"), }, )

go_register_toolchains()

This prevents the build automatically downloading Go for linux_amd64.

The build then fails with

ERROR: ~/.cache/bazel/_bazel_moderation/1d277edd85b03195d275adc4870a9bef/external/com_lyft_protoc_gen_validate/validate/BUILD:35:1: no such package '@com_github_golang_protobuf//ptypes/duration': no such package '@io_bazel_rules_go_repository_tools//': no such package '@go_sdk//': Unsupported host linux_amd64 and referenced by '@com_lyft_protoc_gen_validate//validate:go_default_library'
ERROR: Analysis of target '//source/exe:envoy-static.stripped' failed; build aborted: no such package '@com_github_golang_protobuf//ptypes/duration': no such package '@io_bazel_rules_go_repository_tools//': no such package '@go_sdk//': Unsupported host linux_amd64

My suspicion is that the lyft/protoc-gen-validate is hard coded for linux_amd64 (or considering compilation works on MacOS / Darwin maybe it just doesn't support ARM).

Any help from Bazel wizards like @jmillikin-stripe or from people that have tried non amd64 compilation like @clnperez @costinm @rshriram would be great.

[vielmetti]

Issue noted in https://github.com/lyft/protoc-gen-validate/issues/75 - there are definitely some hard-coded dependencies in lyft/protoc-gen-validate to amd64 binaries.