macros that reference Java libraries don't work

[cushon]

The following example shows that a macro that calls a Java library doesn't work. This is related to bazelbuild/bazel#632. The library jar for :JavaLib on compile-time classpath only contains the compile-time API of that library, but the macro needs the complete implementation to be available at compile-time.

Demo: https://gist.github.com/cushon/a97470ed69f61d966c94bcde1ed018f0

Setup:

$ wget https://gist.github.com/cushon/a97470ed69f61d966c94bcde1ed018f0/archive/31e29b918d9e77cd84ec8bb3cab4cf584e5b2082.zip
$ unzip 31e29b918d9e77cd84ec8bb3cab4cf584e5b2082.zip
$ cd 31e29b918d9e77cd84ec8bb3cab4cf584e5b2082

Building :HelloLib fails:

bazel build :HelloLib
INFO: Analysed target //:HelloLib (23 packages loaded).
INFO: Found 1 target...
INFO: From scala //:MacroTest:
warning: there was one deprecation warning; re-run with -deprecation for details
one warning found
ERROR: /tmp/tmp.UHlgyDbGs0/a97470ed69f61d966c94bcde1ed018f0-31e29b918d9e77cd84ec8bb3cab4cf584e5b2082/BUILD:11:1: scala //:HelloLib failed (Exit 1)
HelloLib.scala:5: error: exception during macro expansion:
java.lang.ClassFormatError: Absent Code attribute in method that is not native or abstract in class file com/test/JavaLib
	at java.lang.ClassLoader.defineClass1(Native Method)
	at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
	at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
	at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at scala.test.MacroTest$.hello_impl(MacroTest.scala:14)

    MacroTest.hello("hi")
                   ^
one error found
one error found
java.lang.RuntimeException: Build failed
	at io.bazel.rulesscala.scalac.ScalacProcessor.compileScalaSources(ScalacProcessor.java:263)
	at io.bazel.rulesscala.scalac.ScalacProcessor.processRequest(ScalacProcessor.java:68)
	at io.bazel.rulesscala.worker.GenericWorker.run(GenericWorker.java:125)
	at io.bazel.rulesscala.scalac.ScalaCInvoker.main(ScalaCInvoker.java:41)

Disabling ijar and header compilation works around the bug:

$ bazel build --nouse_ijars --nojava_header_compilation  :HelloLib
... OK

[johnynek]

Yep. That’s an issue. I think bazel May be able to handle this by possibly hitching to its handling of annotation processors, but maybe there are some speed bumps there.

In the worst case we can write some rule that re-exports the runtime path as a compile time for use in macros.

[johnynek]

As described in the linked bazel issue above this can be improved even more with a scalainfo provider. We can’t totally migrate to new style providers yet because intellij does not support JavaInfo fully yet as far as I know.

I’d rather wait to solve this until we can do it in a migration to only new style providers.

[johnynek]

Note: the fix of repairing the classpath issue can be done now, at the cost of java dependencies seeing the full classpath. We could do that and then narrow the classpath when we move to only using providers.

[sdtwigg]

Yeah, this is a tad unfortunate. I ran into a variant of this issue when trying to get scalac plugins to work on a really old variant of the rules. My main regret is that the macro implementations should generally only be using a small subset of the total dependencies (as they are really only fiddling with ASTs) and so exposing all dependencies is a bit pessimistic. I suppose people could mitigate the impact of this by only having their scala_macro_library rules contain the macro implementations. (I can test this myself later but I don't think the macro invocations like def blah = macro MacroImpl need to be a scala_macro_library?)

…

On Sat, Apr 21, 2018, 4:48 PM P. Oscar Boykin ***@***.***> wrote: Note: the fix of repairing the classpath issue can be done now, at the cost of java dependencies seeing the full classpath. We could do that and then narrow the classpath when we move to only using providers. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#445 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA9_-FNjAoiMWfha0Kt_S20DADROK9HGks5tq8VMgaJpZM4Sw10C> .

[cushon]

Note: the fix of repairing the classpath issue can be done now, at the cost of java dependencies seeing the full classpath.

Is the idea here to have scala_macro_library add its runtime dependencies to the compile-time classpath for downstream targets? I don't think that solves the problem, because there's a dependency ordering issue.

If a some library depends on, say, apache commons directly and via a scala_macro_library, the scala_macro_library can't ensure that the runtime jar appears before the ijar on the compile-time classpath of the downstream library. ScalaInfo.macro_classpath addresses that by keeping the macro's dependencies separate, and making sure they appears before non-macro deps on the downstream library's compilation classpath.

[abhandaru]

Also having this issue when trying to use with TableQueryMacroImpl in Slick.

Artifact in Maven2: com.typesafe.slick:slick_2.12:3.2.3

The workaround suggested by the original poster does work for me, just the --nouse_ijars was sufficient. I am very new to Bazel so not quite sure what all the negative repercussions are for using this hack.

Hoping to see an update!

[ittaiz]

@cushon @johnynek I've read the thread but got lost a bit.
Is there something we can do other than --nouse_ijar? I can see if we can prioritize some time for this if I have an idea on what needs to be done

[johnynek]

Yes, we can do something better.

I think the right solution is to rexport the full transitive runtime classpath as the compile classpath for scala_macro_libraries and to educate users that scala jars need to be imported with scala_import.

https://github.com/johnynek/bazel-deps

does this, for instance.

One problem with bazel is that there are many ways you could consume external maven dependencies. It is hard to give someone feedback when they just describe something not working without knowing how they are depending on the code.

We should add a local test showing two issues:

1. depending on local java code from a scala_macro_library and the macro calling that code. This does not work currently.
2. an example using scala_import to depend on a macro. This works currently I think, but I don't think we actually have a test.

[ioleo]

It seems I'm having exacly this issue when depending on com.sksamuel.avro4s:avro4s-macros_2.11:1.7.0. The --nouse_ijar workaround works, but I would like to disable ijars only for this artifact (not the whole build).

I've tried using scala_import and filegroup as suggested here, but it seems I'm still missing something, I can't get it working. Is there anyone who can share a working example disabling ijar generation for just one artifact?

[johnynek]

@ioleo can you post the stack trace you get and the scala_import you used?

[ioleo]

@johnynek So, I've reverted to my original code already:

load("@io_bazel_rules_scala//scala:scala.bzl", "scala_library", "scala_test")
load("//bzl-includes:iqvia_scala_opts.bzl", "SCALAC_JVM_FLAGS", "SCALAC_OPTS", "SCALAC_PLUGINS")

_3RDPARTY_DEPS = [
    "//3rdparty/jvm/ca/mrvisser:sealerate",
    "//3rdparty/jvm/com/chuusai:shapeless",
    "//3rdparty/jvm/com/fasterxml/jackson/core:jackson_core",
    "//3rdparty/jvm/com/fasterxml/jackson/core:jackson_databind",
    "//3rdparty/jvm/com/sksamuel/avro4s:avro4s_core",
    "//3rdparty/jvm/com/sksamuel/avro4s:avro4s_macros",
    "//3rdparty/jvm/com/typesafe/play:play_functional",
    "//3rdparty/jvm/com/typesafe/play:play_json",
    "//3rdparty/jvm/joda_time:joda_time",
    "//3rdparty/jvm/org/apache/avro:avro",
]

_INTERNAL_DEPS = []

_ALL_DEPS = _3RDPARTY_DEPS + _INTERNAL_DEPS

scala_library(
    name = "acme_common_api_lib",
    srcs = glob(["src/main/scala/**/*.scala"]),
    resources = glob(["src/main/resources/**/*"]),
    scalacopts = SCALAC_OPTS,
    plugins = SCALAC_PLUGINS,
    scalac_jvm_flags = SCALAC_JVM_FLAGS,
    visibility = [
        "//visibility:public"
    ],
    deps = _ALL_DEPS,
)

which

• built with bazel build //acme/common/api:acme_common_api_lib --nouse_ijars builds successfully
• built with bazel build //acme/common/api:acme_common_api_lib throws:

Exception in thread "main" java.lang.ClassFormatError: Absent Code attribute in method that is not native or abstract in class file org/apache/avro/Schema
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
(...)

As you can see I'm importing from 3rdparty directory, which is generated by your bazel-deps tool (kudos for that! <3), the relevant part of jvm-dependencies.yml being:

dependencies:
# ... 
  com.sksamuel.avro4s:
    avro4s:
      lang: scala
      modules: [ "core", "macros" ]
      version: "1.7.0"
# ... 

---

As for the scala_import I've tried replacing the auto-generated scala import with my own:

load("@io_bazel_rules_scala//scala:scala_import.bzl", "scala_import")
// ...

_3RDPARTY_DEPS = [
    "//3rdparty/jvm/ca/mrvisser:sealerate",
    "//3rdparty/jvm/com/chuusai:shapeless",
    "//3rdparty/jvm/com/fasterxml/jackson/core:jackson_core",
    "//3rdparty/jvm/com/fasterxml/jackson/core:jackson_databind",
    "//3rdparty/jvm/com/sksamuel/avro4s:avro4s_core",
    "//3rdparty/jvm/com/typesafe/play:play_functional",
    "//3rdparty/jvm/com/typesafe/play:play_json",
    "//3rdparty/jvm/joda_time:joda_time",
    "//3rdparty/jvm/org/apache/avro:avro",
    ":avro4s_macros",
]

// ...

scala_import(
    name = "avro4s_macros",
    jars = [
        "//external:jar/com/sksamuel/avro4s/avro4s_macros_2_11"
    ],
    visibility = [
        "//visibility:public"
    ]
)

// ...

The only diffrence beuing replacing in _3RDPARTY_DEPS line:
"//3rdparty/jvm/com/sksamuel/avro4s:avro4s_macros",
with ":avro4s_macros", pointing to scala import below.

However this leads to the same build error.

About the filegroup suggested solution as far as I understand it's when I incorporate the jar in my own repo (which I don't want here --> I want to reference maven artifact).

Please forgive any misconceptions, I'm just getting started with bazel.

[johnynek]

The class that is missing is: org/apache/avro/Schema

What jar is that in? It probably isn’t in a scala jar and that’s the issue. Macros can call non macro code even java code. In such cases now we have an issue because we need to remove the ijar and add the runtime jar.

A work around if you are using my bazel-deps tool is to say Avro is a scala/unmangled for the language. This will use scala import for avro which makes it not use ijar.

You really don’t need ijar for external dependencies since they don’t change often and when they do even the ijar likely changes. We should probably update bazel deps to not use ijar even for java dependencies. That would reduce these issues but still not totally solve them since we still can have this kind of problem for macro code defined inside the repo.

[ioleo]

@johnynek It's used by the avro4s-macro library. The dependency is added via sbt plugin.

I agree I don't need the IJAR for the external depepency. As far as I understand, using --nouse_ijars option disables IJARS completely for this build (so, also for my own code ~> forcing to rebuild downstream dependencies even if the API has not changed).

A work around if you are using my bazel-deps tool is to say Avro is a scala/unmangled for the language.

How do I do that?

[johnynek]

If you add the avro jar to your dependencies and give it a lang: scala/unmangled it should do it. That means we don't apply the _2.11 or _2.12 suffix, but use scala_import. That should fix it.