Document/consistent handling of macro and escape (unescape?)

[yuyichao]

From the document, it is probably clear enough why are macros and esc useful. However, IMHO, it is not quite clear when it is necessary to use esc. It's true that most of the cases it's quite intuitive but I could not find anywhere a short summary of the rules that cover all cases.

There also doesn't seem to be a way to "unescape".

The script that shows my confusion (and why I think "unescape" is necessary) is here,

module A
macro test_mac()
    quote
        function f(v::T1)
            v::T1
        end
        function Module.func1(v::T1)
            v::T1
        end
        @inline function Module.func2(v::T1)
            v::T1
        end
        function f{T2}(v::T2)
            v::T2
        end
        function Module.func1{T2}(v::T2)
            v::T2
        end
        @inline function Module.func2{T2}(v::T2)
            v::T2
        end
    end
end
end

println(macroexpand(:(A.@test_mac)))

The output is

begin  # /home/yuyichao/tmp/macro_escape.jl, line 6:
    function #1#f(#2#v::A.T1) # /home/yuyichao/tmp/macro_escape.jl, line 7:
        #2#v::A.T1
    end # line 9:
    function A.Module.func1(#3#v::A.T1) # /home/yuyichao/tmp/macro_escape.jl, line 10:
        #3#v::A.T1
    end # line 12:
    function A.Module.func2(A.v::A.T1)
        $(Expr(:meta, :inline)) # /home/yuyichao/tmp/macro_escape.jl, line 13:
        A.v::A.T1
    end # line 15:
    function #1#f{#5#T2}(#4#v::#5#T2) # /home/yuyichao/tmp/macro_escape.jl, line 16:
        #4#v::#5#T2
    end # line 18:
    function A.Module.func1{#7#T2}(#6#v::#7#T2) # /home/yuyichao/tmp/macro_escape.jl, line 19:
        #6#v::#7#T2
    end # line 21:
    function A.Module.func2{A.T2}(A.v::A.T2)
        $(Expr(:meta, :inline)) # /home/yuyichao/tmp/macro_escape.jl, line 22:
        A.v::A.T2
    end
end

It would be nice to document why are f, v, T2 mangled but not T1 and Module.func1. Also, is there a way to make the inline function definition works as expected (unescape?)?

[vtjnash]

related: #6910

[toivoh]

As it says in the manual under hygiene in the metaprogramming section, symbols found in the AST returned from a macro are classified into two categories: local and global. Local ones are those that are assigned in some way in the AST, and are replaced by gensyms. Others are considered global and are (set to be) looked up in the module that defines the macro. This explains the difference that you see.

[toivoh]

The case with @inline seems like a hygiene bug, or more probably a shortcoming of the current hygiene system. Since Hygiene has to run before @inline is expanded, it cannot see how the symbols inside it should be classified. Here #6910 should hopefully help.

The manual way to unescape is to create gensyms yourself and interpolate into the AST.

[toivoh]

Forgot to say, Module.func1 can't be turned into a gensym because it's not a symbol. Module itself is not assigned so it is regarded as global, and func1 is not a freestanding identifier but the argument of the dot operator (it's always looked up in the left argument and not in the local scope), so it's not eligible for hygiene at all.

[yuyichao]

I guess I understand the difference now after testing for a few more examples.

The "manual way" to unescape should work when I want to avoid name clashing. However, what if what I want to do is to extend a function in current scope. E.g.

julia> module A
       macro B()
       expr = quote
       function C()
       end
       end
       return esc(expr)
       end
       end

julia> macroexpand(:(@A.B))
quote  # none, line 4:
    function C()
    end
end

Should I replace C by A.C and assume A is always available in the module the macro is called? (Is this a grantee)

[toivoh]

I'm not sure that I understand. Do you always want to extend function C in the module A above, or do you want to extend the function C in the scope of the caller of @B? As you have written it, it seems like it will do the latter.

[yuyichao]

I actually would like to do the former.

The esc is meant to come from some other necessary logic that require escape and I use C instead of A.C because I'm not sure if A.C is the correct way to do this either.

[toivoh]

And if you change C to A.C I think it should do the former.

[yuyichao]

Does it mean that A is always available in the scope @B might be called?

[toivoh]

Btw the other way to manually escape something is to interpolate the quoted value of whatever you want to put there. This is how the hygenization works for symbols deemed to be global.

[toivoh]

All that said, maybe it could be explained better in the manual. Now that you understand it a bit better, do you have an idea of what could be changed?

[yuyichao]

I hope you are not asking for a PR to the document since I'm certainly not qualified for that =P.

IMHO, here's something that could have helped me to understand this better.

1. It was not so clear to me why defining a function A is local but "defining" a function B.C is global, especially since defining another macro in the macro returned value seems to follow yet a different rule. I cannot really tell why now because as long as I understand that, it is indeed hard to imagine how I could get it wrong before....

2. A few other examples for local vs global might be helpful too.

3. The document mentioned "global variables are resolved within the macro definition environment" but didn't mention how. (Now I remember that when I checked the expanded AST last time, I saw the module object itself is inserted in the AST.) This might not be very useful for writing a macro but can make it less mysterious, especially since the macro itself is mysterious enough for anyone that doesn't have a lisp background....

   The printing from macroexpand here is also confusing since it cannot show the difference between a symbol and a module. (Maybe color code non-symbol/symbol when printing a AST?)

4. Maybe mention how to deal with nested macros (@inline, or whatever....)

5. An unrelated stuff. When I was working on inline invoke #9642, I saw that inserting values (functions?) into AST can cause compile error. However, this seems to be how the macro system works (inserting modules) to resolve values in the correct scope.

   This probably doesn't matter for current modules since they are compiles for a single session only but would this be OK to do (either automatically by the macro system or manually in order to "unescape" sth) when static module compilation is added? (I thought it was one of the 0.4 milestone a few months ago but couldn't find it anymore.) What about inserting other kind of values into AST?

[toivoh]

Sorry, just realized that the A.C thing only works if you don't escape that part of the AST. A can not be expected to be available by name in any caller of @B, but hygenization would take A and look it up as a global.

In general, you should be able to interpolate any kind of value into the AST, provided that you quote by putting it in a QuoteNode, but there may still be a few syntactic positions where a quoted value is not handled correctly.

I don't know enough about static compilation to say how this would interact with it, @vtjnash?

You can see the difference between symbols and interpolated values if you print the AST using other methods, eg Base.Meta.show_sexpr.

I'm not asking for a PR unless you want to make one :) just wanted to bring attention back to the topic of improving the docs.