feat: metrics including pprof and expvars

[yichya]

A more straightforward (and hopefully better) way to export stats.

Enabling metrics

Quite similar to how to enable Xray APIs

1. Add a top-level configuration called metrics with tag (outbound tag) as its only configuration

{
    "metrics": {
        "tag": "metrics_out"
    }
}

2. Add a dokodemo door inbound. For example here we use 127.0.0.1:11111

{
    "port": 11111,
    "protocol": "dokodemo-door",
    "tag": "metrics_in",
    "settings": {
        "address": "127.0.0.1"
    },
    "listen": "127.0.0.1"
}

3. Use routing to connect them together

{
    "type": "field",
    "inboundTag": [
        "metrics_in"
    ],
    "outboundTag": "metrics_out"
}

Usage

pprof

Access http://127.0.0.1:11111/debug/pprof/ or use go tool pprof to start profiling or inspect running goroutines.

expvars

Access http://127.0.0.1:11111/debug/vars

Variables exported include:

• stats includes statistics about inbounds, outbounds and users
• observatory includes observatory results

for example with luci-app-xray you are likely to get a result like this (standard expvar things like cmdline and memstats are omitted)

{
    "observatory": {
        "tcp_outbound": {
            "alive": true,
            "delay": 782,
            "outbound_tag": "tcp_outbound",
            "last_seen_time": 1648477189,
            "last_try_time": 1648477189
        },
        "udp_outbound": {
            "alive": true,
            "delay": 779,
            "outbound_tag": "udp_outbound",
            "last_seen_time": 1648477191,
            "last_try_time": 1648477191
        }
    },
    "stats": {
        "inbound": {
            "api": {
                "downlink": 0,
                "uplink": 0
            },
            "dns_server_inbound_5300": {
                "downlink": 14286,
                "uplink": 5857
            },
            "http_inbound": {
                "downlink": 74460,
                "uplink": 10231
            },
            "https_inbound": {
                "downlink": 0,
                "uplink": 0
            },
            "metrics": {
                "downlink": 6327,
                "uplink": 1347
            },
            "socks_inbound": {
                "downlink": 19925615,
                "uplink": 5512
            },
            "tproxy_tcp_inbound": {
                "downlink": 4739161,
                "uplink": 1568869
            },
            "tproxy_udp_inbound": {
                "downlink": 0,
                "uplink": 2608142
            }
        },
        "outbound": {
            "blackhole_outbound": {
                "downlink": 0,
                "uplink": 0
            },
            "direct": {
                "downlink": 97714548,
                "uplink": 3234617
            },
            "dns_server_outbound": {
                "downlink": 7116,
                "uplink": 2229
            },
            "manual_tproxy_outbound_tcp_1": {
                "downlink": 0,
                "uplink": 0
            },
            "manual_tproxy_outbound_udp_1": {
                "downlink": 0,
                "uplink": 0
            },
            "tcp_outbound": {
                "downlink": 23873238,
                "uplink": 1049595
            },
            "udp_outbound": {
                "downlink": 639282,
                "uplink": 74634
            }
        },
        "user": {}
    }
}

To get a better view of these numbers, Netdata (with python.d plugin) is a great option:

1. Edit related configuration file (sudo /etc/netdata/edit-config python.d/go_expvar.conf)
2. Take the following configuration file as an example:

xray:
  name: 'xray'
  update_every: 2
  url: 'http://127.0.0.1:11111/debug/vars'
  collect_memstats: false
  extra_charts:
     - id: 'inbounds'
       options:
         name: 'inbounds'
         title: 'Xray System Inbounds'
         units: bytes
         family: xray
         context: xray.inbounds
         chart_type: line
       lines:
         - expvar_key: stats.inbound.tproxy_tcp_inbound.downlink
           id: 'tcp.downlink'
           algorithm: incremental
           expvar_type: int
         - expvar_key: stats.inbound.tproxy_udp_inbound.downlink
           id: 'udp.downlink'
           algorithm: incremental
           expvar_type: int
         - expvar_key: stats.inbound.http_inbound.downlink
           id: 'http.downlink'
           algorithm: incremental
           expvar_type: int
         - expvar_key: stats.inbound.https_inbound.downlink
           id: 'https.downlink'
           algorithm: incremental
           expvar_type: int
         - expvar_key: stats.inbound.socks_inbound.downlink
           id: 'socks.downlink'
           algorithm: incremental
           expvar_type: int
         - expvar_key: stats.inbound.tproxy_tcp_inbound.uplink
           id: 'tcp.uplink'
           algorithm: incremental
           expvar_type: int
         - expvar_key: stats.inbound.tproxy_udp_inbound.uplink
           id: 'udp.uplink'
           algorithm: incremental
           expvar_type: int
         - expvar_key: stats.inbound.http_inbound.uplink
           id: 'http.uplink'
           algorithm: incremental
           expvar_type: int
         - expvar_key: stats.inbound.https_inbound.uplink
           id: 'https.uplink'
           algorithm: incremental
           expvar_type: int
         - expvar_key: stats.inbound.socks_inbound.uplink
           id: 'socks.uplink'
           algorithm: incremental
           expvar_type: int
     - id: 'outbounds'
       options:
         name: 'outbounds'
         title: 'Xray System Outbounds'
         units: bytes
         family: xray
         context: xray.outbounds
         chart_type: line
       lines:
         - expvar_key: stats.outbound.tcp_outbound.downlink
           id: 'tcp.downlink'
           algorithm: incremental
           expvar_type: int
         - expvar_key: stats.outbound.udp_outbound.downlink
           id: 'udp.downlink'
           algorithm: incremental
           expvar_type: int
         - expvar_key: stats.outbound.direct.downlink
           id: 'direct.downlink'
           algorithm: incremental
           expvar_type: int
         - expvar_key: stats.outbound.tcp_outbound.uplink
           id: 'tcp.uplink'
           algorithm: incremental
           expvar_type: int
         - expvar_key: stats.outbound.udp_outbound.uplink
           id: 'udp.uplink'
           algorithm: incremental
           expvar_type: int
         - expvar_key: stats.outbound.direct.uplink
           id: 'direct.uplink'
           algorithm: incremental
           expvar_type: int
     - id: 'observatory'
       options:
         name: 'observatory'
         title: 'Xray Observatory Metrics'
         units: milliseconds
         family: xray
         context: xray.observatory
         chart_type: line
       lines:
         - expvar_key: observatory.tcp_outbound.delay
           id: tcp
           expvar_type: int
         - expvar_key: observatory.udp_outbound.delay
           id: udp
           expvar_type: int

And you will get a nice plot like this:

Additional

Maybe reusing the empty object stats in config file is better than adding metrics here?

Edit: removed prometheus related things and added usage about expvars

[yuhan6665]

Thanks for your great effort! It seems to me the code is well written.
However, as you can see, *ray core has been very cautious on adding more external libraries. Our project is sensitive and we need to be careful not to bring weakness from other sources. More importantly, note that core is used on both client and server sides, used in servers, desktops, phones and routers. The question we need to answer is: are the dependency belong to the "core" feature? Is the feature useful to all platforms?
Maybe, this feature is more suitable for https://github.com/XrayR-project/XrayR or https://github.com/vaxilu/x-ui

I'm curious to hear what other project participants think about this.

[yichya]

@yuhan6665

Thank you for your reply.

However, as you can see, *ray core has been very cautious on adding more external libraries. Our project is sensitive and we need to be careful not to bring weakness from other sources.

Prometheus client has been widely used in many projects, including similar anti-censorship tools like hysteria. It does not do more than formatting some numbers to a specific format so I don't assume that would be too vulnerable.

If you are really concerned about involving external dependencies I could add only expvars and pprof and expose these metrics using expvars. They are included in go standard library and no external dependency is needed. Prometheus format could be then converted from expvars via a external formatter.

More importantly, note that core is used on both client and server sides, used in servers, desktops, phones and routers. The question we need to answer is: are the dependency belong to the "core" feature? Is the feature useful to all platforms?

Personally I think observability as a feature is

• useful to normal users willing to know whether things are working as expected (for example to see bandwidth usage of their outbounds)
• important to power users and mass service providers (to know more about resource consumption and solve performance issues)

Even if you don't need that feature, you could just turn it off and it won't do any harm at all.

[yichya]

I've removed prometheus related code from my commit. That part could be discussed later as another merge request.

At the same time I've added code to expose stats and observatory data via expvars.

[yuhan6665]

Thanks for the update! I will do some test on my side tomorrow and update you if I see anything need change.

[yichya]

Added some more detailed description about expvars and its usage.

[yuhan6665]

Looks great to me! Thanks for your work and thorough documentation!

[xyz3282836]

step 1 2 3 done
but observatory and stats is null
@yichya