Beta testing Gatekeeper

[dahnevskiy]

Hello!

Im trying to inject prefix to FIB, using lua/examples/example_of_dynamic_config_request.lua

As I can see, this example file is not working:

because its missing:
require "gatekeeper/dylib"
which we need to use to do dylib.c.add_fib_entry function for example.

I fixed this, so this is my simple final lua script:

require "gatekeeper/staticlib"
require "gatekeeper/dylib"

local dy_conf = staticlib.c.get_dy_conf()
        if dy_conf == nil then
                error("Failed to allocate dy_conf")
        end

local ret = dylib.c.add_fib_entry("192.168.0.0/16", "10.255.0.66", "10.255.0.225", dylib.c.GK_FWD_GRANTOR, dy_conf.gk)
if ret < 0 then
        return "gk: failed to add an FIB entry\n"
end

but its not working anyway, when I try to apply changes:

[root@srv351531 build]# ./gkctl lua/dyn.config
./lua/gatekeeper/dylib.lua:68: ';' expected near '=' at line 2

Perhaps we have an error in dylib.lua, or maybe I do something wrong, because C++ and lua is not my strong side:)

[cjdoucette]

I don't think we need to specify require "gatekeeper/dylib" in the file because we add the path to the dynamic configuration file when Gatekeeper is started, and load that file:

https://github.com/AltraMayor/gatekeeper/blob/master/config/dynamic.c#L486-L499

Instead, I think that the call to add_fib_entry() might be failing because the FIB is not correctly set-up. If you look in the Gatekeeper log, do you see entries that say GATEKEEPER: lpm: IPv4 lookup miss?

If so, it is likely because:

1. The back interface of your Gatekeeper server is not in the same network as the gateway address in the add_fib_entry() call.

2. Even if you fixed (1), your back interface/gateway and Grantor appear to be in the same network (10.255.0.0/24?), a feature which is not currently supported (What are the expected network configurations across a deployment? #267 (comment)) because it is not in our targeted deployment scenarios.

If it seems to be one of the above issues, let me know, as I think we should certainly add a log message with more information.

[dahnevskiy]

In my log file I see:

GATEKEEPER: lpm: IPv4 lookup miss
GATEKEEPER: lpm: IPv4 lookup miss

Here my network config:

	local front_ports = {"enp133s0f0"}
	local front_ips  = {"10.255.0.18/29"}
	local front_bonding_mode = staticlib.c.BONDING_MODE_ROUND_ROBIN
	local front_vlan_tag = 0x123
	local front_vlan_insert = false
	local front_mtu = 1500

	local back_ports = {"enp133s0f1"}
	local back_ips  = {"10.255.0.226/29"}
	local back_bonding_mode = staticlib.c.BONDING_MODE_ROUND_ROBIN
	local back_vlan_tag = 0x456
	local back_vlan_insert = false
	local back_mtu = 1500

I double checked, that my back interface (10.255.0.226/29) is in the same subnet with gateway (10.255.0.225)
Also, grantor server (10.255.0.66) not in my gatekeeper subnet (10.255.0.224/29)

[AltraMayor]

Hi @dahnevskiy,

Although my sidenotes below don't address the problem on hand, they are still relevant here:

1. There is a non-zero bit after the prefix length in the network prefix of the back network: 10.255.0.226/29. In detail, the last octet 226 in binary is 1110,0010, and the mask 29 covers the first 5 bits, namely "1110,0", therefore last bits (i.e. "010") should be all zeros to avoid confusion. Replacing 226 with 224 would solve this issue.

2. The MTU of the back network should be slightly bigger than the MTU of the front network to avoid the heavy work of fragmenting the encapsulated packets sent to Grantors servers. This extra, unnecessary load can cost one a lot during a DDoS attack.

[AltraMayor]

Please ignore item 1 above; my mistake.

[cjdoucette]

Thanks for the clarifications @dahnevskiy, I will see if I can replicate the issue.

[cjdoucette]

Are you actually seeing gk: failed to add an FIB entry or is it failing silently?

Try adding this to the end of your Lua script to dump the ARP table:

os.execute("sleep 3")
local llsc = staticlib.c.get_lls_conf()
if llsc == nil then
        return "lls: failed to fetch config to dump caches"
end
return dylib.list_lls_arp(llsc, dylib.print_lls_dump_entry, "")

If the gateway entry you're adding (10.255.0.225) doesn't respond to ARP requests, it will show up as stale in the ARP table and you'll get the GATEKEEPER: lpm: IPv4 lookup miss errors in the log.

[dahnevskiy]

I added lls check to lua script, and yes, I get the stale ARP:

[root@srv351531 build]# ./gkctl lua/dyn.config
LLS cache entry:: [state: stale, ip: 10.255.0.225, mac: 00:00:00:00:00:00, port: 0]

I found and fix problem in configuration, thanks for the help!

By the way, I can't use /30 netmask in my net.lua.

If I use it, for example in my net.lua:

        local front_ports = {"enp133s0f0"}
        local front_ips  = {"10.255.0.226/30"}
        local front_bonding_mode = staticlib.c.BONDING_MODE_ROUND_ROBIN
        local front_vlan_tag = 0x123
        local front_vlan_insert = false
        local front_mtu = 1500

        local back_ports = {"enp133s0f1"}
        local back_ips  = {"10.255.0.18/30"}
        local back_bonding_mode = staticlib.c.BONDING_MODE_ROUND_ROBIN
        local back_vlan_tag = 0x456
        local back_vlan_insert = false
        local back_mtu = 1500

in my logs, when I launch gatekeeper:

GATEKEEPER: main: cycles/second = 2200008189, cycles/millisecond = 2200008, picosec/cycle = 454
resolve_xsym(1): EBPF_PSEUDO_CALL to external function: init_ctx_to_cookie
rte_bpf_elf_load(fname="./lua/bpf/granted.bpf", sname="init") successfully creates 0x7f7044475000(jit={.func=0x7f7044474000,.sz=316});
resolve_xsym(1): EBPF_PSEUDO_CALL to external function: pkt_ctx_to_cookie
resolve_xsym(15): EBPF_PSEUDO_CALL to external function: pkt_ctx_to_pkt
resolve_xsym(33): EBPF_PSEUDO_CALL to external function: gk_bpf_prep_for_tx
rte_bpf_elf_load(fname="./lua/bpf/granted.bpf", sname="pkt") successfully creates 0x7f7044473000(jit={.func=0x7f7044472000,.sz=181});
rte_bpf_elf_load(fname="./lua/bpf/declined.bpf", sname="init") successfully creates 0x7f7044471000(jit={.func=0x7f7044470000,.sz=4});
rte_bpf_elf_load(fname="./lua/bpf/declined.bpf", sname="pkt") successfully creates 0x7f704446f000(jit={.func=0x7f704446e000,.sz=8});
resolve_xsym(1): EBPF_PSEUDO_CALL to external function: init_ctx_to_cookie
rte_bpf_elf_load(fname="./lua/bpf/grantedv2.bpf", sname="init") successfully creates 0x7f704446d000(jit={.func=0x7f704446c000,.sz=386});
resolve_xsym(1): EBPF_PSEUDO_CALL to external function: pkt_ctx_to_cookie
resolve_xsym(4): EBPF_PSEUDO_CALL to external function: pkt_ctx_to_pkt
resolve_xsym(60): EBPF_PSEUDO_CALL to external function: gk_bpf_prep_for_tx
rte_bpf_elf_load(fname="./lua/bpf/grantedv2.bpf", sname="pkt") successfully creates 0x7f704446b000(jit={.func=0x7f704446a000,.sz=350});
resolve_xsym(1): EBPF_PSEUDO_CALL to external function: init_ctx_to_cookie
rte_bpf_elf_load(fname="./lua/bpf/web.bpf", sname="init") successfully creates 0x7f7044469000(jit={.func=0x7f7044468000,.sz=386});
resolve_xsym(1): EBPF_PSEUDO_CALL to external function: pkt_ctx_to_cookie
resolve_xsym(4): EBPF_PSEUDO_CALL to external function: pkt_ctx_to_pkt
resolve_xsym(148): EBPF_PSEUDO_CALL to external function: gk_bpf_prep_for_tx
rte_bpf_elf_load(fname="./lua/bpf/web.bpf", sname="pkt") successfully creates 0x7f7044467000(jit={.func=0x7f7044466000,.sz=960});
GATEKEEPER: net: port 0 (0000:1c:00.0) on the front interface only supports RSS hash functions 0x38d34, but Gatekeeper asks for 0x8c
GATEKEEPER: net: port 1 (0000:1c:00.1) on the back interface only supports RSS hash functions 0x38d34, but Gatekeeper asks for 0x8c
GATEKEEPER: lls: calculate_mempool_config_para: total_pkt_burst = 2144 packets, total_rx_desc = 640 descriptors, total_tx_desc = 256 descriptors, max_num_pkt = 3040 packets, num_mbuf = 4095 packets.
HASH: rte_hash_create has invalid parameters
GATEKEEPER GK: Cannot create hash table for neighbor FIB
GATEKEEPER GK: Failed to setup the FIB entry for the front network prefixes at init_fib_tbl
GATEKEEPER GK: Failed to initialize the FIB table at setup_gk_lpm
Device with port_id=1 already stopped
Device with port_id=0 already stopped

Its not a huge problem for us, we can use /29 or less specific network, but I guess its maybe a bug and you wanna know about it.

[AltraMayor]

Hi @dahnevskiy,

We are going to investigate the /30 bug, but would you mind describing how you solved the configuration issue? Understanding it may help us to conceive ways to make the configuration more robust.

[dahnevskiy]

I continue my tests, and im trying to generate 1 mpps synflood attack to destination 10.254.71.130

In my grantor server I see logs:

GATEKEEPER LLS: 10.255.0.65: 3c:8a:b0:81:f1:fa (port 0) (0 holds)
GATEKEEPER: lpm: IPv4 lookup miss
GATEKEEPER GT: gt_neigh_get_ether_cache: receiving an IPv4 packet with destination IP address 10.254.71.130, which is not on the same subnet as the GT server
GATEKEEPER: lpm: IPv4 lookup miss
GATEKEEPER GT: gt_neigh_get_ether_cache: receiving an IPv4 packet with destination IP address 10.254.71.130, which is not on the same subnet as the GT server
GATEKEEPER LLS: LLS cache (arp)

I guess its problem, because on 10.254.71.130 I can't see any traffic.
I guess its some kind of misconfiguration on my side, what I can to check?

[dahnevskiy]

Hi @dahnevskiy,

We are going to investigate the /30 bug, but would you mind describing how you solved the configuration issue? Understanding it may help us to conceive ways to make the configuration more robust.

just have a problem with links from gatekeeper server to asr9k. nothing to do with gatekeeper software

[AltraMayor]

The log message GATEKEEPER GT: gt_neigh_get_ether_cache: receiving an IPv4 packet with destination IP address 10.254.71.130, which is not on the same subnet as the GT server reflects a design/implementation decision derived from our first, ongoing deployment: we assumed that Grantor servers are in the same subnet of the protected machines. This assumption is true in our first deployment and simplifies the code of Grantor servers since it doesn't need to have a routing table like Gatekeeper servers. This assumption is not true in your deployment environment since your Grantor server has the IP address 10.255.0.66 and the protected host 10.254.71.130.

This is not a fundamental limitation and we can think of how to support it. Would it be possible for we schedule a meeting, so our team can understand your deployment environment?

[cjdoucette]

We are going to investigate the /30 bug

Looks like the DPDK cuckoo hash table must have a length of at least 8.

[dahnevskiy]

The log message GATEKEEPER GT: gt_neigh_get_ether_cache: receiving an IPv4 packet with destination IP address 10.254.71.130, which is not on the same subnet as the GT server reflects a design/implementation decision derived from our first, ongoing deployment: we assumed that Grantor servers are in the same subnet of the protected machines. This assumption is true in our first deployment and simplifies the code of Grantor servers since it doesn't need to have a routing table like Gatekeeper servers. This assumption is not true in your deployment environment since your Grantor server has the IP address 10.255.0.66 and the protected host 10.254.71.130.

This is not a fundamental limitation and we can think of how to support it. Would it be possible for we schedule a meeting, so our team can understand your deployment environment?

here our deployment environment:

we have a lot of hosting switches, which terminates many small clients subnets (in my example 8.8.8.0/29, 8.8.8.8/29 etc...) , so we can't use the current design of gatekeeper, because in current design we need to deploy grantor server on each small client subnet, and use a part of client subnet for grantor server addressing.

Instead, we want to deploy grantor machines in our aggregation layer, which can protect multiple clients subnets on different switches. But in this scheme grantor servers will not in the same subnet with protected machines.

If you need more information, we can schedule a meeting, why not. Or I can answer here:)

[AltraMayor]

Thank you for the helpful diagram.

Based on your diagram, there's only one gateway for all traffic that is not local to a Grantor server. Is this assumption correct? If so, Grantor servers can support a single gateway for non-local traffic and still avoid having an LPM lookup. Would this solution work for you?