Bluetooth: Non functional on nRF51 with fault

[thedjnK]

Describe the bug
It seems that bluetooth is non functional on nRF51 devices, running the peripheral_hr sample application gives the following output:

*** Booting Zephyr OS build v3.6.0-6522-g90ecdf0dab8c ***
[00:00:00.316,192] <inf> bt_hci_core: HW Platform: Nordic Semiconductor (0x0002)
[00:00:00.316,345] <inf> bt_hci_core: HW Variant: nRF51x (0x0001)
[00:00:00.316,467] <inf> bt_hci_core: Firmware: Standard Bluetooth controller (0x00) Version 3.6 Build 99
[00:00:00.320,556] <inf> bt_hci_core: Identity: C1:58:CF:2B:AB:12 (random)
[00:00:00.320,739] <inf> bt_hci_core: HCI: version 5.4 (0x0d) revision 0x0000, manufacturer 0x05f1
[00:00:00.320,922] <inf> bt_hci_core: LMP: version 5.4 (0x0d) subver 0xffff
Bluetooth initialized
Starting Legacy Advertising (connectable and scannable)
Advertising successfully started
Connected
[00:00:04.741,607] <err> os: ***** HARD FAULT *****
[00:00:04.741,699] <err> os: r0/a1:  0x02000400  r1/a2:  0x20001acc  r2/a3:  0x000100b9
[00:00:04.741,790] <err> os: r3/a4:  0x200025b7 r12/ip:  0xff774201 r14/lr:  0x00013429
[00:00:04.741,821] <err> os:  xpsr:  0x01000011
[00:00:04.741,882] <err> os: Faulting instruction address (r15/pc): 0x000100ba
[00:00:04.742,034] <err> os: >>> ZEPHYR FATAL ERROR 0: CPU exception on CPU 0
[00:00:04.742,095] <err> os: Fault during interrupt handling

[00:00:04.742,248] <err> os: Current thread: 0x20001010 (unknown)
[00:00:04.815,399] <err> os: Halting system

This address is zephyr/subsys/bluetooth/controller/ll_sw/ull_llcp_common.c:936. I have also tried this in a custom application which has more logic and that faulted with zephyr/subsys/bluetooth/controller/ll_sw/nordic/lll/lll_conn.c:583 (discriminator 3) instead, that one is shown here:

E: ***** HARD FAULT *****
E: ARCH_EXCEPT with reason 3

E: r0/a1:  0x00000003  r1/a2:  0x0203001b  r2/a3:  0x40001000
E: r3/a4:  0x00000003 r12/ip:  0x000000a5 r14/lr:  0x0001eae9
E:  xpsr:  0x01000011
E: Faulting instruction address (r15/pc): 0x0001eaf8
E: >>> ZEPHYR FATAL ERROR 3: Kernel oops on CPU 0
E: Fault during interrupt handling

E: Current thread: 0x200010a8 (unknown)
E: Halting system

To Reproduce
Run peripheral_hr with default config on an nrf51-based board

Expected behavior
Bluetooth to work

Impact
Showstopper

Environment (please complete the following information):

• Commit SHA or Version used: 90ecdf0

[thedjnK]

Have had a look back and seemingly issue has been present since zephyr 3.2, in 3.1 it seems stable but in 3.2 it starts faulting

[thedjnK]

e1c2c36f569a8e06184a2c5574f1d25ba00df8d7 is the offending commit

[cvinayak]

Tried the peripheral_hr sample on a nrf51dongle/nrf51822, I do not see an issue:

[2024-06-16 06:42:00.006] *** Booting Zephyr OS build v3.6.0-6486-gae61a77781fc ***
[2024-06-16 06:42:00.013] [00:00:00.007,873] bt_hci_core: HW Platform: Nordic Semiconductor (0x0002)
[2024-06-16 06:42:00.025] [00:00:00.007,995] bt_hci_core: HW Variant: nRF51x (0x0001)
[2024-06-16 06:42:00.034] [00:00:00.008,148] bt_hci_core: Firmware: Standard Bluetooth controller (0x00) Version 3.6 Build 99
[2024-06-16 06:42:02.702] [00:00:00.012,084] bt_hci_core: Identity: D1:54:BC:A1:F7:9B (random)
[2024-06-16 06:42:02.711] [00:00:00.012,298] bt_hci_core: HCI: version 5.4 (0x0d) revision 0x0000, manufacturer 0x05f1
[2024-06-16 06:42:02.724] [00:00:00.012,481] bt_hci_core: LMP: version 5.4 (0x0d) subver 0xffff
[2024-06-16 06:42:02.733] Bluetooth initialized
[2024-06-16 06:42:02.735] Starting Legacy Advertising (connectable and scannable)
[2024-06-16 06:42:02.741] Advertising successfully started
[2024-06-16 06:42:02.745] Checking LED device...done.
[2024-06-16 06:42:02.748] Configuring GPIO pin...done.
[2024-06-16 06:42:02.751] Start blinking LED...
[2024-06-16 06:42:08.035] Connected
[2024-06-16 06:42:08.036] Stop blinking LED.
[2024-06-16 06:42:26.764] [00:00:25.776,153] hrs: HRS notifications enabled
[2024-06-16 06:42:26.771] HRS notification status changed: enabled
[2024-06-16 06:42:40.915] [00:00:39.926,025] bas: BAS Notifications enabled
[2024-06-16 06:42:50.816] [00:00:49.827,148] bas: BAS Notifications disabled
[2024-06-16 06:42:50.822] [00:00:49.827,392] hrs: HRS notifications disabled
[2024-06-16 06:42:50.829] HRS notification status changed: disabled
[2024-06-16 06:42:50.835] Disconnected (reason 0x13)
[2024-06-16 06:42:50.838] Starting Legacy Advertising (connectable and scannable)
[2024-06-16 06:42:50.843] Start blinking LED...

1. What toolchain are you using? I am on upstream zephyr 3b85e8b and using the Zephyr SDK: zephyr-sdk-0.16.5-
2. Which peer central device did you use to connect from? I used LightBlue v5.0.6, iPhone 13 Pro, iOS 17.5.1

[thedjnK]

Try connecting and disconnecting a few times, from trying many commits trying to find the issue it can happen instantly or it can be on the 2nd or 3rd connection. Current commit: 6663096 but it happens on every commit after and including e1c2c36. Was using sdk-ng 0.16.5-1 where I saw issue, updated to 0.16.8 and still see issue. Remote device is nRF connect 2.4.9, iPhone SE 2020, iOS 13.4.1 (same configuration for connecting to other zephyr devices e.g. nrf52 and nrf53 without issue)

[thedjnK]

Retried with LED support enabled:

*** Booting Zephyr OS build v3.6.0-6528-g66630961627f ***
[00:00:00.317,077] <inf> bt_hci_core: HW Platform: Nordic Semiconductor (0x0002)
[00:00:00.317,230] <inf> bt_hci_core: HW Variant: nRF51x (0x0001)
[00:00:00.317,352] <inf> bt_hci_core: Firmware: Standard Bluetooth controller (0x00) Version 3.7 Build 0
[00:00:00.321,441] <inf> bt_hci_core: Identity: C1:58:CF:2B:AB:12 (random)
[00:00:00.321,624] <inf> bt_hci_core: HCI: version 5.4 (0x0d) revision 0x0000, manufacturer 0x05f1
[00:00:00.321,807] <inf> bt_hci_core: LMP: version 5.4 (0x0d) subver 0xffff
Bluetooth initialized
Starting Legacy Advertising (connectable and scannable)
Advertising successfully started
Checking LED device...done.
Configuring GPIO pin...done.
Start blinking LED...
Connected
ASSERTION FAIL [0] @ WEST_TOPDIR/zephyr/subsys/bluetooth/controller/ll_sw/ull_llcp_common.c:707
[00:00:22.656,494] <err> os: ***** HARD FAULT *****
[00:00:22.656,524] <err> os: ARCH_EXCEPT with reason 3

[00:00:22.656,616] <err> os: r0/a1:  0x00000003  r1/a2:  0x0000007c  r2/a3:  0x00000000
[00:00:22.656,707] <err> os: r3/a4:  0x00000003 r12/ip:  0x8100efeb r14/lr:  0x0001810b
[00:00:22.656,738] <err> os:  xpsr:  0x01000011
[00:00:22.656,799] <err> os: Faulting instruction address (r15/pc): 0x000100b4
[00:00:22.656,951] <err> os: >>> ZEPHYR FATAL ERROR 3: Kernel oops on CPU 0
[00:00:22.657,012] <err> os: Fault during interrupt handling

[00:00:22.657,165] <err> os: Current thread: 0x20001040 (unknown)
[00:00:22.747,955] <err> os: Halting system

That line is:

static void lp_comm_send_req(struct ll_conn *conn, struct proc_ctx *ctx, uint8_t evt, void *param)
{
        switch (ctx->proc) {
...
        default:
                /* Unknown procedure */
                LL_ASSERT(0); <----
        }

[cvinayak]

Is this reproducible on nRF development boards or is this custom board?

[thedjnK]

Is this reproducible on nRF development boards or is this custom board?

This is a third party nRF51-based module (rm1xx_dvk) with XLR3, all that's different in the config for this application is the UART mapping, I was able to fault it also by building for nrf51dk and flashing that instead (albeit no UART output). Only nordic nRF51 dev board I have is from 2014 and pre-XLR3 silicon which I assume will not work at all

[cvinayak]

Can’t think of anything other than to increase call stack for Bluetooth rx thread. Not sure which kconfig, stack has been undergoing changes lately.

[thedjnK]

Have tried increasing stack sizes, no difference. Though changing enabled drivers changes where it happens, having a look at it it's generally on lines setting a variable. Are the structs all properly setup for cortex M0 i.e. 4-byte boundaries or packed, since it doesn't support non-aligned memory access?