Flash operations break PSRAM interface

[technoblogy]

I have a program using LittleFS that works OK when not using PSRAM, but hangs when using PSRAM.

Here's a minimal demonstration:

#include <LittleFS.h>

uint32_t Workspace[10000] PSRAM;
const int len = 1000;

bool checkdata () {
  for (int i=0; i<len; i++) if (Workspace[i] != i) return false;
  return true;
}

void savedata () {
  for (int i=0; i<len; i++) Workspace[i] = i;
  if (checkdata()) Serial.println("Data setup OK");
  LittleFS.begin();
  File file;
  file = LittleFS.open("/DATA.DAT", "w");
  file.close();
}

void setup() {
  Serial.begin(9600);
}

void loop() {
  savedata();
  Serial.println("Saved");
  if (!checkdata()) Serial.println("Data error");
  delay(5000);
}

The program simply opens and closes a file on LittleFS.

Running the program displays Data error indicating that LittleFS has corrupted the array Workspace. If you remove the PSRAM attribute from Workspace the program runs without error.

[technoblogy]

Sorry, I forgot to give configuration details:

• Pimoroni Pico Plus 2 board running at 150MHz (it has 16MB of flash and 8MB of PSRAM).
• Running on a MacBook Pro with Catalina (MacOS 10.15), using Arduino IDE 1.8.19.
• I'm using the Raspberry Pi Pico/RP2040/RP2350 core version 4.1.1.
• Settings: Board is PimoroniPicoPlus2, Flash Size set to 16MB (Sketch: 8MB, FS: 8MB), others at defaults.

I'm running on the ARM core, but I think the behaviour is the same with the RISC-V core.

[maxgerhardt]

Do you have a debugger available in the form of another Raspberry Pi Pico (RP2040 based)? You could use it to flash debugprobe on it, connect it to the SWD port of the RP2350 board and then set a "hardware data write breakpoint" at the first address of the Workspace PSRAM array after it was initialized, so the debugger would halt at the first instruction that tries to corrupt the array.

[technoblogy]

I do have an RP2040 Raspberry Pi Pico, but I'm not familiar with the rest of the procedure you describe. Can you suggest a link to more details?

[maxgerhardt]

I've done a writeup at https://github.com/maxgerhardt/pio-psram-debug.

Sadly the PimoroniPicoPlus2 has its debugging pins (SWD) at a different, smaller-pitch connector (JST-SH). I'm not sure if you have a 3-pin JST-SH adapter (like https://www.aliexpress.us/item/2251832780304648.html) or whether you can solder to it. But without those pins, debugging is not possible.

You can check though whether the same problem reproduces in the PlatformIO build system with only the regular USB connection and doing the first 2 chapters of the README.

[technoblogy]

Gosh - thank you very much. That's very detailed.

It's all a bit new to me as I haven't used those tools before, but I'll give it a go!
David

[maxgerhardt]

My previously ordered Pimoroni Pico Plus 2W (with WiFi chip) should arrive next week anyways, and it should have the same PSRAM as the Plus 2. So I should be able to reproduce and debug that bug, too; no worries if you can't get it it work.

[technoblogy]

OK, that's good to know!

[earlephilhower]

I haven't run this, but I do have boards w/PSRAM in them. But I think I know what's happening w/o running anything...

The QSPI interface is shared between flash and PSRAM. In general operation, it's owned by the cache hardware and untouched by the CPU itself. Reads "just" happen as far as the CPU is concerned, be they PSRAM or flash.

To erase or write a flash block, however, the CPU needs to take over the QSPI interface by writing some registers and manually force it to send specific commands to flash. When done, it should reset ownership to the HW by restoring those registers.

I have the feeling that when the QSPI registers are restored, they're only restored for flash operation. The PSRAM interface is then lost.

That would explain the LittleFS issue. It's not LittleFS, it's flash_range_erase and its brothers that's the issue. There was also the same symptom seen in a discussion when trying to read the Pico ID (which reads the flash chip's serial), so even more evidence.

[technoblogy]

Is there a workaround, or will it need an update to the core?

[earlephilhower]

Is there a workaround...

Probably not

...or will it need an update to the core?

Probably so

😆

[earlephilhower]

A little bit of testing shows the ROM flash command operations change the configuration of the 2nd QMI memory space (aka PSRAM)...can't see why they would do so, but it effectively breaks comms to the PSRAM chip since the QMI will now not have the right timing nor commands.

A quick test that saves and restores the QMI state around flash operations seems to work, but this will need to be wrapped around all flash_xxx calls in core_wrap.

Simple working LittleFS example (warning, formats the onchip FS). Set FS size to anything, and if you comment out the QMI restore it will fail. Leave it in and it will work...

#include <hardware/address_mapped.h>
#include <hardware/clocks.h>
#include <hardware/gpio.h>
#include <hardware/regs/addressmap.h>
#include <hardware/spi.h>
#include <hardware/structs/qmi.h>
#include <hardware/structs/xip_ctrl.h>

#include <LittleFS.h>

void dump() {
  Serial.printf("qmi_hw->m[1].timing = %08lx\n", qmi_hw->m[1].timing);
  Serial.printf("qmi_hw->m[1].rfmt   = %08lx\n", qmi_hw->m[1].rfmt);
  Serial.printf("qmi_hw->m[1].rcmd   = %08lx\n", qmi_hw->m[1].rcmd);
  Serial.printf("qmi_hw->m[1].wfmt   = %08lx\n", qmi_hw->m[1].wfmt);
  Serial.printf("qmi_hw->m[1].wcmd   = %08lx\n", qmi_hw->m[1].wcmd);
  Serial.printf("xip_ctrl_hw->ctrl   = %08lx\n", xip_ctrl_hw->ctrl);
}

int data[10000] PSRAM;

void check() {
  bool fail = false;
  for (int i = 0; i < 10000 && !fail; i++) {
    if (data[i] != i) {
      fail = true;
    }
  }
  Serial.printf("%s\n", fail ? "FAIL" : "PASS");
}

void prep() {
  for (int i = 0; i < 10000; i++) {
    data[i] = i;
  }
}

void setup() {
  delay(5000);
  prep();
  auto s = qmi_hw->m[1];
  Serial.printf("Before flash write\n");
  dump();
  check();
  Serial.printf("FLASHWRITE\n");
  LittleFS.format();
  dump();
  Serial.printf("Restoring QMI\n");
  qmi_hw->m[1] = s;
  __compiler_memory_barrier();
  dump();
  check();
}

void loop() {
}

Before flash write
qmi_hw->m[1].timing = 61a48102
qmi_hw->m[1].rfmt   = 000612aa
qmi_hw->m[1].rcmd   = 000000eb
qmi_hw->m[1].wfmt   = 000012aa
qmi_hw->m[1].wcmd   = 00000038
xip_ctrl_hw->ctrl   = 00000883
PASS
FLASHWRITE
qmi_hw->m[1].timing = 60007203
qmi_hw->m[1].rfmt   = 000492a8
qmi_hw->m[1].rcmd   = 000000eb
qmi_hw->m[1].wfmt   = 000012aa
qmi_hw->m[1].wcmd   = 00000038
xip_ctrl_hw->ctrl   = 00000883
Restoring QMI
qmi_hw->m[1].timing = 61a48102
qmi_hw->m[1].rfmt   = 000612aa
qmi_hw->m[1].rcmd   = 000000eb
qmi_hw->m[1].wfmt   = 000012aa
qmi_hw->m[1].wcmd   = 00000038
xip_ctrl_hw->ctrl   = 00000883
PASS

[maxgerhardt]

This should definitely be logged to https://github.com/raspberrypi/pico-sdk/issues if this behavior is not specified in the RP2350 datasheet.