Debugging environment proposal

[d-a-v]

Hello,

Once esp8266/Arduino#7404 is merged,(merged)
if you are tunning linux, (or WSL on windows),
you can run your ftp example on your PC and use host debug tools.

I have tried with ftp cmdline client and littlefs-ftp-server: it works with little files but not big ones (600KB), failing log:
(on both esp and emulation environment)

[FTP] Transfer 536 bytes net->FS
[FTP] Transfer 536 bytes net->FS
[FTP] Transfer 536 bytes net->FS
[FTP] Transfer 536 bytes net->FS
[FTP] Transfer 152 bytes net->FS
(both client and server frozen at that point)

IMPORTANT: you must replace all your %S by %s (and you can leave them like that) (they are very harmful for linux's printf, and man page advise to not use them).

To run the emulation environment, provided the above PR is merged and if you use esp8266 git master version,

cd /path/to/esp8266ArduinoCore/tests/host
export LIB=/path/to/libraries/esp8266FTPServer
make ULIBDIRS=${LIB} FORCE32=0 D=1 V=1 ${LIB}/examples/FTPServerSample/LittleFSSample/LittleFSSample

./bin/LittleFSSample/LittleFSSample  -v 
valgrind ./bin/LittleFSSample/LittleFSSample  -v 
gdb ./bin/LittleFSSample/LittleFSSample 
  -> run -v

port will be 9021 (not 21)

[dplasa]

Hello David,
I just tried to do so, but there is something missing. So I check out latest https://github.com/esp8266/Arduino and tried to compile as you have written but if fails immediately with:

Makefile:37: compiling in native mode
make -f Makefile MKFLAGS=-Wextra INODIR=/home/dplasa/workspace/Arduino/FTPClientServer//examples/FTPServerSample/LittleFSSample/ INO=LittleFSSample bin/LittleFSSample/LittleFSSample
make[1]: Verzeichnis „/home/dplasa/workspace/esp8266_core/tests/host“ wird betreten
Makefile:37: compiling in native mode
userlib: using directory '/home/dplasa/workspace/Arduino/FTPClientServer/'
cc -include common/mock.h -include common/c_types.h  -std=c99 -DDEBUG_ESP_PORT=Serial -DDEBUG_ESP_SSL -DDEBUG_ESP_TLS_MEM -DDEBUG_ESP_HTTP_CLIENT -DDEBUG_ESP_HTTP_SERVER -DDEBUG_ESP_CORE -DDEBUG_ESP_WIFI -DDEBUG_ESP_HTTP_UPDATE -DDEBUG_ESP_UPDATER -DDEBUG_ESP_OTA -DDEBUG_ESP_MDNS -Wall -O0 -fno-common -g  -DHTTPCLIENT_1_1_COMPATIBLE=0 -DLWIP_IPV6=0 -DHOST_MOCK=1 -DNONOSDK221=1 -Wextra -Wimplicit-fallthrough=2  -funsigned-char -I. -Icommon -I../../cores/esp8266 -I../../libraries/DNSServer/src -I../../libraries/ESP8266AVRISP/src -I../../libraries/ESP8266HTTPClient/src -I../../libraries/ESP8266HTTPUpdateServer/src -I../../libraries/ESP8266WebServer/src -I../../libraries/ESP8266WiFi/src -I../../libraries/ESP8266WiFiMesh/src -I../../libraries/ESP8266httpUpdate/src -I../../libraries/ESP8266mDNS/src -I../../libraries/Ethernet/src -I../../libraries/GDBStub/src -I../../libraries/Hash/src -I../../libraries/LittleFS/src -I../../libraries/SD/src -I../../libraries/SDFS/src -I../../libraries/SPISlave/src -I../../libraries/Servo/src -I../../libraries/Ticker/src -I../../libraries/esp8266/src -I../../libraries/ArduinoOTA -I../../libraries/DNSServer -I../../libraries/EEPROM -I../../libraries/ESP8266AVRISP -I../../libraries/ESP8266HTTPClient -I../../libraries/ESP8266HTTPUpdateServer -I../../libraries/ESP8266LLMNR -I../../libraries/ESP8266NetBIOS -I../../libraries/ESP8266SSDP -I../../libraries/ESP8266SdFat -I../../libraries/ESP8266WebServer -I../../libraries/ESP8266WiFi -I../../libraries/ESP8266WiFiMesh -I../../libraries/ESP8266httpUpdate -I../../libraries/ESP8266mDNS -I../../libraries/Ethernet -I../../libraries/GDBStub -I../../libraries/Hash -I../../libraries/LittleFS -I../../libraries/SD -I../../libraries/SDFS -I../../libraries/SPI -I../../libraries/SPISlave -I../../libraries/Servo -I../../libraries/SoftwareSerial -I../../libraries/TFT_Touch_Shield_V2 -I../../libraries/Ticker -I../../libraries/Wire -I../../libraries/esp8266 -I../../tools/sdk/include -I../../tools/sdk/lwip2/include -I/home/dplasa/workspace/Arduino/FTPClientServer/ -I/home/dplasa/workspace/Arduino/FTPClientServer//examples/FTPServerSample/LittleFSSample//.. -MD -MF bin/.lfs.c.d -c -o ../../cores/esp8266/../../libraries/LittleFS/src/lfs.c.o ../../cores/esp8266/../../libraries/LittleFS/src/lfs.c
../../cores/esp8266/../../libraries/LittleFS/src/lfs.c:10:10: fatal error: ../lib/littlefs/lfs.c: Datei oder Verzeichnis nicht gefunden
 #include "../lib/littlefs/lfs.c"
          ^~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
Makefile:229: recipe for target '../../cores/esp8266/../../libraries/LittleFS/src/lfs.c.o' failed
make[1]: *** [../../cores/esp8266/../../libraries/LittleFS/src/lfs.c.o] Error 1

Any ideas?

[d-a-v]

You should import submodules with git submodule update --init (from within the core repo)

[dplasa]

IMPORTANT: you must replace all your %S by %s (and you can leave them like that) (they are very harmful for linux's printf, and man page advise to not use them).

OK for linux/mock environment but on the esp8266 itself -- don't I need them if a PROGMEM string is output? Or do the printf* functions in the esp8266 core some magic internally if they detects that the pointer directs to flash memory?

[dplasa]

You should import submodules with git submodule update --init (from within the core repo)

Ok thanks, now I'm able to compile. This mock environment is pretty cool! ;-) I noticed a lot of

(mock) TODO getRemotePort()
(mock) TODO getRemoteAddress()
[FTP] control server got connection from (IP unset):0
...
(mock) TODO: ClientContext::discard_received()

output, indicating that some (mock) parts are missing. I'm not quite sure, how to debug further: if I run the compiled program and upload a small file (as little as 10 bytes ... 10k) the program either works as expected or the ftp server loops forever in FTPServer::processCommand processing the STOR command by the client but no data seem to be transferred at all or the ftp server just does nothing, and again my feeling is, no data is transferred at all. If I break the server in gdb, it is in __GI___nanosleep doing nothing....

[d-a-v]

I tried with the mock environment to try to find out the issues I have on the esp itself.
edit: if mandatory functions are missing in the mock environment, we can add them.

the program either works as expected or the ftp server loops forever in FTPServer::processCommand

What do you mean by "either", is it predictable ?

[d-a-v]

OK for linux/mock environment but on the esp8266 itself -- don't I need them if a PROGMEM string is output? Or do the printf* functions in the esp8266 core some magic internally if they detects that the pointer directs to flash memory?

I don't think so, %s is supposed to work with PSTR too (that's the espressif's os_printf_plus() function, also @earlephilhower's newlib extension more recently)

[dplasa]

I had the "stuck in processCommand" situation once, don't know how to reproduce. For now, it either works (small files) or just doesn't start at all (no data seem to be exchanged at all)
I hat to modify my code in FTPServer.cpp's handling of PASV mode to
sendMessage_P(227, PSTR("Entering Passive Mode (0,0,0,0,%d,%d)."), dataPort >> 8, dataPort & 255); since the mock function control.localIP().toString() would return "(IP unset)" instead of my interface's ip.

So yes in a way, it seems predictable: small files get sent, large files just don't even start to get sent.

[d-a-v]

So yes in a way, it seems predictable: small files get sent, large files just don't even start to get sent.

Only in mock environment ?

[dplasa]

Yes, only in mock environment. On a real ESP8266, I get e.g.

dplasa@folio1020:/tmp$ ftp -p 192.168.11.102
Connected to 192.168.11.102.
220 (espFTP 0.9.7-20200529)
Name (192.168.11.102:dplasa): ftp
331 Please specify the password.
Password:
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> dir
227 Entering Passive Mode (192,168,11,102,195,89).
150 Accepted data connection
-rw-r--r--    1    0    0      1677 Jun 20 20:54 cfg.ini
drwxr-xr-x    2    0    0         0 Jan 01  1970 css
-rw-r--r--    1    0    0       738 Jan 01  1970 index.html.gz
drwxr-xr-x    2    0    0         0 Jan 01  1970 scripts
-rw-r--r--    1    0    0        88 Jan 01  1970 wifi.txt
226 5 matches total
ftp> put test.bin 
local: test.bin remote: test.bin
227 Entering Passive Mode (192,168,11,102,195,89).
150 Connected to port 50009
226 File successfully transferred, 2650 ms, 38.641510 kB/s.
102400 bytes sent in 2.18 secs (45.8978 kB/s)
ftp> dir
227 Entering Passive Mode (192,168,11,102,195,89).
150 Accepted data connection
-rw-r--r--    1    0    0      1677 Jun 20 20:54 cfg.ini
drwxr-xr-x    2    0    0         0 Jan 01  1970 css
-rw-r--r--    1    0    0       738 Jan 01  1970 index.html.gz
drwxr-xr-x    2    0    0         0 Jan 01  1970 scripts
-rw-r--r--    1    0    0    102400 Jul 05 19:56 test.bin
-rw-r--r--    1    0    0        88 Jan 01  1970 wifi.txt
226 6 matches total

To me that looks like it's doing fine. I'm using "v2 - Lower Memory" compile option.

[dplasa]

Although... there seems something wrong if I put an 1M file:

ftp> put test.bin 
local: test.bin remote: test.bin
227 Entering Passive Mode (192,168,11,102,195,89).
150 Connected to port 50009
226 File successfully transferred, 20903 ms, 50.163898 kB/s.
1048576 bytes sent in 20.34 secs (50.3411 kB/s)
ftp> dir
227 Entering Passive Mode (192,168,11,102,195,89).
150 Accepted data connection
-rw-r--r--    1    0    0      1677 Jun 20 20:54 cfg.ini
drwxr-xr-x    2    0    0         0 Jan 01  1970 css
-rw-r--r--    1    0    0       738 Jan 01  1970 index.html.gz
drwxr-xr-x    2    0    0         0 Jan 01  1970 scripts
-rw-r--r--    1    0    0         0 Jul 05 21:04 test.bin
-rw-r--r--    1    0    0        88 Jan 01  1970 wifi.txt
226 6 matches total

The ftp client sucessfully transfers - on the client side a 0 byte file is created.... will investigate.

[dplasa]

If I put e.g. 512k (5121024) bytes (which works ok) and after that put 525312 (5131024) the file still stays 512k bytes in length - I'm starting to believe, this is some of problem with LittleFS?!?

dplasa@folio1020:/tmp$ ftp -p 192.168.11.102
Connected to 192.168.11.102.
220 (espFTP 0.9.7-20200529)
Name (192.168.11.102:dplasa): ftp
331 Please specify the password.
Password:
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> dir test.bin
227 Entering Passive Mode (192,168,11,102,195,89).
150 Accepted data connection
-rw-r--r--    1    0    0    524288 Jul 05 21:20 test.bin
226 1 matches total
ftp> put test.bin
local: test.bin remote: test.bin
227 Entering Passive Mode (192,168,11,102,195,89).
150 Connected to port 50009
226 File successfully transferred, 16186 ms, 32.454716 kB/s.
525312 bytes sent in 15.53 secs (33.0287 kB/s)
ftp> dir test.bin
227 Entering Passive Mode (192,168,11,102,195,89).
150 Accepted data connection
-rw-r--r--    1    0    0    524288 Jul 05 21:26 test.bin
226 1 matches total
ftp> del test.bin
250 Delete operation successful.
ftp> put test.bin
local: test.bin remote: test.bin
227 Entering Passive Mode (192,168,11,102,195,89).
150 Connected to port 50009
226 File successfully transferred, 14171 ms, 37.069508 kB/s.
525312 bytes sent in 13.69 secs (37.4706 kB/s)
ftp> dir test.bin
227 Entering Passive Mode (192,168,11,102,195,89).
150 Accepted data connection
-rw-r--r--    1    0    0    525312 Jul 05 21:26 test.bin
226 1 matches total
ftp> 

[dplasa]

OK, this is obviously due the fact to that LittleFS does COW, so when overwriting a file it could use up to 2xfilesize and in all my tests I exhausted space on the FS... see littlefs-project/littlefs#123
I fixed this upstream: when a file is written by the FTP server (STOR command) it first truncates, syncs (by closing the file) then re-opens it and thus LittleFS frees the space used by the file before.

I opened an issue: esp8266/Arduino#7426