add FTP and local file system API

[tryhus]

Hi,

I added an API to use FTP (not with SSL/TLS) and an other to manage local file system.
You can run the example project FTP.ino, you will need a FTP server (with write access).

Let me know if you are interested by these features or if you need some information.

[bastiri]

Hey,
i have tested the FTP.ino but it works not correctly.

In this part of the programm:

Serial.println("Create remote file : test");
if (ftp.mkdir("test") == false) {
Serial.println("Failed to create the file.");
}

it creates a directory and not a file.

An upload of the file after that is getting failed.
Here is the output:

Starting Arduino FTP client.
Connect to FTP server.
Change of directory
Free space 19562496
Create remote file : test
Rename remote file : test to test2
Write a binary file in local memory
Send the file to the server
Failed to upload the file.
Retreive the file from the server to local memory
Failed to download the file.
Check that the original file is identical to the one that was received
Failed to read file, value is corrupted
Free space 19560448
Display local files
myFile 8
myFileToLocalMemory 0
Remove local files
Display local files
Display remote files
Delete the created files
Failed to remove files : test2.
Failed to remove files : myFileToServer.
Display remote files
Disconnect to FTP server

Or is there any problem with my FTP? The user has full access and the directory was already created.
I hope you could help me!

Best regards
bastiri

[tryhus]

Hi Bastiri,

The user has full access

Have you test upload/download from an other FTP client (as Filezilla) to your FTP server and with the same user?

and the directory was already created.

You mean, the test2 directory is created by the Arduino FTP client?

What FTP server did you use? Is it a public FTP server? If so, can you send me the hostname and the password?

I'll look if I can find an online public FTP server with read/write access...

Best regards

[bastiri]

Hi Tryhus,

thank you for your fast answer. I have test it on two different ftp servers, one by ionos and the other one is a filezilla-server on my local windows system. If i connect to the servers by filezilla-client, i`ll be able to upload/download files and i can create and delete directories with the same user which is already entered in the secrets.h
But your ftp-client only creates and renames the directory but no file in this case.

Now i tested your ftp-client on this public ftp-server: https://dlptest.com/ftp-test/
That was a good idea. Here it works fine!

So it must depend on the my ftp-server. What could be different? I could create an account for you this evening.

Best regards

[tryhus]

Is that all the steps are successful with https://dlptest.com/ftp-test/ for you?

I could create an account for you this evening.

ok, you can send me hostname, username and password by private mail.

Best regards

[tryhus]

As you did I tested with a Filezilla server and I had the same error.
I solved the problem by enabling the FTP passive mode.
Now passive mode is enable by default. I also tested passive mode with my synology FTP server and it works both with active mode and passive mode.

If you use filezilla server you will need to do some extra tuning:

• go to passive mode settings
• select use custom port range then open these ports from your router
• select use the following IP and enter your public IP (or domain name)

I also noticed that deleting directory didn't work with some FTP server. I fixed it by added one specific function to remove directory and an other one to remove file.
I think the old function worked only for linux server because in the linux world everything is a file Anyway the documentation is confusing about file/directory :)

And lately, I added the possibility to choose the FTP server port (21 by default). See in arduino_secrets.h.

Please, let me know if the code works with your FTP server.

Best regards

[bastiri]

Great! Now it works on both of my servers without any problems! Thats a very good feature!
In the next step i try to upload a file from a mkr sd shield to the server.

Best regards and thanks a lot!

[bastiri]

Hi Tryhus,
the non blocking function for ftp also works fine with local files.
Now i would like to upload files form a SD card and i use the SD library: https://www.arduino.cc/en/Reference/SD. When i try to compile the programm i get in trouble with char and String.
I think your program expects a String instead of a char. How can I solve this problem?

Code:
//open file on SD card and read it
File myFile;
myFile = SD.open("datalog.txt",FILE_READ);

Serial.println("Upload a file with non blocking function");
uploadFileNonBlocking(myFile, "datalog.txt");

Error:

bool uploadStart(const String& localFileName, const String& remoteFileName);
^
exit status 1
invalid conversion from 'const char*' to 'char' [-fpermissive]

[tryhus]

Hi Bastiri,

You can not pass myFile as argument of uploadStart function because there is not implicit conversion from type "File" to const String&
To get the file name you need to call the member function name() of myFile that return *char that can be implicitly convert to const String&

You must also check if the file has been opened, if an error has happened myFile is empty.

File myFile;
myFile = SD.open("datalog.txt", FILE_READ);
if (myFile == false) {
	Serial.println("File is not open");
}else {
	uploadFileNonBlocking(myFile.name(), "datalog.txt");
	//...
}

[bastiri]

Thank you very much! Now i can compile the programm!
But I get an error during the upload:

Send the file to the server from SD card
Upload error.
FTP last error : 8,68

The files I have tested are very small (1KB and 39KB)

[tryhus]

FTP last error : 8,68

This code means the file can not be open when sending to FTP server.
Did you close the file after reading it SD.close() ?

[bastiri]

Yes I have closed the file. When I close it directly after the reading or before the upload I get an error "File is not open".
If I close the file directly after the upload i get the error "FTP last error 8,68".
I can not get it started somehow

[tryhus]

Hi Bastiri,

I think the problem is that the data you are trying to send are is not saved in the local file system.
The functions download/upload are used to transfer a file between the local file system (persistent arduino memory) and the FTP server.
I added new functions in the FTP class that should fit your case.
Let's try to use write/read functions which takes as an argument the raw data of your file.

[bastiri]

Hi Tryhus,

I have tried your new programm and now I can upload a file to the FTP server, but the content of the uploaded file includes not readable characters. Its not the same as in the file on the SD card. I have seen that the file has the size of the buffer (default 128). I tried to enlarge the buffer size but I am not sure how to handle this correctly.

This is the code:

//--- Test direct upload/download ---
//direct transfer doesn't use local file system but volatile memory
//upload volatile memory => FTP server
//download FTP server => volatile memory

//String fileName = "myFile.txt";
char bufferWR[128];
char bufferRD[128];
for (int i = 0; i < 128; ++i) {
bufferWR[i] = 33 + i;
bufferRD[i] = 0;
}

File myFile;
myFile = SD.open("datalog.txt");
if (myFile == false) {
Serial.println("File is not open");
}else {
Serial.println("File is open!");
test("Direct upload from volatile memory to FTP server",
ftp.write(&bufferWR[0], sizeof(bufferWR), myFile.name(), 10000));
}
myFile.close();
Serial.println("File is closed!");

[tryhus]

Your code send the buffer content bufferWR to the FTP server not the content of your SD file datalog.txt.
To send SD file you have to do the following steps :

• Open the file in SD card
• Read it, which means store the data in a buffer
• Send the buffer to the FTP server

You can try with this code :

  File myFile;
  myFile = SD.open("datalog.txt");
  if (myFile == false) {
    Serial.println("File is not open");
  }
  else {
    Serial.println("File is open!");

    //File is open, now we need to read it
    char* fileData = nullptr;
    size_t fileSize = myFile.size();
    if (fileSize > 0) {
      //dynamic allocation of a buffer to store file data
      fileData = new char[fileSize];
      if (fileData == nullptr) {
        Serial.println("Allocation error");
      }
      //file data is stored in the buffer
      else if (myFile.readBytes(fileData, fileSize) != fileSize) {
          Serial.println("Error when reading file");
      }
      //send data to FTP server
      else {
        test("Direct upload from volatile memory to FTP server",
          ftp.write(fileData, fileSize, myFile.name()));
       }
    }

    myFile.close();
    //free allocated memory
    if (fileData != nullptr) {
      delete[] fileData;
    }

    else {
      Serial.println("invalid file size " + String(fileSize));
    }
  }

Notice, if the SD file is very large, the dynamic allocation may fail.
So if you need to send large file, we need to send data to FTP server and read SD file in the same time.

[bastiri]

Thank you very much for your fast answer and your great support. It works fine with files about 20KB. I tried to upload a file with 39KB but it fails. I think its exactly that what you told. The internal buffer seams to be too small for this.

So if you need to send large file, we need to send data to FTP server and read SD file in the same time.

How could that work? if you want to upload files with <1MB

Best regards

[tryhus]

I added functions to stream data to/from FTP server.
This makes it possible to read the SD file at the same time as sending it to the server.
In this way, we no longer have the previous memory limitation.
The following function should allow you to send large file >1MB to the FTP server.
The packetSize argument is the length of data read and send at once.

Increasing the packet size should increase the transfer speed.
You can start with :
sendSDFile("datalog.txt", "datalogInFTPServer.txt", 1024);

bool sendSDFile(const String& SDfileName, const String& FTPfileName, uint16_t packetSize)
{
  bool ok = true;
  int res = 0;
  uint32_t remainingBytes = 0;
  File file;

  char* packet = new char[packetSize];
  if (packet == nullptr) {
    Serial.println("Allocation error");
    return false;
  }

  file = SD.open(SDfileName);
  if (file == false) {
    Serial.println("Failed to open SD file");
    goto Err;
  }

  remainingBytes = file.size();

  //Start FTP transfer in stream mode
  if (ftp.streamInStart(FTPfileName) == false) {
    Serial.println("Failed to start FTP transfer");
    goto Err;
  }

  while (remainingBytes > 0) {
    uint32_t bytes = (remainingBytes >= packetSize) ? packetSize : remainingBytes;

    //read a packet of data
    if (file.readBytes(packet, bytes) != bytes) {
      Serial.println("Failed to read SD file");
      goto Err;
    }

    //send the packet to the FTP server
    if (ftp.streamOut(packet, bytes) == false) {
      Serial.println("Failed to send data to FTP server");
      goto Err;
    }
    remainingBytes -= bytes;
  }

  //wait for the transfer to be completed
  while (res == 0) {
    res = ftp.streamOutReady();
  }

  if (res != 1) {
    Serial.println("Failed to send data to FTP server");
    goto Err;
  }

  goto Out;

Err:
  ok = false;
Out:
  file.close();
  if (packet != nullptr) {
    delete[] packet;
  }
  return ok;
}

[bastiri]

Yeeesss! It works perfectly. I have tried out files with a size up to 1,5MB.
Thank you very much for this great function!!!

Best regards Bastiri

[tryhus]

your are welcome 👍

[salvq]

Just tried your routines and something is wrong and is not working...do you know what is ?

I copied all the src files from your forked section and used above sendSDFile but getting error.

I tried with Total Commander to create or delete the file and working just fine.

Sketch:
sendSDFile("SYSTEM.CSV", "datalogInFTPServer.txt", 1024);
Serial monitor shows this:

18:44:41.897 -> Starting Arduino FTP client.
18:44:57.227 -> Initializing SD card...OK - Connect to FTP server
18:44:58.600 -> OK - Change current remote directory
18:44:58.978 -> drwxr-xr-x 2 85471 1000 4096 Dec 12 18:43 .
18:44:58.978 -> drwxr-xr-x 7 85471 1000 4096 Dec  9 08:42 ..
18:44:58.978 -> OK - Display remote files
18:44:59.767 -> Failed to start FTP transfer

[salvq]

I have added these lines into main script to test other functions and they just work except the SD card.

Where the problem is, can you help to debug or let me know where to dig more ?

Thanks

.......
  test("Non blocking upload from local file system to FTP server",
    uploadFileNonBlocking("downloadedFile", "uploadFile"));

  test("Non blocking upload from SD card file system to FTP server",
    sendSDFile("SYSTEM.CSV", "datalogInFTPServer.txt", 1024));

  test("Display local files",
    FILESYSTEM.remove("downloadedFile"));
.......

19:24:32.073 -> Starting Arduino FTP client.
19:24:46.809 -> Initializing SD card...OK - Connect to FTP server
19:24:48.187 -> OK - Change current remote directory
19:24:48.256 -> OK - Create remote directory
19:24:48.395 -> OK - Rename remote directory
19:24:48.531 -> OK - Local file system write
19:24:48.566 -> OK - Local file system free space
19:24:49.224 -> OK - Upload from local file system to FTP server
19:24:49.977 -> OK - Download from FTP server to local file system
19:24:50.011 -> OK - Local file system read
19:24:50.011 -> OK - Check local file consistency after upload, download then read local file system
19:24:50.045 -> myFile 8
19:24:50.080 -> myFileToLocalMemory 8
19:24:50.080 -> OK - Display local files
19:24:50.114 -> OK - Remove local files
19:24:50.149 -> OK - Display local files
19:24:50.459 -> drwxr-xr-x 3 85471 1000 4096 Dec 12 19:24 .
19:24:50.459 -> drwxr-xr-x 7 85471 1000 4096 Dec  9 08:42 ..
19:24:50.459 -> -rwxr-xr-x 1 85471 1000 9 Dec 12 19:07 downloadFile
19:24:50.459 -> -rwxr-xr-x 1 85471 1000 8 Dec 12 19:24 myFileToServer
19:24:50.459 -> drwxr-xr-x 2 85471 1000 4096 Dec 12 19:24 test2
19:24:50.459 -> OK - Display remote files
19:24:50.596 -> OK - Delete remote directory
19:24:50.735 -> OK - Delete remote file
19:24:50.974 -> drwxr-xr-x 2 85471 1000 4096 Dec 12 19:24 .
19:24:50.974 -> drwxr-xr-x 7 85471 1000 4096 Dec  9 08:42 ..
19:24:50.974 -> -rwxr-xr-x 1 85471 1000 9 Dec 12 19:07 downloadFile
19:24:50.974 -> OK - Display remote files
19:24:51.524 -> Download 0.00 %
19:24:52.210 -> Download 100%
19:24:52.210 -> OK - Non blocking download from FTP server to local file system
19:24:52.280 -> downloadedFile 9
19:24:52.280 -> OK - Display local files
19:24:52.899 -> OK - Non blocking upload from local file system to FTP server
19:24:53.898 -> Failed to start FTP transfer
19:24:53.898 -> ERROR - Non blocking upload from SD card file system to FTP server
19:24:53.933 -> OK - Display local files
19:24:54.035 -> OK - Delete remote file
19:24:58.226 -> OK - Direct upload from volatile memory to FTP server
19:25:00.178 -> OK - Direct download from FTP server to volatile memory
19:25:00.178 -> OK - Direct upload/download tranferred data consistency
19:25:00.315 -> OK - Delete remote file
19:25:15.466 -> OK - Stream data to server

When I tried to specify non existing file on SD, it failed on missing SD file so SD card works...
sendSDFile("SYSTEM1.CSV", "data.txt", 1024));

19:28:42.754 -> OK - Display local files
19:28:43.443 -> OK - Non blocking upload from local file system to FTP server
19:28:43.443 -> Failed to open SD file
19:28:43.443 -> ERROR - Non blocking upload from SD card file system to FTP server
19:28:43.477 -> OK - Display local files
19:28:43.545 -> OK - Delete remote file
19:28:47.707 -> OK - Direct upload from volatile memory to FTP server
19:28:49.250 -> OK - Direct download from FTP server to volatile memory
19:28:49.250 -> OK - Direct upload/download tranferred data consistency
19:28:49.354 -> OK - Delete remote file
19:29:04.510 -> OK - Stream data to server
19:29:17.202 -> OK - Stream data from server
19:29:17.270 -> OK - Delete remote file
19:29:17.610 -> OK - Disconnect to FTP server

[salvq]

@tryhus
Not sure how it works on your end but for me it needs to change below line to make it work.

Just for a reference for anybody else...

from

//Start FTP transfer in stream mode
  if (ftp.streamInStart(FTPfileName) == false) {

to

  //Start FTP transfer in stream mode
  if (ftp.streamOutStart(FTPfileName) == false) {

[salvq]

Hello @tryhus, first of all, thanks for the ftp library, this is very appreciated you have put together FTP processes. I need some help here, please.

I am trying to read downloaded file from modem file system to SD card. I have the files already in file system however trying to transfer them to SD card.

Is it somehow possible to transfer data either directly from FTP to SD card or from modem file system to SD card ?

I am really stuck here :(

[salvq]

If somebody will look for receiving FTP file from FTP to Arduino SD card (I am using Arduino MKR GSM 1400) use this sketch

See input data:

1. deviceId is name of the file to copy from FTP and save to SD card
2. fileSize is size of the file in bytes
3. FTP data:
   #define SECRET_FTP_HOST "ftp.xxxx.xxx" // replace with your FTP host server
   #define SECRET_FTP_USER "AAA" // replace with your FTP user
   #define SECRET_FTP_PASSWORD "BBB-" // replace with your FTP password
   #define SECRET_FTP_PORT 21 // replace with your FTP port (optional)
   #define SECRET_FTP_REMOTE_DIR "/tmp/" // replace with your FTP default remote directory

Call
receiveFtpFile(deviceId, deviceId, 1024, fileSize)

Function

bool receiveFtpFile(const String& sdFirmware, const String& ftpFirmware, uint16_t packetSize, uint32_t remainingBytes)
{
  ftp.connect(SECRET_FTP_HOST, SECRET_FTP_USER, SECRET_FTP_PASSWORD, SECRET_FTP_PORT);
  ftp.cd(SECRET_FTP_REMOTE_DIR);

  int res = 0;
  char* packet = new char[packetSize];

  if (packet == nullptr) {
    //Serial.println("Allocation error");
    // fail routine
    return 0;
  }
  if (SD.exists(sdFirmware.c_str())) {
    SD.remove(sdFirmware.c_str());
  }
  
  file = SD.open(sdFirmware, O_CREAT | O_WRONLY);
  //File dataFile = SD.open(sdFirmware, O_CREAT | O_WRITE);
  if (!file) {
    //Serial.println("Failed to open SD file");
    sdCardStatus = false;
    return 0;
  }
  
  if (ftp.streamInStart(ftpFirmware) == false) {
    //Serial.println("Failed to start FTP transfer");
    // fail routine
    return 0;
  }
  
  while (remainingBytes > 0) {
    uint32_t bytes = (remainingBytes >= packetSize) ? packetSize : remainingBytes;
    ftp.streamIn(packet, bytes);
    file.write(packet, bytes);
    remainingBytes -= bytes;
  }


  while (res == 0) {
    res = ftp.streamInReady();
  }

  if (res != 1) {
    //Serial.println("Failed to send data to FTP server");
    // fail routine
    return 0;
  }

  if (packet != nullptr) {
    delete[] packet;
  }

  file.flush();
  file.close();
  ftp.disconnect();
  
  return 1;
}

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[JLuc26]

Hi, several years later... ;-)
I'm trying to use your FTP library on a Arduino MKR NB1500...
I already compiled your code with GSM1400 but i must used a Modem.h an Modem.cpp elsewhere!
In NB1500, Modem.cpp if provided, even with an NBmodem, but without ::escapesequence and ::read().
2 weeks that i'am trying to upload a file on 2 differents FTP server, without success...
I hope someone can help me.

I can connect and disconnect to server FTP, open the direct link mode (seen passed code), (read 1 or several packets of SDcard file, or made a memory file), but write bytes on FTP is perhaps compromised? I don't know, seen that's there is no respond code of modem...
Escapesequence gave me a wrong code... and disconnect too...

I had good codes if i just connect and disconnect to the FTP server.

I can also makedir, delete dir on FTP...

FileZilla show me results on dir, but nothing on files.

I traced and checked enteries in GSMFTP.cpp, Modem.cpp, with a lot of Serial.println(" ")...

Can you help me? @tryhus ? @salvq ?