Update from .tar.gz with an additional file - gzProcessTarBuffer failed reading

[r-zlotorzynski]

Hello

I'm using the TarGzUnpacker class and the tarGzStreamUpdater method.
The .tar.gz file that I store in SPIFFS contains the spiffs.bin update files, firmware.bin and additionally I have to include the package.json file (for my own purposes). Without a package.json file, the update works fine. However, if the file is attached, the firmware.bin batch is updated correctly. The spiffs.bin batch is skipped.

I tried to rename the "package.json" file to the name "zpackage.json" (to change the order of files in the package - but in the final version the file name MUST be package.json). This seems to solve the problem. However, there are errors and warnings in the debug.

I also tried to use the setTarIncludeFilter method, but that also did not improve.

Is there any other way to update in my case (i.e. with an additional package.json file)?

Thanks in advance for all the tips.
Greetings

[tobozo]

Hey @r-zlotorzynski thanks for your feedback

In my opinion, bundling json+firmware+spiffs in a tar.gz file this is the most efficient way to bundle an update package.

See tarHeaderUpdateCallBack() for more info on why the json file is ignored, this will make sense on why using setTarIncludeFilter() did not produce any effect. Also check for tarBlockIsUpdateData, tarSkipThisEntryIn and tarSkipThisEntryOut if you want to experiment.

This highlights a limitation of tarGzStreamUpdater where only two file types are expected, and unintended behaviours may appear when more than two files are found in the archive.
There is a feature opportunity in handling non-update files though, but it raises the chicken-and-egg filesystem issue and requires some logic controls (e.g. json parsing) that are way outside the scope of this library.
The implementation of setTarIncludeFilter() was an attempt to provide these controls although it seems incomplete in the Update context.

The .tar.gz file that I store in SPIFFS contains the spiffs.bin update files

⚠️ Calling tarGzStreamUpdater from a .tar.gz file saved on the SPIFFS partition is limited to firmware only, if you need to update both firmware and SPIFFS partitions, the stream source must be SD, HTTP or PsRamFS (otherwise you're sawing the branch you're sitting on and Update errors will occur).

Now there are probably some bugs involved here too, could you provide the debug log, along with a basic hello-world example of a .tar.gz file similar to what you've been using in this attempt so I can reproduce the issue?

Here are the feature opportunities that come up with your issue:

• Enabling setTarIncludeFilter in tarGzStreamUpdater, depends on:
  • Attaching additional custom tarCallbacks (e.g. json streamreader)
  • Hot-swapping custom tarCallbacks with default tarCallbacks when a condition is met (e.g. filename=="package.json")

Meanwhile here are some ideas:

• If your ESP32 has psram and your .tar.gz file is smaller than 4MB, try PsRamFS as a temporary volatile filesystem
• You'll probably want to use tar -T option rather than renaming to solve the what-file-comes-first problem (this creates upstream complexity though)
• See how ESP32-FOTA handles the situation, and why joining forces as highlighted by @scubachristopher would be a good compromise

[r-zlotorzynski]

Thank you for the quick reply. Soon I will get to know the information in more detail.

At the moment I will only correct one piece of information (my mistake - sorry).

"The .tar.gz file that I store in SPIFFS contains the spiffs.bin update files"

The update file .tar.gz is in a different partition (let's call it spiffs_2). However, only the "spiffs_1" partition is updated. And it works very well (but only without the package.json file).

Greetings