v1.4.0

This is the fourth minor release of sqsh-tools. This release does not change anything about the command line utils, never the less, there are lots of changes behind the scene. There are new user facing APIs in libsqsh and various refactorings throughout all of the application. Most notable changes are:

sqsh-ls

If a file contains non-printable characters, they will get escaped and printed as \n, \r, \t, ...

sqsh-unpack

sqsh-unpack was moved to the traversal API. This harmonizes the source with sqsh-ls and allows to increase the test coverage of the traversal API.

Also a potential TOCTOU issue has been fixed.

libsqsh

easy traversal API

This releases exposes sqsh_easy_tree_traversal() that returns an array of all files within this path.

support for custom mappers

libsqsh was always quite flexible when in comes to the transport. We support mounting files via http for example. With the new release we expose this API to the users, allowing them to implement their own memory efficient transport. This was implemented to integrate well with radare2

rewrite of SqshTreeTraversal

This release bring a new a rewrite of SqshTreeTraversal. This should make the source more read- and maintainable. The rewrite also brings a new methods:

• sqsh_tree_traversal_set_max_depth() allows to set a maximal traversal depth when creating a tree.

new features of the SqshFile API

Based on the learnings from integrating libsqsh into radare2, we revised the SqshFile API and added a few more functions:

• sqsh_lopen() opens a path without trying to resolve the symlink if it encounters one.
• sqsh_file_symlink_resolve() and sqsh_file_symlink_resolve_all() if the file is a symlink, these function allow to resolve them in one level (sqsh_file_symlink_resolve()) or recursively (sqsh_file_symlink_resolve_all()). libsqsh performs recursion detection, so sqsh_file_symlink_resolve_all() should be safe to be called even on untrusted squashfs archives.

Deprecation

No functionality was deprecated in this release.

Bugs

A bug with the traversal API that sometimes misses to report trailing SQSH_TREE_TRAVERSAL_STATE_DIRECTORY_END events has been fixed.

Internals

This release contains a few refactorings. Notably when an archive is missing an export table, libsqsh tracks the inode number to inode reference (that represents the physical location of an inode inside of the archive) with its own data structures. Before this release, this was a flat array which allowed malicious archives to allocate up to 4GB of data which could lead to crashes on 32bit systems. After this release, the collector uses a radix map as a data backend, which allocates data segments as they are written.

Outlook

The next releases will again focus on the write support of libsqsh. This work has mostly stalled throughout the implementation of this release.

Furthermore I plan to move libcurl support from the library to a subproject and move it into its own repository. Starting with the next major release, curl support won't be a feature of libsqsh itself any more, but provided as a third party library.

libsqsh currently uses hash maps to track the different segments of the archive. This has the downside of unpredictable runtime characteristics and can be used to slow libsqsh down and increase the memory consumption with well crafted archives. To prevent this, libsqsh will replace these hash maps with radix trees, which have predicable memory layouts.

What's Changed

• traversal: rewrite by @Gottox in #198
• Improve/macros by @Gottox in #199
• inode_map: use a vtable instead of branching. by @Gottox in #200
• inode_map: initialise atomics by @Gottox in #201
• inode_map: replace atomics with a mutex. by @Gottox in #202
• ls: escape output filename paths by @Gottox in #203
• write support for uncompressed metablocks by @Gottox in #204
• Chores/license 2024 by @Gottox in #205
• subprojects: update cextras by @Gottox in #206
• fix build on darwin by @Gottox in #207
• utils: fix _log2 return types for platforms where int isn't 32bit by @Gottox in #208
• compression_options: fix build with gcc-13 by @Gottox in #210
• common: the build artifact was renamed to give more context by @Gottox in #212
• introduce utest.h as a replacement for cextras testlib by @Gottox in #213
• test: fix wextra-semi when using utest by @Gottox in #214
• replace testlib with utest by @Gottox in #215
• test: fix -Wextra-semi by @Gottox in #216
• test: move cpp-test to utest.h by @Gottox in #217
• fix clang build on macos by @Gottox in #218
• Fix/alpine container by @Gottox in #220
• unpack: fix dangling file stream by @Gottox in #221
• directory_iterator: fix sonarqube reports. by @Gottox in #222
• extract: disconnect zlib return value from out value by @Gottox in #219
• Improve/remove testlib by @Gottox in #223
• chores: replace Makefile by .Mk.yaml by @Gottox in #225
• Improve/tree traversal documentation by @Gottox in #227
• common: update Dockerfile to Alpine 3.20 by @Gottox in #229
• Improve/release pipeline by @Gottox in #230
• meson.build: do not print warning when sqsh-tools is built with zstd … by @Gottox in #231
• README.md: add reference to backhand. by @Gottox in #232
• mapper: add official support for custom mappers by @Gottox in #228
• mmap_mapper: fix mac/openbsd build by @Gottox in #233
• mapper: make SqshMemoryMapperImpl a public API. by @Gottox in #234
• file: use the generic constructurs/destructors for open and close by @Gottox in #235
• file: add support for sqsh_file_symlink_resolve by @Gottox in #236
• common: do not export sqsh__log2 by @Gottox in #238
• .Mk.yaml: configure to be compatible with abi-dumper by @Gottox in #239
• Improve/path resolution cleanup by @Gottox in #237
• Add/easy traversal by @Gottox in #226
• Fix/resolve all export by @Gottox in #240
• test: reactivate xattr integration tests by @Gottox in #241
• Fix/obsd test by @Gottox in #243
• Better testing of mmap, move posix specific functions to own module. by @Gottox in #242
• traversal: rewrite implementation by @Gottox in #244
• chores: update cextras to fix MacOS build issue by @Gottox in #245
• Fix/openbsd build by @Gottox in #246
• .github: upload failing inputs from fuzzer runs by @Gottox in #248
• common: use CX_*_IMPL macros instead of defining our own. by @Gottox in #249
• chores: update cextras to fix mac/openbsd build by @Gottox in #250
• file: remove auto search of directory inodes by @Gottox in #251
• test: move inputs to their original names. by @Gottox in #252
• inode_map: fix deadlock when the inode map encounters inconsistencies by @Gottox in #253
• Improve/remove magic dir node number by @Gottox in #247
• traversal: push element to the stack as soon as possible by @Gottox in #254
• v1.4.0 by @Gottox in #255
• file: Do not export dir_inode getters/setters by @Gottox in #256
• .github: fix ci pipeline for releases by @Gottox in #257

Full Changelog: v1.3.0...v1.4.0