v0.10.1

Bugfixes

• Allow building utils_test against a non-compatible, system-installed version of gtest. This is a common issue when trying to integrate dwarfs into a package manager, as these generally disallow fetching external dependencies at build time.
• dwarfsck was always reporting a block size of 1 byte rather than the actual block size of the image.
• DWARFS_HAVE_LIBBROTLI was not set correctly in the config file, causing build errors if the library was built without brotli.
• Several small fixes for building with Homebrew.

Full Changelog: v0.10.0...v0.10.1

SHA-256 Checksums

53bb766f3a22f019c4bac7cbf995376cb4f3f0ad69e4793471af11c954185227  dwarfs-0.10.1-Linux-aarch64-clang-stacktrace.tar.xz
f272f667649d71ec7d29d6822ad4198e13a33e997d722f74f2bca23b239de72f  dwarfs-0.10.1-Linux-aarch64-clang.tar.xz
671ce264938ab4cacc8af0aabcacb1ecfffa01284b4959441e921264ae19b47e  dwarfs-0.10.1-Linux-x86_64-clang-stacktrace.tar.xz
84894bf6a26cac2eb2c8d43d6fccf1ece7665c4c15050cec494d09199bd8310e  dwarfs-0.10.1-Linux-x86_64-clang.tar.xz
4041ed9aa19e03f44dbe69b470f31423a3c358bcd07e78230311b859629785b6  dwarfs-0.10.1-Windows-AMD64.7z
db785e0e0f257fa4363d90153db34127add4552791a72998b30ded787840d039  dwarfs-0.10.1.tar.xz
3e003c9a5fbf31b75548c11a2c2c1958f606ce2c2022db4baa6d62b80201c76d  dwarfs-universal-0.10.1-Linux-aarch64-clang
44ad0a3f2d89e373b0279d1db7c19aeca46879972a2db226e31ec7ebe8ff103e  dwarfs-universal-0.10.1-Linux-aarch64-clang-stacktrace
18f99297c7425bd1bea87d47a2046bfc7e00fe9cc566f024e631ed39a6bb1913  dwarfs-universal-0.10.1-Linux-x86_64-clang
c60821be4a248be2feb54905b5bb6c5cd323014bcb7107f0d586ba7f70deb488  dwarfs-universal-0.10.1-Linux-x86_64-clang-stacktrace
768d013d55cd030c1fbabd35ad522648581c79435da4994cc39de75b3a7eda30  dwarfs-universal-0.10.1-Windows-AMD64.exe

dwarfs-0.10.0

This release doesn't bring a lot of new features or bugfixes to the command-line tools. However, large parts of the code base have been refactored (in more than 300 commits since the last release) and a couple of long-standing issues have been resolved:

• It is finally possible to properly install the libraries that implement most of DwarFS. These libraries come with all necessary headers and a CMake config script so you can start building your own tools.

• Shared library builds are now explicitly supported. These have caused some trouble in the past, but the problematic code has been refactored to avoid issue like missing compression / categorizer code.

• The code now builds about twice as fast. This is due to shipping generated code in the source tarball and removing a lot of dependencies into the folly library.

• It is now possible to do modular builds, i.e. the library, tools and FUSE driver can be built independently.

Bugfixes

• Fixed a race condition identified by ThreadSanitizer in the root node name processing.

• The terminal abstraction code did not check any errors when trying to determine the terminal width, leading to a random terminal width value. This caused the manual page tests to occasionally crash.

• Fixed some flaky tests, e.g. unmounting the FUSE driver on macOS.

Features

• Two sets of universal binaries and binary tarballs are provided for Linux platforms: one without any debug symbols, the other with minimal debug symbols and support for stack traces. For the universal binary, only the version without debug symbols will be UPX-compressed, as the stack trace functionality doesn't work with a compressed binary.

• Symbolic links to the universal binary may now be suffixed with a version (i.e. any part of the name starting with - and followed by a digit will be ignored, e.g. the symlink could be mkdwarfs-0.10 and it would be treated as mkdwarfs).

• Introduced support for extended attributes on Windows, including a new utility for cross-platform xattr manipulation (pxattr, for portable xattr).

• Enhanced file system API, adding error-code based and exception-safe versions for getattr, access, and similar functions.

• Filter rules now consistently use Unix path separators, even for the root path component. Addresses a comment in github discussion #228.

Improvements

• Extensive refactoring to improve code modularity, maintainability and to provide proper libraries. The library code has been moved to different namespaces to make it easier to understand the role of different components (e.g. reader, writer, extractor).

• Replaced all folly library dependencies in the public DwarFS library interface with alternatives from libraries like e.g. boost or nlohmann::json which are more broadly available. folly and fbthrift are still used as implementation details, but no longer leak into the public library interfaces.

• A much smaller subset of folly is now used in DwarFS and only the necessary components are built, significantly reducing the number of compilation units when building DwarFS.

• It is now possible to do modular builds in addition to the default monolithic build, i.e. you can build and install just the DwarFS libraries and later build/install the tools (mkdwarfs, ...) and/or the FUSE driver against these libraries. This is particularly useful for packaging (e.g. in Homebrew, which has removed all FUSE support from the core formulae).

• Shared library builds are now explicitly supported. This fixes issues such as github #184.

• The source tarball now contains all auto-generated code, e.g. manual pages or generated thrift code. This reduces the number of build-time dependencies (e.g. ronn or mistletoe are no longer required) and significantly reduces the build steps (it is no longer necessary to build the thrift compiler). The build is now roughly twice as fast as in the 0.9.x releases.

• The parallel-hashmap, xxHash and zstd submodules have been removed from the git repo and are no longer added to the source tarball. Both xxHash and zstd are now widely available. If a suitable version of parallel-hashmap is found on the system, this will be used, otherwise it will be fetched during the build. Being a header-only library and only used internally, there's no need for it to be installed.

• A lot of GCC warnings have been fixed and upstreamed to folly / fbthrift.

New Contributors

• @mindfocus made their first contribution in #222

Full Changelog: v0.9.10...v0.10.0

SHA-256 Checksums

52045506839249823a9b05b711f89e3fd18a729b264149fbe74f557bd8f16b81  dwarfs-0.10.0-Linux-aarch64-clang-stacktrace.tar.xz
b12a33650694baa6f546ac18768b8a7b2e1dd90c1c674d26bec7f5149a285537  dwarfs-0.10.0-Linux-aarch64-clang.tar.xz
8593eaecc1bed4f570626ff07b0e56a7f6a025c3a5849f4b0056006ab9801393  dwarfs-0.10.0-Linux-x86_64-clang-stacktrace.tar.xz
53f02da7ff7fe484b3bb91e6532987c3fb59db28131309ccbb46d2c2c9cec9ce  dwarfs-0.10.0-Linux-x86_64-clang.tar.xz
f24ee132cf0c77b0c94e49b3d8108fdf7e0ff965031bbc34aa51e3c2e066fa43  dwarfs-0.10.0-Windows-AMD64.7z
c01ae59d4662e4f027a7c8a5934b7aebe6edc4f7affd836fa73e6d861d18bf35  dwarfs-0.10.0.tar.xz
573656e33a171017046a32500b9ebe50ebd117df16212e6ef8f8d4b034618210  dwarfs-system-gtest.patch
149c9518d5229bf6dd07b5c39df02ed5cbc4bddbf9bf54f6fa5617d76a594b1c  dwarfs-universal-0.10.0-Linux-aarch64-clang
bfaad42a15c687fdaf502283792322692e479c563c56e8734efd9a298956d716  dwarfs-universal-0.10.0-Linux-aarch64-clang-stacktrace
5f3401cdd7267b6eec6f758f902d7d874534cc77b5be7233944832992e2c0081  dwarfs-universal-0.10.0-Linux-x86_64-clang
307f973fd7da525c5f06f6090b4a4f7a0bea041d785dcb718fb226f6b15f942f  dwarfs-universal-0.10.0-Linux-x86_64-clang-stacktrace
99cf71972156ec6f5d88ab0331709ecdc1e80f571582fce95c89b694d3be258b  dwarfs-universal-0.10.0-Windows-AMD64.exe

dwarfs-0.9.10

Bugfixes

• When cloning LZMA compressor objects, the LZMA filter options of the cloned instance would still point to an options object in the original instance. This could lead to LZMA errors when initializing a new compressor. Fixes github #224.
• Fetch range-v3 if no suitable version is found. Fixes github #221.
• Filter rules did not work correctly when input is root dir
• duf reports odd sizes due to using bsize instead of frsize

dwarfs-0.9.9

Bugfixes

• A bug introduced by an optimization to skip hashing of large files if they already differ in the first 4 KiB could, under rare circumstances, lead to an unexpected "inode has no file" exception after the scanning phase. This bug did not cause any file system inconsistency issues; mkdwarfs either crashes with the exception, or its output will be correct. Fixes github #217 (see also for more details).

Features

• A sequential access detector was added to the block cache, which can trigger a prefetch of blocks assumed to be read in the future. This improves sequential read throughput roughly by a factor of two. Random access should typically be unaffected. Can be configured / disabled using -o seq_detector.

• Added tracing support in FUSE driver and dwarfsextract, which allows simple performance analysis using chrome://tracing. Traces can be enabled using -o perfmon_trace and --perfmon-trace.

• Added performance monitoring and tracing support for the block cache.

Performance

• Significantly improved the speed of dwarfsck --checksum.

dwarfs-0.9.8

Bugfixes

• Build custom version of libcrypto to link with the release binaries in order for them to run properly on FIPS-enabled setups. Fixes github #210.

• When mounting a DwarFS image on macOS and viewing the volume in Finder, only the directories were shown, but no files. The root cause was that a non-existent extended attribute is reported via a different error code in macOS (ENOATTR) compared to Linux (ENODATA) and the wrong error code was returned for certain Finder-related attributes. Fixes github #211.

• macOS builds using jemalloc were crashing when calling mallctl("version", ...). The root cause of the crash is still unclear, but as a workaround, the jemalloc version is compiled in from a preprocessor constant rather than using mallctl.

dwarfs-0.9.7

Bugfixes

• Handle root uid correctly in access() implementation. Fixes github #204.

Features

• Show and track library dependencies. Dependencies will be displayed in the command line help; they will also be tracked in the history metadata of a DwarFS image. See also github #207.

Documentation

• Describe nilsimsa ordering algorithm more accurately.

Performance

• Reorder branches to improve ricepp speed with real world data.
• Some tweaks to improve segmenter speed.

dwarfs-0.9.6

UPDATE: All prebuilt binaries available for download here are unaffected by the xz/liblzma backdoor issue. The binaries are statically linked against liblzma-5.4.1, which does not include the backdoor, so even on systems with an affected version of liblzma, the statically linked binaries are safe. #207 has more details. Thanks to @bionade24 for bringing this to my attention.

Bugfixes

• Add workaround for new glog release breaking the folly build. Fixes github #201.

Performance

• Improve ricepp decoding speed by about 25% on x86 and arm, and up to 100% on Windows. Also improve encoding speed on Windows by 25%.

dwarfs-0.9.5

Bugfixes

• Windows path handling was wrong and didn't work properly for e.g. network shares. This is hopefully fixed for all tools now.

dwarfs-0.9.4

Bugfixes

• (fix) Prevent installation of ricepp headers/libs. Fixes github #195.

• (fix) Don't fetch googletest in ricepp build if the targets are already available. Fixes github #194.

Features

• Added blocksize option to the FUSE driver, which allows the st_blksize value to be configured for the mounted file system. This can be used to optimize throughput.

• Added experimental readahead option to the FUSE driver. This can potentially increase throughput when performing sequential reads.

dwarfs-0.9.3

Bugfixes

• v0.8.0 removed the implementation of the null decompressor under the assumption that it was no longer used; it was, however, still used when recompressing an image with null-compressed blocks. The change to remove the implementation was reverted and a new test case was added. Fixes github #193.

Performance

• Some more ricepp compression speed improvements. Also, the universal binaries for x86_64 now automatically choose a ricepp version based on CPU capabilities.

• For Windows, there's an experimental -ricepp package/binary. This contains a "hybrid" build where the ricepp library was built using clang and everything else using cl. This binary offers significantly faster ricepp compression. Decompression speeds are similar to the regular package/binary. If you don't care about compressing large amounts of FITS files on Windows, just stick to the regular package/binary.