Skip to content

Commit

Permalink
Changed the description about invalid UTF-16 paths
Browse files Browse the repository at this point in the history
* Windows used to report unassigned Unicode code points as invalid
  and now converts them to UTF-16, so the example in the comment
  was no longer valid. The underlying problem is still there,
  however, and a path on Windows may have invalid UTF-16 sequences,
  which are handled by the code around the comment.
  • Loading branch information
StoneStepsInc committed Oct 27, 2023
1 parent 28e0c6b commit afa115d
Showing 1 changed file with 6 additions and 6 deletions.
12 changes: 6 additions & 6 deletions src/file_tracker.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -749,10 +749,10 @@ void file_tracker_t::run(void)
progress_info.failed_files++;

//
// Windows may store file paths with invalid UCS-2 characters,
// Windows may store file paths with invalid UTF-16 characters,
// which fail to convert to UTF-8 paths. For example, a file
// path may have a code point `\x1F7FB` in it, which doesn't
// have any Unicode character assigned and cannot be converted
// path may have a UCS-2 code that represents and incomplete
// surrogate pair, such as `\xD83D`, which cannot be converted
// to a UTF-8 string.
//
// Use a crude ASCII conversion and replace all non-ASCII
Expand All @@ -764,9 +764,9 @@ void file_tracker_t::run(void)
std::u16string filepath_u16 = dir_entry.value().path().u16string();
std::transform(filepath_u16.begin(), filepath_u16.end(),
std::back_inserter(filepath),
[] (char16_t chr) -> char {return (chr >= ' ' && chr < '\x7f' ? static_cast<char>(chr) : '?');});
[] (char16_t chr) -> char {return (chr >= u' ' && chr < u'\x7f' ? static_cast<char>(chr) : '?');});

print_stream.error("Cannot process a path with invalid UTF-16 characters (%s) %s", error.what(), filepath.c_str());
print_stream.error("Cannot convert path to UTF-8 (%s) %s", error.what(), filepath.c_str());
files_lock.lock();
continue;
}
Expand Down Expand Up @@ -996,7 +996,7 @@ void file_tracker_t::run(void)
std::u16string filepath_u16 = dir_entry.value().path().u16string();
std::transform(filepath_u16.begin(), filepath_u16.end(),
std::back_inserter(filepath),
[] (char16_t chr) -> char {return (chr >= ' ' && chr < '\x7f' ? static_cast<char>(chr) : '?');});
[] (char16_t chr) -> char {return (chr >= u' ' && chr < u'\x7f' ? static_cast<char>(chr) : '?');});
}

print_stream.error("Cannot process file %s (%s)", filepath.c_str(), error.what());
Expand Down

0 comments on commit afa115d

Please sign in to comment.