Inaccurate encoding detection output in debug mode

[j9t]

Operating system

macOS Sonoma 14.6.1 (23G93)

Eleventy

3.0.0

Describe the bug

Running all Eleventy operations in debug mode (DEBUG=Eleventy* npx @11ty/eleventy), I noticed inaccurate encoding output (in this case of a project with 10k+ files, more than 10,000 output messages).

Example:

  Eleventy:TemplateContent 'ISO-8859-1' encoding detected on './tools/feed/htaccess.njk'. +8ms
  Eleventy:TemplateContent 'ISO-8859-1' encoding detected on './tools/feed/rss.njk'. +1ms
  Eleventy:TemplateContent 'ISO-8859-1' encoding detected on './_views/default.njk'. +18ms
…
  Eleventy:TemplateContent 'ISO-8859-1' encoding detected on './tools/exploration/typography/type-scale-generator-jeremy-church.md'. +1ms
  Eleventy:TemplateContent 'ISO-8859-1' encoding detected on './tools/exploration/typography/type-scale-generator-tim-brown.md'. +0ms
  Eleventy:TemplateContent 'ISO-8859-1' encoding detected on './tools/exploration/typography/web-font-generator.md'. +0ms

The problem is that the files are all in UTF-8 (encoding for everything in my env). I took some samples to confirm, and every file I checked was UTF-8. WebStorm indicates this conveniently:

Interestingly, it doesn’t seem Eleventy would generally fail with encoding recognition, as these examples show:

  Eleventy:TemplateContent 'UTF-8' encoding detected on './pages/tags.njk'. +52ms
  Eleventy:TemplateContent 'UTF-8' encoding detected on './pages/tools.njk'. +32ms
  Eleventy:TemplateContent 'UTF-8' encoding detected on './pages/tag/rss.njk'. +1ms

Filing this to improve encoding detection, but also to help in case this could be something to improve Eleventy performance with.

(Note: I’d say I have sold-enough knowledge of Unicode, but can’t rule out I’m missing something here that would explain why Eleventy would misidentify the encoding.)

Reproduction steps

Can give access to the currently private repo in question:

1. Run Eleventy in debug mode, e.g. DEBUG=Eleventy* npx @11ty/eleventy --incremental
2. Pay attention to “encoding detected” messages
3. In project exclusively using UTF-8, Eleventy would detect ISO-8559-1

Expected behavior

3. Eleventy would detect the correct encoding

Reproduction URL

No response

Screenshots

No response