optimize countFilesInFolder and getFoldersInFolder regarding memory c…

[ifx-cw]

…onsumption.

We had the problem that we had a lot of "folders" in an s3 repository on the root level (about 120), each containing a lot fo files.
This lead to the file list running into the memory limit (for the tree view as well as for the "folder" view). We found that the counting functions mainly lead to the problem, so we optimized them as far as possible (as the main "problem" of course lies within the way S3 works in this case).
The original patch was done in an older version of the driver (for Typo3 7.6) so the recursive call in getFoldersInFolder is untested so far.

[weakbit]

Hey @ifx-cw thanks for Contributing.
This is a long time ago, and i want to ask if this Problem still occurs in current versions of this extensions and you still need to apply your Patch? This would help me to get a glue if i spent time analyzing this.

[ifx-cw]

Hi @weakbit
We do not maintain the project where we needed to make this adjustment anymore. However, I know that at least until version 1.11.1 the fix still had to be applied to get the file list in Typo to work. So, I guess the problem should still be present as I don't see any major changes in getFilesInFolder which is used for counting the files.

[weakbit]

@ifx-cw
We moved to the TYPO3 caching framework some time ago.
The merge of #143 will make redis cachebackend work. There would be multiple was to work with the memory problem then, as Redis can be configured to throw away entries if the memory usage is too high.

It also supports compression and makes it easy to distribute the cache to an other machine.

If the data is changed from outside of TYPO3 the cache lifetime should be set to 1 second as long as there is no external webhook or something to control the caches.

Less memory usage is done by the null cache backend at costs of request count and time.

So to bring things together, i think the problem here was that every request is transient saved to memory while the 309e478 uses s3client directly.

I will keep the PR open for now for further review, thank you for your contribution.