Implement S3 data storage.

[s3rius]

Create S3 Data storage to be able to store files on S3.

The main idea is to create many file chunks 5MB each and concatenate them after upload is finished.

[s3rius]

It's quite a challenge. I won't do it for now. For all people who want to work with S3 as a primary storage, you can use s3fs-fuse to mount s3 in a directory.

[Kaelten]

Would it be more straightforward if the implementation used the volume storage as scratch space but then pushed to s3 as part of finishing the upload?

[s3rius]

Hi @Kaelten, it took a long time, but this feature is finally coming in the next release! Currently we will only have a hybrid-s3 which works exactly as you said. The fully operating S3 storage would be implemented a little bit later.

Related PR: #108.

[s3rius]

Hybrid-s3 storage is released in 0.6.0. S3 storage without storing files locally will come later. Leaving this feature open until new storage is released.

[genofire]

For an S3 only implementation, would not be S3 Multipart upload the solution?
https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html
(So concat should not be needed, that makes S3 for you)

[s3rius]

Hi, @genofire. Indeed, that's exactly it. But the tricky part here is that minimum size for a chunk is 5Mb. For some systems it just don't work. That's why I implemented hybrid storage first. Hybrid storage doesn't have such a restriction.

[genofire]

Does S3File in chicon helps?

[s3rius]

Hi, @genofire. Yes, I guess it may work. I need to try it.

I would have no time in 1-2 weeks. If you feel that you might take this over, please do. If you don't, I'll take it as soon as I have more time.

Thank you for pointing this out.