Use cpor id everywhere?

[Menduist]

Idea I had during the Dagger meeting, writing it down to spark some discussions:

Currently we have two addressing system:

• The cpor ID, which uniquely identifies a chunk of a dataset, using this tuple: (Dataset Public Key, Block indice in the dataset) (+ some data like filename which we can ignore for this discussion)
• The content-id, which is the hash of the block

The cpor ID is used during validation, content-id during retrieval. So the question is, do we need to keep the content-id? We could take our current model, and replace every content id by a cpor ID, for instance "publickey:blockindice" (in the DHT, for block exchange, etc)
To verify that a block is valid instead of hashing it, you can use the publickey, blockindice & tags

Pros:

• Only have one addressing system instead of two
• Easy to know which dataset a block is part of
• Easy to know the list of block in a dataset (it's just [publickey:0 ... publickey:blockcount])

Cons:

• We loose block deduplication in the network
• The peer who originally signed the file could reuse the key to do "bad stuff" with it (mainly upload a second file with the same key). So, you have to trust the file uploader. Is this bad?

[dryajov]

The main disadvantage of this kind of approach is the lack of content addressability, i.e. it isn't anymore possible to guarantee that the content being pointed to is the actual content being retrieved.

There doesn't seem to be a good way to reconcile hash based content addressing with CPoR tags and aggregated blocks. It would in fact be desirable to unify the two systems, but it seems, at least right now, that the two pursue two different objectives - content addressing in the case of hashes vs cryptographically verifiable aggregation of both data and tags in the case of CPoR.

[Menduist]

I don't fully agree.

Let's say you want to download a file using CIDs. You have to find the CIDs out-of-band, and trust that the CID is the hash of the file that you want to download. Except if you already have the file, then you can hash it and check, but that would only be relevant when you are the uploader
And if you want to download a file using CIDs, you're forced to download blocks in a certain order (top-down) to be sure that all blocks are part of the dataset, and which blocks to download

With a system based on cpor public key instead, you have to trust that the public key you're using to retrieve the file is the public key of the file you want to download, so that doesn't change anything on that front.
But, as a bonus, you can download in any order, and check using cpor that any block you downloaded is part of the dataset, and at his right place

[dryajov]

Let's say you want to download a file using CIDs. You have to find the CIDs out-of-band, and trust that the CID is the hash of the file that you want to download. Except if you already have the file, then you can hash it and check, but that would only be relevant when you are the uploader

Thats an orthogonal problem, the assumption at least at that stage, is that the hash being download has been properly vetted to be the one from the desired dataset.

The problem I was pointing out is a bit different. With content addressing, there is deterministic way of always coming up with the same identifier for any piece of data, no matter who or where it was generated, i.e. the identifier is only subject to the actual content and nothing else, adding a public key into the mix changes that assumption.

And if you want to download a file using CIDs, you're forced to download blocks in a certain order (top-down) to be sure that all blocks are part of the dataset, and which blocks to download

Not really, thats why you use an authenticated data-structure to generate the top level hash, when downloading a chunk, you would also expected to provide accompanying proofs that the chunk is really part of the dataset - for example, you'd get internal nodes for a merklelized dataset, so order doesn't matter.

With a system based on cpor public key instead, you have to trust that the public key you're using to retrieve the file is the public key of the file you want to download, so that doesn't change anything on that front.

See the my response to the first paragraph, but it seems like the addition of a public key to the mix, breaks the premise of content addressability, as now, depending on what public key is being used, you end up with different identifiers for the same content. This is a problem not only because you loose de-duplication, but because there is now another level of indirection and hence the content -> id and id -> content mapping is lost as you now have pk + content -> id and id -> pk + content and you have to trust that both the PK as well as content are the correct ones for a given id, moreover you cannot anymore generate the id yourself, once you have the original data.

It would be really nice if we can use the same system for both, but to me it feels like the particular systems we've chosen (for now), based on homomorphic tags and BLS signatures (Compact Proofs of Retrievability - S&W) doesn't directly allow us to use it as a content addressing mechanism, which is pity and we should look for alternatives, but this is something we don't have to solve right now. Even if the two systems are somewhat duplicating each others functionality, this isn't a showstopper and we can address this later on. But the bottom line is that it doesn't seem like the PoR mechanism we're using give us content addressability out of the box and that is a showstopper.