Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for working with binary indexes. #80

Open
wants to merge 15 commits into
base: master
Choose a base branch
from

Conversation

aalekhpatel07
Copy link

This is an attempt to provide first-class support for Faiss binary indexes consuming the C API (along with a couple of utility functions introduced in facebookresearch/faiss#3318)

Note: #77 got merged so this is in lieu of #79

This should be complete in terms of functionality (at least for a default BinaryIndexImpl) so I'm hoping to learn of any major concerns before continuing down this path and adding the IndexBinaryFlat, IndexBinaryHNSW, etc, concrete impls.

aalekhpatel07 and others added 7 commits March 27, 2024 12:10
…macros for binary indexes.

Signed-off-by: Aalekh Patel <aalekh.gwpeck.7998@icloud.com>
Signed-off-by: Aalekh Patel <aalekh.gwpeck.7998@icloud.com>
Signed-off-by: Aalekh Patel <aalekh.gwpeck.7998@icloud.com>
Signed-off-by: Aalekh Patel <aalekh.gwpeck.7998@icloud.com>
Signed-off-by: Aalekh Patel <aalekh.gwpeck.7998@icloud.com>
@aalekhpatel07
Copy link
Author

Seems like the upstream changes for cloning and creating binary indexes via the C api have landed.

Please let me know if I can contribute towards any other blockers for landing this.

Thanks!

Signed-off-by: Aalekh Patel <aalekh.gwpeck.7998@icloud.com>
Signed-off-by: Aalekh Patel <aalekh.gwpeck.7998@icloud.com>
…y indexes implemented yet.

Signed-off-by: Aalekh Patel <aalekh.gwpeck.7998@icloud.com>
Signed-off-by: Aalekh Patel <aalekh.gwpeck.7998@icloud.com>
Signed-off-by: Aalekh Patel <aalekh.gwpeck.7998@icloud.com>
Signed-off-by: Aalekh Patel <aalekh.gwpeck.7998@icloud.com>
…vectors.

Signed-off-by: Aalekh Patel <aalekh.gwpeck.7998@icloud.com>
@aalekhpatel07
Copy link
Author

So I enabled the CI for these changes and if we bumped upstream to use the latest revision, then the CI test suite continues to pass.

I'm leaning towards waiting for a new release of upstream so we can build against v1.8.1 and hopefully the binary index clone/read changes would be included in that release.

The CI helped discover a couple of bugs and their fixes have now been cherry-picked to this PR.

@bh1cqx
Copy link

bh1cqx commented Oct 8, 2024

@aalekhpatel07 Thanks a lot for your work. Looks like 1.9.0 is out. Do you intend to follow up on this?

@aalekhpatel07
Copy link
Author

@aalekhpatel07 Thanks a lot for your work. Looks like 1.9.0 is out. Do you intend to follow up on this?

Certainly! Thanks for the ping. I'll try to contribute as life permits.

In the meantime, I'm thinking we should probably sync up Enet4/faiss upto v1.9.0 and bump the checkout of ./faiss-sys/faiss to d243e62 so that the binary index IO is in place for CI.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants