Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sampling strategy for FewShotClassifier #986

Closed
BenjaminBossan opened this issue Jun 26, 2023 · 1 comment
Closed

Sampling strategy for FewShotClassifier #986

BenjaminBossan opened this issue Jun 26, 2023 · 1 comment

Comments

@BenjaminBossan
Copy link
Collaborator

We just released the FewShotClassifier class in #979.

One feature we thought could be interesting to implement is a way to change the sampling strategy for the few shot samples, i.e. how the samples are chosen for few-shot learning.

Right now, the sampling is hard-coded and basically tries to add each label at least once. This seems reasonable but there are situations where other strategies could make sense. Therefore, I would like to see a feature that allows setting the sampling strategy as a parameter. Options that come to mind:

  • Stratified sampling: roughly what we have now, but not quite
  • Fully random sampling: sample regardless of label
  • Similarity-based sampling: use the current sample to find similar samples from the training data
  • Custom sampling: Allow users to pass a callable that performs the sampling
@BenjaminBossan
Copy link
Collaborator Author

Closing this in favor of #989, which collects all TODOs in relation to LLM classification.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant