Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tokenize parsing #359

Draft
wants to merge 3 commits into
base: main
Choose a base branch
from
Draft

Tokenize parsing #359

wants to merge 3 commits into from

Conversation

MikkelPaulson
Copy link
Collaborator

@MikkelPaulson MikkelPaulson commented Sep 26, 2024

This is an(other) attempted solution for #104.

  1. Commands are moved out of a grab bag of enums (eg. StorageCommand) into one module per command contained in a flat hierarchy, eg. crate::command::about.
  2. Commands are represented by a token which can itself consist of other tokens.
  3. The input string is first parsed into tokens by the parser, after which the series of tokens is passed back to the command logic to evaluate according to its own rules.
  4. Metadata and tags can be associated with tokens to allow the command logic to identify which tokens were matched and respond accordingly (eg. including metadata of a matched Thing when handling the token that matched its name).

This is accomplished using async streams. The proof-of-concept mostly works now, but I'm still a bit tangled up in the ergonomics of the new setup.

See about.rs for a simple example.

Copy link

cloudflare-workers-and-pages bot commented Sep 26, 2024

Deploying initiative-sh with  Cloudflare Pages  Cloudflare Pages

Latest commit: 4bbe161
Status: ✅  Deploy successful!
Preview URL: https://4b862057.initiativesh.pages.dev
Branch Preview URL: https://tokenize-parsing-2.initiativesh.pages.dev

View logs

@MikkelPaulson
Copy link
Collaborator Author

@ChrisRenfrow It's actually coming together this time! Not sure how much sense you can make of this, especially in its draft form, but there will be so much less redundancy when this is done.

@MikkelPaulson MikkelPaulson linked an issue Sep 26, 2024 that may be closed by this pull request
@MikkelPaulson MikkelPaulson force-pushed the tokenize-parsing-2 branch 5 times, most recently from 84db7f7 to 7503296 Compare October 8, 2024 19:09
@MikkelPaulson MikkelPaulson force-pushed the tokenize-parsing-2 branch 5 times, most recently from 3cd9c82 to 1cf3623 Compare October 31, 2024 15:35
New method returns the original input phrase to the word.
Add iterator to test phrases of varying length, for cases where the
parser may need to consume multiple words at once.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Improve developer ergonomics of adding commands
1 participant