Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

reconsider using ZooKeeper for partitioned job database #163

Open
soxofaan opened this issue Oct 14, 2024 · 0 comments
Open

reconsider using ZooKeeper for partitioned job database #163

soxofaan opened this issue Oct 14, 2024 · 0 comments

Comments

@soxofaan
Copy link
Member

soxofaan commented Oct 14, 2024

The current implementation of the "partitioned jobs" feature (large area processing, cross-backend processing, ...) depends on ZooKeeper for internal housekeeping.
At the time of the original implementation, this was the most obvious choice as we were also using it for the job registry. This job regustry solution did not scale very well so we migrated to an Elastic Search based solution.
Partitioned job execution is not yet commonly used, so it's not highly urgent, but at some point we should consider eliminating this dependency and replace it, e.g. with ES or SQLite

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant