Implement observer with SQL query support and parquet export #744
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
masih
force-pushed
the
masih/observer-with-sql-support
branch
from
0d08bfd
to
82c2371
Compare
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #744 +/- ##
==========================================
- Coverage 75.15% 69.34% -5.81%
==========================================
Files 70 74 +4
Lines 6939 7497 +558
==========================================
- Hits 5215 5199 -16
- Misses 1312 1884 +572
- Partials 412 414 +2
|
masih
commented
Nov 12, 2024
| @@ -1,75 +0,0 @@ | |||
| package msgdump | |||
There was a problem hiding this comment.
Rewrote the stuff here, because:
- there seem to be an issue with decoding parquet files previously generated. So far we have just been using the ndjson dump to query things.
- I wanted to have a SQL interface that: 1) can query the latest messages and 2) can query the rotated messages all in one convenient view.
- These changes are significant enough that it deserved its own package etc.
| @@ -0,0 +1,22 @@ | |||
| package main | |||
There was a problem hiding this comment.
Renamed the file due to typo in file name and moved common pubsub utility stuff from here to internal/psutil package.
| @@ -0,0 +1,111 @@ | |||
| package observer | |||
There was a problem hiding this comment.
This allows us to run the observer under F3 passive testing infra as a HTTP server and CURL it with SQL statements, e.g.:
$ curl -X POST -d 'select distinct sender from messages' https://where-it-is-gonna-run/query
82c2371
to
9219213
Compare
masih
force-pushed
the
masih/observer-with-sql-support
branch
2 times, most recently
from
4bbc9af
to
4914105
Compare
This was referenced Nov 18, 2024
Kubuxu
reviewed
Nov 18, 2024
Kubuxu
reviewed
Nov 18, 2024
Kubuxu
approved these changes
Nov 18, 2024
Implement an F3 observer that: * listens to F3 gossipsub messages * collects them as long as they can be decoded as CBOR * offers a SQL interface to query them over a HTTP * rotates them into Parquet files * deletes any rotated files based on a configured maximum age Fix base image to distroless/cc because of duckdb libstdc++ dependencies.
masih
force-pushed
the
masih/observer-with-sql-support
branch
from
d3bfd33
to
d9a5ac4
Compare
|
My only nit would be maybe support for BasicAuth, but I don't think it is immediately necessary as it can be reverse proxied onto it. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Implement an F3 observer that: