Roadmap 2022

[larsnaesbye]

It's really cool how things are progressing. Would it make sense to try and make a roadmap so contributors can know what to expect to contribute to?

[gwbres]

[larsnaesbye]

--strict sounds good for adhering to the actual standard, rather than emulating the behaviour of CRX2RNX.

[gwbres]

I just checked the V3/V4 specifications, since I'm working on this compression topic these days.
Specifications are not super clear about this, there is no mention of the maximal line width, page 65 of RINEX3.pdf for instance.
In their example they do overflow (slighly), the provided example epochs are about 83 character long, which would point towards an overflow tolerated in case of Observation V > 2 format. Across all rinex formats, this seems to be the only tolerated case.

CRX2RNX produces wrapped epochs for V < 3 (80c), and single line epoch for V > 3.
This is also the behavior I developped, I will keep it as is

[gwbres]

Hello @larsnaesbye

I'm making good progress regarding data compression. I will get back to you soon with a todo list update.
I concluded the low level compression methods. I'm working on the general RINEX compression method, i'm about to cross the line where 1 epoch gets compressed correctly in CRNX1 fashion. All of this is only available in the hatanaka branch and I don't recommend switching to this branch yet.

Question:

1. Do you think it makes sense to automatically deduce the CRINEX revision we are about to generate, from the RINEX revision ?
2. Or should we have some interface for the user to select its own choice ?

Option 1/ would mean the user would always compress RINEX1,2 into CRINX1, and RINEX3,4 into CRINX3.
Option 2/ is not complex to implement and would allow RINEX1,2->CRINX3 and Rinex3,4->CRINX1 cross operations

I'm leaning towards option 2/

see you

[larsnaesbye]

Hello @gwbres
I've been mostly trying to see if I can promote this project. So far it's not going so fast.
Did you notice that there is a third fork over at https://github.com/FerdinandvHagen/rinex with some custom properties being added? Never heard of that...

I think most people will compress RINEX 1,2 to CRINX1, and RINEX 3,4 to CRINX3. But yes, to give them the option seems like a good way to do it.

[gwbres]

Ok thanks for your input.

Did you notice that there is a third fork

I did see that, it's a correction of our orbit description for GPS I think.
Ideally I'd like people to submit PR so the main repo gets improved, I'll leave it a few days

I've been mostly trying to see if I can promote this project. So far it's not going so fast.

I presume having more users but more importantly contributors is the only way to tackle all possible topics.
As far as I'm concerned, I'll conclude the CRINEX file production and Observation file production as you suggested.
Then I will focus on the command line interface and possible data visualization methods

[gwbres]

Hello @larsnaesbye

could you help me verify the way we format clock offsets is correct and match standards (ideally CRX1 and CRX3)?
It has never been tested, simply because all CRINEX I have access to (test_resources/CRNX) do not have Clock Offset fields.

To test that, you can invoke the crx2rnx binary on your own files. I'm expecting slight formatting issues (like missing whitespaces..) data parsing should be OK.

This is where the clock offset is squeezed in (if any), and a few lines down below for CRX3

[larsnaesbye]

I can try. I don't think most of the data we produce have clock offsets before Hatanaka compression, but maybe I can find some anyway.

[larsnaesbye]

@gwbres I have not found any CRX files with clock offset. Do you still need them, you think?

[gwbres]

Hello  @larsnaesbye
I am confident about the correct parsing of clock offset fields.
There is a much higher chance we get it wrong when we format ourselves, on the other hand.

(receiver) clock offsets are important part of compensation equations.
It would probably have a bigger impact in terms of example of use / advanced usage, that we would then be able to showcase, than anything else

[gwbres]

Hello @larsnaesbye

here's a little update on what's new.

• Data compression

  • compression algorithm is implemented and works fine.
    It is limited to M=6 which is more than enough.
    We lost the maximal compression order flexibility, but that was probably useless anyway.
    I removed the ability control M to simplify the operation of this whole thing.
    We're only interested in controlling M when compressing data anyway (rnx2crx, which is now being released),
    because this library core is capable of compression up to 6, while RNX2CRX is limited to 3.
    We"ll see if there's an interest in doing so.. they say M=4 is the optimum.
    This will be easy to test with rnx2crx: compare generated file sizes, and verify data sanity.

  • RNX2CRNX1 is under development is not fully stabilized. Some issues still remain,
    mostly related to omitted fields and compression kernels need to be re-intialized.
    Also facing some data formatting issues.

  • RNX2CRNX3 to be developed once CRNX1 is stablized

• Data production

  • I fixed several header fields formatting, it's getting close to something very nice.
    ⚠️ Our Header does not generate PRN / # OF OBS enumerations. This field is optionnal, it is possible to have Obs/V2 without it. V3+ has also all this information integrated in other forms. But I suspect this enumeration to be convenient to old programs that analyze Obs/V2. I think we should generate it as long as V < 3.
  • Epoch based data generation.
    I reworked the data generation method (Rinex::to_file) to have an epoch basis approach,
    just like the parser, which is very powerful and will provide more abstraction very quickly (especially in the Compression/Decompression operations)
  • Meteo V2/V3/V4 were verified and work like a charm
  • Observation Data V2 is unlocked and is very close to be working ❇️
    There is some whitespace issues when formatting data.
  • Observation Data V3 seems to work better than V2 ❇️
  • Navigation V2, V3, V4 are under development, there are a lot more formatting issues in this case as compared to Observation data

⚠️ there is a slight format difference when we generate epoch/timestamps: the rust library can only produce double digit Year/Month/Day fields. We might have to improve that, as it appears to be different from RINEX specifications.

💡 You will notice that Space Vehicules are naturally sorted by the library core. They will not appear in a random order when we generate data with this library. This is not a problem and might even prove convenient.

I'm providing a new release as is, because I'm interested in releasing the rnx2crnx  binary.
I will update it as the CRNX topic progresses.
Expect these topic to progress in the next few days, I will provide new tags

[gwbres]

Hello, yeah I just did a quick cleaning phase.
I know I'm sorry.. they have a tendancy to become overwhelming very quickly. it's just i'm working on so many topics at the same time 😆

So there's a problem with the RINEX to CRINX1 compression, when an epoch is terminated by an empty line (all last fields omitted), the internal logic is not smart enough to detect that condition: we keep parsing and data gets mixed up across vehicules

[gwbres]

Hello @larsnaesbye
if you want to check the current status you can do something like this

cargo run --bin rnx2crx -- -f test_resources/OBS/V2/npaz3550.21o -o /tmp/test.21d
vimdiff test_resources/CRNX/V1/npaz3550.21d /tmp/test.21d

Looks fairly decent 😄

To test our retrocompatibility:

CRX2RNX /tmp/test.21d  # official tool
ERROR at line 130 : New satellite, but data arc is not initialized.
  start>3&241780<end

I just demonstrated this is only due to whitespaces mismatch in the generated format.
The format they use being far from obvious.. I actually just looked at crx2rnx.c: we're missing an equivalent of the following - when formatting observables:

    if(dddu == 0){
        p_buff += sprintf(p_buff,"%ld",dddl);
    }else{
        p_buff += sprintf(p_buff,"%ld%5.5ld",dddu,labs(dddl));
    }

where dddu and dddl are apparently defined as the first 5 upper digits, and all lower digits

[gwbres]

    if(dddu == 0){
        p_buff += sprintf(p_buff,"%ld",dddl);
    }else{
        p_buff += sprintf(p_buff,"%ld%5.5ld",dddu,labs(dddl));
    }

where dddu and dddl are apparently defined as the first 5 upper digits, and all lower digits

So it turned out not to be a printf() formatting issue, but instead the official tools simply append an empty whitespace when data fields were omitted. I also fixed several things regarding flags management today. We're actually very close to a working version1. For instance, compression of zegv0010.10d works. There seems to be two issues left.

1. when forced reinitialization needs to happen, we decode empty data flags while the other tools do not. I need to recheck every single line and understand what's happening

2. they're seem to be an issue when new vehicules appear, but I 'm not there yet

once revision 1 is done, it should not take long to have revision 3 also working, considering the only difference is how and where we identify the current vehicule

[gwbres]

Hello,
I improved a few things regarding NAV files 🛰️
V2 generation is close to be working, a whitespace/identation issue still exist, but data is correct. V3 and V4 will soon follow afterwards.
I also declared new data types to represent satellite Health condition. When parsing, recognized data is much more meaningful

[larsnaesbye]

Cool. V3 is what we use almost exclusively, and I suspect most of the world does too.

[gwbres]

Hello @larsnaesbye,

today I introduced some benchmarks, currently only focused on NumDiff::decompression and TextDiff::decompression.
Because the benchmark is properly focused, you can see that two identical contexts converge to similar performances (which is good 👍 ). I spent sometime today improving these two methods yet maintaining identical behavior.
The current benchmark results are "12us/1us" for epoch/flag decompression, in the main branch, and now down to "730ns/90ns" in the improv branch.
I just ran a quick test using the two CRNX files you provided last summer (because they're the largest and realistic).
The test consisted in requesting --sv enumeration on each of these file.
It produced the following results

File	main	improv	improv compiled with --release
ESBC00DNK	26s	14s	2s 😮
ESBC00DNK.gz	26s 😮	14s 😮	2s 😮
MOJN00DNK	28s	13s 👍	2s 😮
MOJN00DNK.gz	28s	13s	2s 😮

Most major impact is using a --released binary and not a debug product.
I will add a Huge Note on this, at the beginning of our readme file.
improv's improvements really show up if we compare today's with yesterday's implementation.
.gz compression has absolutely no impact! which is awesome 👍 and the result of operating the GzDecoder correctly.

Next step regarding this topic, is to benchmark a real decompression, but I will probably have to dig further in the Bench parameters to filter out noise and average a lot more.

[larsnaesbye]

Wow! That is quite some improvement in speed.

[gwbres]

Wow! That is quite some improvement in speed.

Indeed 👍
but to be fair, I was too lazy to rerun the test on main + --released, I suspect it is already quite good in that state.
So keep in mind that the optimized compilation is the way to go ;)

[gwbres]

Hello @larsnaesbye
just to keep you updated
I will conclude the conversion from chrono to hifitime in the next couple of days.
It is a little bit painful, there are still several things to stabilize and verify, but the basis is already there.

After that, I will tag 0.8.0 to officialize so many improvements and new features.
I foresee 0.9.0 as being this stage + the Navigation and CRINEX formatting finalized.

Moving to hifitime is a major improvement that should not be under valued

• sub second precision barrier is crossed.
• we support the Observation RINEX precision, which is the most stringent to my knowledge: 100 ns, because we'll have a theoretical 1ns precision
• Supporting Time scales correctly and Leap seconds will come very handy for RINEX operations and accurate definitions
• getting rid of an old "chrono" dependency

[larsnaesbye]

Wouldn't you say that formatting could be 0.9.0 too?

[gwbres]

Hello @larsnaesbye,
on my side I will focus on new processing methods and the QC report, rather than concluding the two RINEX format we're still unable to generate as of today. Mostly because I need a break from these low level "problems"

There are two of them (1) RNX2CRX3 (2) Navigation data.
I will open two issues to clarify their status, Navigation data is mostly completed.

I am not saying I will never do that, but I will rather have 1 or 2 months of fun developping algorithms and the QC than working once again on parsing related issues

[larsnaesbye]

Oh I'm sorry. I meant the code formatting - our 'rustfmt' setup :)

The milestone for that was set to 1.0.0 like the others and I thought we could have that in 0.9.0.

[gwbres]

Oh I'm sorry. I meant the code formatting - our 'rustfmt' setup :)

no worries, yes all ongoing topics will be integrated in the 0.9. The general CI improvements, the Keplerian equations (real Navigation data usage) that will come with new cli features and visualization methods, and maybe the python bindings too. For the latter it all depends on how we keep progressing regarding this topic. The only breaking point is the case of complex enums and I have not found a solution.

Edit: as for the main front table status, I should have open issues because I don't even remember what truly remains to be supported, I'll run a few tests

[larsnaesbye]

@gwbres We have closed the issues added to the 0.9.0 milestone, but I guess we still have some way to go before a release. Can we sum them up?

[gwbres]

Hello @larsnaesbye,

Yes of course. I may increment the list as I may forget a few topics.

• Merge rinex-qc and processing toolkit #86
  • newly developed filter toolkit which is similar to the ESA python toolkit in its syntax: quite effective
  • new "web-based" plot and graphics: much easier and way more efficient API
  • QC web-based reports
  • QC products and session management
• Fix Decoded Epoch type often has wrong timescale #81
  The initialization logic needs to be more complex
• Try to provide a solution to Support CNAV ephemeris types #76
  We need to add a MessageType field to the dictionnary description, and use it when we parse such a file
• Python bindings #48
  topic is stuck and currently limited by one or two aspects that were detailed in the PR. This topic should not block this library from evolving, it is a rust library primarily. I'm also sure as the binding framework evolves, they will get naturally resolved.

[larsnaesbye]

I agree. Python bindings are a fine idea, but not something that should block any point release.