imi-2.0.0

IMI 2.0

This is the official release of IMI 2.0:

High-level feature updates

For more information please see the IMI 2.0 research paper.

• Blended TROPOMI+GOSAT Dataset Integration: Uses ML based correction to reduce biases associated with TROPOMI observations
• Point Source Support: Integrates point source observations into state vector generation.
• Enhanced Computational Efficiency: 10x faster Jacobian construction speeds.
• Super-Observations: Uses aggregated observations to improve error characterization.
• Improved Boundary Conditions: Includes options to optimize boundary conditions.
• Adaptive Spatial Resolution: Dynamically adjusts spatial resolution based on expected information content.
• Methane Sink Optimization: Allows OH optimization as part of the inversion.
• Global Inversion Capability: Enables global emissions inversions.
• Continuous Monitoring: Kalman filter option for continuous emission tracking.
• Updated Emission Inventories: Incorporates new prior emission data sources.
• Lognormal Error PDFs: Supports lognormal distributions to better capture heavy-tailed emissions data.
• Enhanced Visualization Options: Adds new sectoral emissions and temporal variability outputs.
• Docker Containerization: Facilitates local deployment and compatibility with automated cloud platforms.

Bugfixes since the last beta version

• Bugfix/precomputed k sfs by @laestrada in #283
• update the readthedocs with faqs, glossary, and best practices by @laestrada in #284
• Bugfix for calculating jacobian scale factor by @megan-he in #285

Full Changelog: imi-2.0.0-beta.4...imi-2.0.0

imi-2.0.0-beta.4

What's Changed

• Auto format python and shell scripts by @laestrada in #273
• Rename 'prior_run' and component scripts to reflect HEMCO standalone simulation and not true prior run by @msulprizio in #274
• Bugfix/jacobian sf by @laestrada in #275
• Plot downloads by @JohnThomasSEAS in #277
• fix boundary condition readme by @nicholasbalasus in #278
• Stats Save from Notebook by @JohnThomasSEAS in #279

New Contributors

• @JohnThomasSEAS made their first contribution in #277

Full Changelog: imi-2.0.0-beta.3...imi-2.0.0-beta.4

imi-2.0.0-beta.3

Full Changelog: imi-2.0.0-beta.1...imi-2.0.0-beta.3

imi-2.0.0-beta.2

Full Changelog: imi-2.0.0-beta.1...imi-2.0.0-beta.2

imi-2.0.0-beta.1

See the IMI 2.0 preprint manuscript for detailed technical information on the updates in IMI 2.0.

What's Changed

• Point source incorporation into clustering algorithm via csv file by @laestrada in #127
• Feature/blended-tropomi-gosat by @nicholasbalasus in #139
• Feature/create IMI docker image by @laestrada in #142
• Feature/kalman filter refactor by @laestrada in #133
• chore/component-variable-manifests: improve traceability of variables by @laestrada in #159
• Point source visualization and implementation into clustering algorit… by @laestrada in #152
• Feature/dynamic kf clustering by @laestrada in #160
• Feature/update to gc 14.1.0 by @laestrada in #161
• KF notebook bugfixes by @laestrada in #163
• fix regridding bug for domains with missing lat/lon data by @laestrada in #162
• [Draft] low latency boundary condition generation by @nicholasbalasus in #156
• Update boundary condition restart file from 72 L to 47 L by @nicholasbalasus in #170
• Feature/optimize domain edges by @laestrada in #169
• update to GC 14.2.1 by @nicholasbalasus in #172
• Bugfix preventing generation of one massive cluster to fill the grid by @laestrada in #178
• GCClassic 14.2.3 update by @laestrada in #179
• Fix bugs in make_state_vector.py by @laestrada in #182
• conda activate bugfix by @laestrada in #185
• Feature: cleanup script by @laestrada in #184
• Add global inversion capability + OH optimization by @msulprizio in #157
• Add stale Github action to automatically tag and close stale issues by @msulprizio in #192
• Bug fix for environment file activations by @nicholasbalasus in #194
• Re-run capability for Jacobian simulations by @eastjames in #188
• bugfix for defining geoschem domain based on grid cell edges (not centers) by @laestrada in #190
• Bugfix/run_inversion.sh error handling by @laestrada in #196
• Fix sbatch within sbatch printing by @nicholasbalasus in #199
• update bc smoothing to use -30 day smoothing as a backup by @laestrada in #195
• fix bug where symlinked restart would be overwritten by default restart by @laestrada in #200
• Bugfix/docker env by @laestrada in #201
• bugfix/ align restart path and boundary condition version by @laestrada in #204
• Feature/blended TROPOMI+GOSAT download script by @laestrada in #205
• bugfix/geoschem domain: fix bug in specification of geoschem domain for some custom domains by @laestrada in #207
• bugfix accounting for clustering in superobservation calculation in p… by @laestrada in #208
• Update operational TROPOMI data download script by @nicholasbalasus in #209
• Give operational download script more resources by @nicholasbalasus in #214
• update aws downloads to not require aws credentials by @laestrada in #215
• Feature: Lognormal error specification by @laestrada in #175
• plot state vector as part of imi preview by @laestrada in #216
• Bugfix/soil absorption accounting by @laestrada in #213
• Feature/ubuntu docker container by @laestrada in #220
• update parse yaml function to convert to shell variables by @laestrada in #218
• feature/imi-metadata collect geoschem version, IMI version, and tropomi retrieval version by @laestrada in #223
• Feature/point_sources: new point source capabilities and clustering updates by @eastjames in #219
• Feature/v2024-06-bcs by @nicholasbalasus in #227
• Fix logic for case when no plumes available by @eastjames in #229
• bugfix for scaling of So by @laestrada in #235
• Bugfix for TROPOMI operator retrieval times by @megan-he in #236
• Bugfix/jacobian perturbation fix by @laestrada in #226
• Updates to speedup Jacobian runtime by @msulprizio in #193
• Bugfix/allow zero buffer elems by @laestrada in #224
• Bugfix/prior emissions: switch to daily precompiled emissions by @laestrada in #239
• fix postproc_diags to include 0 hour for all tracers and only merge 0th hour by @laestrada in #242
• Replace sectoral emissions piechart with barchart in viz notebook by @laestrada in #243
• Expand variables in parse_yaml.py script. by @laestrada in #241
• Prior data dry run download by @laestrada in #246
• only apply scale factors to emission data variables by @laestrada in #249
• Bugfix/perturbation sfs by @laestrada in #251
• Clean up overlap between the IMI preview and the prior simulation by @msulprizio in #245
• Additional changes for global inversion, including option to use a custom periods.csv to define Kalman Filter update periods by @megan-he in #250
• Feature/upgrade conda env by @laestrada in #221
• Add lat/lon bounds for 2x2.5 by @megan-he in #254
• Restore use of archived HEMCO standalone output for generating state vectors by @msulprizio in #255
• Add Period CSV file options to default config.yml file and set MakePeriodsCSV to false by default by @msulprizio in #256
• Feature/Include Water Observations and add dynamic range colorbar for xch4 plots by @laestrada in #260
• bugfix for printing posterior emissions by @nicholasbalasus in #261
• default to End for DiagnFreq by @nicholasbalasus in #262
• account for GC bounds being grid box edges by @nicholasbalasus in #263
• Bugfix/new-env by @nicholasbalasus in #264
• improve error handling to prevent imi runs from continuing when sbatc… by @laestrada in #265
• Remove calc_sensi.py and calculate sensitivities on XCH4 by @eastjames in #252
• Bugfix/hemco grid precision by @laestrada in #267
• bugfix/handle-operator-updates-in-bcs by @nicholasbalasus in #268
• Fix and improve IMI Preview DOFS calcula...

imi-1.2.1

What's Changed

We recommend updating to the new version of the IMI, as it includes a number of updates, bug fixes, and new features. We ask that users read bugfixes and features with bolded text to see if any bugs in prior versions of the IMI may have affected their inversions.

Bugfixes

• Important: after consultation with SRON we have removed the previously used albedo filters for processing the TROPOMI retrieval, see the linked pull request for more information. drop TROPOMI albedo filters, and Rocky 8 on Cannon by @nicholasbalasus
• bugfix to catch errors in TROPOMI variables by @sabourbaray in #128
• Bugfix/reduce obs array size by @laestrada in #135
• bugfix to only turn on extra observation switches for prior and posterior runs by @sabourbaray in #131
• Bugfix/fix harvard geos chem env files by @nicholasbalasus in #138
• Bugfix: Fix DOFS calculation by @laestrada in #140

New Features

• Important: We have rolled out a new default boundary conditions version,v2023-06. v2023-04 used the legacy CH4 Boundary Conditions repository. v2023-06 has two important updates: (1) removal of a misapplied stratospheric correction in the legacy repository and (2) removal of albedo filters for boundary condition generation with TROPOMI correction applied. We recommend that all users switch to the new boundary condition version by setting BCversion: "v2023-04" in the config.yml file.
• feature allow specification of partition for sbatch commands by @laestrada in #132 -- this is for ease of use on local clusters
• Enable nc4 compression by @laestrada in #143 This update greatly reduces the file size of the output directory generated by the IMI.

New Contributors

• @sabourbaray made their first contribution in #128
• v2023-06-BCs
  Full Changelog: imi-1.2.0...imi-1.2.1

imi-1.2.0

What's Changed

We recommend updating to the new version of the IMI, as it includes a number of updates, bug fixes, and new features. We ask that users read bugfixes with bolded text to see if any bugs in prior versions of the IMI may have affected their inversions.

Bugfixes

• *Important* Bugfix/tropomi data download -- Recent reprocessing of TROPOMI processor to v020400/020500 (in mid March 2023) has caused duplicate files with previous processor versions to be downloaded by the IMI. This could erroneously double the number of observations used in your inversion. We have fixed this issue in this version update. We recommend updating to IMI 1.2.0 for all future inversions. For past inversions, check for repeated orbit numbers in your tropomi data directory. If you find duplicates, delete files that are not v020400 or 020500, delete the inversion and posterior run directories and rerun the inversion and posterior setup and run steps. No need to rerun the jacobian, or spinup steps.
• *Important*Bugfix/IMI Boundary conditions -- The IMI uses tropomi smoothed boundary conditions to simulate background concentrations outside of the domain. We found a bug that caused the tropomi adjustment to be limited to 30ppb, which is insufficient for post 2019 inversions. We removed this 30ppb cap and regenerated the BCs on march 8, 2023. If you ran an inversion prior to this we recommend rerunning the IMI. Additionally, we have since added stable IMI BC versions, which can be specified in the IMI configuration file. See the relevant feature below. v2023-04 includes these updates. However, we plan to release v2023-06 soon, which includes a bugfix removing a misapplied stratospheric correction and transitions to using the scripts in src/write_BCs instead of the CH4-boundary-condition-scripts repository
• Bugfix/preview estimate -- previously the IMI Preview did not effectively take into account domain size when estimating cost of the inversion
• Bugfix/restart path -- update restart paths to use absolute paths instead of relative in jacobian and posterior directories
• Bugfix/merra2 compatibility -- fix paths to met data for .5 degree inversion option
• Bugfix/save viz output data -- fix issue where individual tropomi data was not saved properly (only relevant for visualizations in visualization_notebook)
• Bugfix/get num statevector elements update approach to getting the number of state vector elements in a less resource intensive way
• Add .readthedocs.yaml file to configure ReadTheDocs builds

New Features

• Feature/clustering options -- provide automated clustering options for reducing the state vector dimension. This can greatly reduce the cost of the IMI. Read the linked documentation for more information and see the relevant imi config file documentation. PRs include #88 and #102
• Feature/offshore emissions -- include offshore emissions in state vector based on emissions threshold (see relevant config file documentation)
• Feature/resource allocation settings -- add configuration settings for allocation of resources to simulations and jacobian simulations. See the config file documentation for more info.
• Feature/Add Boundary conditions scripts to IMI source code -- bring the imi BC file generation code into the IMI. Additionally, this update unifies how we process the tropomi data for the BCs and in the inversion. Additionally we implement several improvements to the BC generation code, including:
  • Removal of a 30ppb cap on TROPOMI bias corrections. This affected 2019-2022 where the average correction was much higher than 30ppb and could lead to high prior simulation bias from TROPOMI observations
  • Infilling missing values using the latitudinal mean for +/- 15 days. For latitudes with no TROPOMI observations, the closest available latitudinal average is used
  • Outliers in the .01 percentile are now replaced with the next highest values (.02 percentile). To deal with potentially nonphysical values
  • Script for bias correction was rewritten in python from the original R script.
• Add BCversion variable to config file -- we will be publishing discrete IMI Boundary conditions versions (the latest one is v2023-06). This will allow users to specify a fixed BC version to use in their configuration files via a new configuration variable eg. BCversion: v2023-06.
• Feature/Check for duplicate TROPOMI files -- Print error if there are any duplicate tropomi orbit files being used in inversion.
• Update cannon env -- update imi conda env for cannon cluster
• Add templates for github issues and pull requests
• Display IMI logo and favicon on the imi.readthedocs.io site

Contributors

• @djvaron
• @laestrada
• @msulprizio
• @nicholasbalasus
• @yantosca

Full Changelog: imi-1.1.0...imi-1.2.0

imi-1.1.0

What's Changed

• Average TROPOMI observations into superobservations see (pull request) [https://github.com//pull/57]
• Update the IMI to use GEOS-Chem 14.0.2 see (pull request) [https://github.com//pull/83]
• Add additional nested domains. Now includes: AF, AS, EU, ME, NA, OC, RU, SA
  *** Additionally a bug was found in the boundary conditions files affecting transport. This bug affected files from February 2022 onwards. The bug is now fixed and the updated boundary conditions are now uploaded to s3 and the washu server. We recommend redownloading and rerunning the inversions for any time periods past February 2022. ***

Full Changelog: imi-1.0.0...imi-1.1.0

IMI 1.0.0 Release

What's Changed

• Feature: script to find corrupted met files on aws by @laestrada in #38
• Bugfix/download landcover files in create shapefile notebook by @laestrada in #39
• Update main with bugfixes. by @laestrada in #40
• Bugfix/corrupt tropomi_files: Handle corrupt tropomi files by @laestrada in #43
• Feature/safe mode: implement safe mode to prevent overwriting files by @laestrada in #44
• feature: add timers for each step in the IMI by @laestrada in #47
• Cleanup IMI configuration file, setup script, and run script by @msulprizio in #48
• Update IMI to use GEOS-Chem 13.4.1 by @msulprizio in #51

Full Changelog: imi-1.0.0-beta.1...imi-1.0.0

imi 1.0.0-beta.2 release

What's Changed

• Feature: script to find corrupted met files on aws by @laestrada in #38
• Bugfix/download landcover files in create shapefile notebook by @laestrada in #39
• Update main with bugfixes. by @laestrada in #40
• Bugfix/corrupt tropomi_files: Handle corrupt tropomi files by @laestrada in #43
• Feature/safe mode: implement safe mode to prevent overwriting files by @laestrada in #44
• feature: add timers for each step in the IMI by @laestrada in #47
• Cleanup IMI configuration file, setup script, and run script by @msulprizio in #48
• Update IMI to use GEOS-Chem 13.4.1 by @msulprizio in #51

Full Changelog: imi-1.0.0-beta.1...imi-1.0.0-beta.2