Unable to open Sentinel 1 product #180

guillemc23 · 2024-11-11T10:05:43Z

guillemc23
Nov 11, 2024

Hello! I'm getting started with this library. I'm trying to follow the tutorial on SAR imagery by reading a Sentinel 1 product but I'm having no luck with this task.

Currently, I'm developing inside a devcontainer built from the latest GDAL image, which contains GDAL 3.9.3 and its bindings and Python 3.12. I'm also using UV as my Python package manager. I installed SNAP 11 as well on this image (I believe it is properly installed).

When I try a simple command like

s1_prod = Reader().open(path)

I see no errors and it looks like the product is properly read, but when I try to perform a simple operation like:

bands = [VV, HH]
 # Only keep those selected
ok_bands = [band for band in bands if s1_prod.has_band(band)]

stack = s1_prod.stack(ok_bands, stack_path="s1_stack.tif")

I get the following errors, if my S1 product is zipped:

2024-11-11 10:02:51,107 - [DEBUG] - Loading bands ['VV']
Failed to open dataset (flags=68): /vsizip/S1A_IW_GRDH_1SDV_20241107T110059_20241107T110130_056447_06EB04_D455.SAFE.zip!preview/map-overlay.kml`

if my S1 product is not zipped:

2024-11-11 09:55:56,130 - [WARNING] - Product filename is not a valid Sentinel-1 name, and the retrieved name is missing the unique ID.
2024-11-11 09:56:08,038 - [DEBUG] - Loading bands ['VV']
Unable to open datasource `ERROR_1: S1A_IW_GRDH_1SDV_20241107T110059_20241107T110130_056447_06EB04_D455/preview/map-overlay.kml' with the following drivers.
  -> `PCIDSK'
  -> `PDS4'
  -> `VICAR'
  -> `JP2OpenJPEG'
  -> `PDF'
...

This is a Sentinel 1 product downloaded straight from the CreoDIAS cloud, it has not been modified. Not sure what's happening, but there is definitely a preview/map-overlay.kml inside my zip and ogr2ogr is working in my command line and in my Python environment, from osgeo import ogr works 🤔

Also, opening the kml file with QGIS works, it's not corrupt.

Not sure if related, but if I perform another simple operation like:

s1_prod.footprint()

I get the following error:

Failed to open dataset (flags=68): /vsizip/S1A_IW_GRDH_1SDV_20241107T110059_20241107T110130_056447_06EB04_D455.SAFE.zip!preview/map-overlay.kml
eoreader.exceptions.InvalidProductError: Cannot determine the WGS84 extent of S1A_IW_GRDH_1SDV_20241107T110059_20241107T110130_056447_06EB04_D455.SAFE-report-20241107T120409

Any help is appreciated. Thank you!

EDIT: I tested updating to latest 0.27 but still getting the same errors

Answered by remi-braun

Nov 14, 2024

Ok so this is definitely a setup issue, not a eoreader one.
For some reason, ogr2ogr cannot read your KML 🤔

In the end I'll try not to use map-overlay.kml to determine the extent, but rather what's in the metadata, even if it's a bit more complicated to handle. See issue #182 :)

View full answer

remi-braun · 2024-11-12T08:09:48Z

remi-braun
Nov 12, 2024
Maintainer

Hi,

EDIT: I tested updating to latest 0.27 but still getting the same errors

Could you confirm that you tested with v0.21.7?

I don't see any KML driver in this list you displayed:

Unable to open datasource `ERROR_1: S1A_IW_GRDH_1SDV_20241107T110059_20241107T110130_056447_06EB04_D455/preview/map-overlay.kml' with the following drivers.
  -> `PCIDSK'
  -> `PDS4'
  -> `VICAR'
  -> `JP2OpenJPEG'
  -> `PDF'
...

Do you confirm you have the driver installed in your docker image?

16 replies

guillemc23 Nov 13, 2024
Author

The command works but the output is not exactly the same:

>>> from sertit import misc
>>> misc.run_cli("ogr2ogr")
ERROR 1: dst_dataset_name: 1 argument(s) expected. 0 provided.
Usage: ogr2ogr [--help] [--long-usage] [--help-general]
               [-of <output_format>] [-dsco <NAME>=<VALUE>]... [-lco <NAME>=<VALUE>]...
               [[-append]|[-upsert]|[-overwrite]]
               [-update] [-sql <statement>|@<filename>] [-dialect <dialect>] [-spat <xmin> <ymin> <xmax> <ymax>]
               [-where <restricted_where>|@<filename>] [-select <field_list>] [-nln <name>] [-nlt <type>]...
               [-s_srs <srs_def>]
               [[-a_srs <srs_def>]|[-t_srs <srs_def>]]
               <dst_dataset_name> <src_dataset_name> [<layer_name>]...

Field related options:
               [-addfields] [-relaxedFieldNameMatch] [-fieldTypeToString All|<type1>[,<type2>]...]
               [-mapFieldType <srctype>|All=<dsttype>[,<srctype2>=<dsttype2>]...] [-fieldmap <field_1>[,<field_2>]...]
               [-splitlistfields] [-maxsubfields <n>] [-emptyStrAsNull] [-forceNullable] [-unsetFieldWidth]
               [-unsetDefault] [-resolveDomains] [-dateTimeTo UTC|UTC(+|-)<HH>|UTC(+|-)<HH>:<MM>] [-noNativeData]

Advanced geometry and SRS related options:
               [-dim layer_dim|2|XY|3|XYZ|XYM|XYZM] [-s_coord_epoch <epoch>] [-a_coord_epoch <epoch>]
               [-t_coord_epoch <epoch>] [-ct <pipeline_def>] [-spat_srs <srs_def>] [-geomfield <name>]
               [-segmentize <max_dist>] [-simplify <tolerance>] [-makevalid] [-wrapdateline]
               [-datelineoffset <val_in_degree>] [-clipsrc [<xmin> <ymin> <xmax> <ymax>]|<WKT>|<datasource>|spat_extent]
               [-clipsrcsql <sql_statement>] [-clipsrclayer <layername>] [-clipsrcwhere <expression>]
               [-clipdst [<xmin> <ymin> <xmax> <ymax>]|<WKT>|<datasource>] [-clipdstsql <sql_statement>]
               [-clipdstlayer <layername>] [-clipdstwhere <expression>] [-explodecollections] [-zfield <name>]
               [-gcp <ungeoref_x> <ungeoref_y> <georef_x> <georef_y> [<elevation>]]... [-tps] [-order 1|2|3]
               [-xyRes <val>[ m|mm|deg]] [-zRes <val>[ m|mm]] [-mRes <val>] [-unsetCoordPrecision]

Other options:
               [--quiet] [-progress] [-if <format>]... [-oo <NAME>=<VALUE>]... [-doo <NAME>=<VALUE>]...
               [-fid <FID>] [-preserve_fid] [-unsetFid]
               [[-skipfailures]|[-gt <n>|unlimited]]
               [-limit <nb_features>] [-ds_transaction] [-mo <NAME>=<VALUE>]... [-nomd]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/workspaces/oneo-processor-common-steps/steps/sar_preprocessor/.venv/lib/python3.12/site-packages/sertit/misc.py", line 377, in run_cli
    raise RuntimeError(f"Exe {cmd[0]} has failed.")
RuntimeError: Exe o has failed.

I can try running this in a Docker container without GDAL, this one has osgeo and all of these add ons because they come within the official GDAL image, maybe if I run it in a clean Python 3.12 image... 🤔

remi-braun Nov 13, 2024
Maintainer

And what about:

from sertit import strings, misc
cmd_line = [
        "ogr2ogr",
        "-fieldTypeToString DateTime",  # Disable warning
        "-f GeoJSON",
        strings.to_cmd_string("your/output.geojson"),  # dst
        strings.to_cmd_string("path/to/preview/map-overlay.kml"),  # src
]
misc.run_cli(cmd_line)

guillemc23 Nov 13, 2024
Author

This produces a very similar error as the first one 🤔

Python 3.12.3 (main, Sep 11 2024, 14:17:37) [GCC 13.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from sertit import strings, misc
>>> cmd_line = ["ogr2ogr", "-fieldTypeToString DateTime", "-f GeoJSON", strings.to_cmd_string("output.geojson"), strings.to_cmd_string("S1A_IW_GRDH_1SDV_20241107T110059_20241107T110130_056447_06EB04_D455.SAFE/preview/map-overlay.kml")]
>>> misc.run_cli(cmd_line)
ERROR 1: Unable to open datasource `S1A_IW_GRDH_1SDV_20241107T110059_20241107T110130_056447_06EB04_D455.SAFE/preview/map-overlay.kml' with the following drivers.
  -> `PCIDSK'
  -> `PDS4'
  -> `VICAR'
  -> `JP2OpenJPEG'
  -> `PDF'
  -> `MBTiles'
  -> `EEDA'
  -> `OGCAPI'
  -> `ESRI Shapefile'
  -> `MapInfo File'
  -> `UK .NTF'
  -> `LVBAG'
  -> `OGR_SDTS'
  -> `S57'
  -> `DGN'
  -> `OGR_VRT'
  -> `Memory'
  -> `CSV'
  -> `NAS'
  -> `GML'
  -> `GPX'
  -> `KML'
  -> `GeoJSON'
  -> `GeoJSONSeq'
  -> `ESRIJSON'
  -> `TopoJSON'
  -> `Interlis 1'
  -> `Interlis 2'
  -> `OGR_GMT'
  -> `GPKG'
  -> `SQLite'
  -> `WAsP'
  -> `PostgreSQL'
  -> `OpenFileGDB'
  -> `DXF'
  -> `CAD'
  -> `FlatGeobuf'
  -> `Geoconcept'
  -> `GeoRSS'
  -> `VFK'
  -> `PGDUMP'
  -> `OSM'
  -> `GPSBabel'
  -> `OGR_PDS'
  -> `WFS'
  -> `OAPIF'
  -> `EDIGEO'
  -> `SVG'
  -> `Idrisi'
  -> `XLS'
  -> `ODS'
  -> `XLSX'
  -> `Elasticsearch'
  -> `Carto'
  -> `AmigoCloud'
  -> `SXF'
  -> `Selafin'
  -> `JML'
  -> `PLSCENES'
  -> `CSW'
  -> `VDV'
  -> `GMLAS'
  -> `MVT'
  -> `NGW'
  -> `MapML'
  -> `GTFS'
  -> `PMTiles'
  -> `JSONFG'
  -> `MiraMonVector'
  -> `TIGER'
  -> `AVCBin'
  -> `AVCE00'
  -> `HTTP'
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/workspaces/oneo-processor-common-steps/steps/sar_preprocessor/.venv/lib/python3.12/site-packages/sertit/misc.py", line 377, in run_cli
    raise RuntimeError(f"Exe {cmd[0]} has failed.")
RuntimeError: Exe ogr2ogr has failed.

remi-braun Nov 14, 2024
Maintainer

Ok so this is definitely a setup issue, not a eoreader one.
For some reason, ogr2ogr cannot read your KML 🤔

In the end I'll try not to use map-overlay.kml to determine the extent, but rather what's in the metadata, even if it's a bit more complicated to handle. See issue #182 :)

Answer selected by guillemc23

guillemc23 Nov 15, 2024
Author

Bonjour Rémi!
I've been performing some more testing. I managed to make it work with no issues on a Sentinel-2 product in the same docker container but keeping UV as my Python package manager. I also updated my Dockerfile to use GDAL 3.10 (released a few days ago).

I downloaded another Sentinel-1 product (since the zip file seemed to be not valid), and I managed to make it work unzipped in the same environment (at least the basic functionalities I tested), but the zipped version was still giving me the same error:

2024-11-15 12:14:12,807 - [DEBUG] - Loading bands ['VV']
Failed to open dataset (flags=68): /vsizip/S1A_IW_GRDH_1SDV_20241113T060116_20241113T060141_056532_06EE5B_B99A.SAFE.zip!preview/map-overlay.kml

It could maybe be related to compression issues? I'm not sure, since my background is not geospatial, but maybe when reading the product it's expecting it to be zipped with no compression 🤔

Any way, I was able to make it work with some thorough testing, now I only need to properly set up GPT's parameters so it doesn't blow up my small PC 😄 I'll keep you posted

One thing I changed during my tests was pasing a Path instead of a string to the reader, I think we might want to enforce a Path... I'll try PR with this or other issues, this package is promising 🤘🏻

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unable to open Sentinel 1 product #180

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 1 comment 16 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

Unable to open Sentinel 1 product #180

guillemc23 Nov 11, 2024

Replies: 1 comment · 16 replies

remi-braun Nov 12, 2024 Maintainer

guillemc23 Nov 13, 2024 Author

remi-braun Nov 13, 2024 Maintainer

guillemc23 Nov 13, 2024 Author

remi-braun Nov 14, 2024 Maintainer

guillemc23 Nov 15, 2024 Author

guillemc23
Nov 11, 2024

Replies: 1 comment 16 replies

remi-braun
Nov 12, 2024
Maintainer

guillemc23 Nov 13, 2024
Author

remi-braun Nov 13, 2024
Maintainer

guillemc23 Nov 13, 2024
Author

remi-braun Nov 14, 2024
Maintainer

guillemc23 Nov 15, 2024
Author