Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

support less than in filter_labels along time dimension #333

Open
soxofaan opened this issue Oct 15, 2024 · 3 comments
Open

support less than in filter_labels along time dimension #333

soxofaan opened this issue Oct 15, 2024 · 3 comments
Assignees
Labels
bug Something isn't working

Comments

@soxofaan
Copy link
Member

soxofaan commented Oct 15, 2024

In context of https://forum.dataspace.copernicus.eu/t/sentinel-3-lst-day-time-images/1237 I was playing with filter_labels along time dimension:

import openeo
connection = openeo.connect(url="openeo.dataspace.copernicus.eu").authenticate_oidc()

raw = connection.load_collection(
    "SENTINEL3_SLSTR_L2_LST",
    spatial_extent={"west":3, "south": 51, "east": 3.1, "north": 51.1},
    temporal_extent=["2024-09-01", "2024-09-05"],
    bands=["LST"],
)

filtered = raw.filter_labels(
    (lambda label: label < "2024-09-01T15"), 
    dimension="t",
)

(synchronous) download fails with

OpenEoApiError: [400] BadRequest: java.lang.IllegalArgumentException: Process [lt] expects a y argument. These arguments were found: xfunction tree: lt (ref: r-2410155ebb1949f1ae070a685954329e)

While the process graph actually looks fine here (has a y argument in the lt process):

{
  "process_graph": {
    "loadcollection1": {
      "process_id": "load_collection",
      "arguments": {
        "bands": ["LST"],
        "id": "SENTINEL3_SLSTR_L2_LST",
        "spatial_extent": {"west": 3, "south": 51, "east": 3.1, "north": 51.1},
        "temporal_extent": ["2024-09-01", "2024-09-05"]
      }
    },
    "filterlabels1": {
      "process_id": "filter_labels",
      "arguments": {
        "condition": {
          "process_graph": {
            "lt1": {
              "process_id": "lt",
              "arguments": {
                "x": {"from_parameter": "value"},
                "y": "2024-09-01T15"
              },
              "result": true
            }
          }
        },
        "data": {"from_node": "loadcollection1"},
        "dimension": "t"
      },
      "result": true
    }
  }
}

So this looks like a bug in the filter_labels implemenation

@soxofaan
Copy link
Member Author

dump from relevant logs:

  File "/opt/openeo/lib/python3.8/site-packages/openeogeotrellis/layercatalog.py", line 114, in load_collection
    return self._load_collection_cached(collection_id, load_params, WhiteListEvalEnv(env, WHITELIST))
  File "/opt/openeo/lib/python3.8/site-packages/openeogeotrellis/layercatalog.py", line 260, in _load_collection_cached
    datacubeParams, single_level = datacube_parameters.create(load_params, env, jvm)
  File "/opt/openeo/lib/python3.8/site-packages/openeogeotrellis/datacube_parameters.py", line 67, in create
    labels_filter = GeoPySparkBackendImplementation.accept_process_graph(
  File "/opt/openeo/lib/python3.8/site-packages/openeogeotrellis/backend.py", line 985, in accept_process_graph
    return GeotrellisTileProcessGraphVisitor.create(default_input_parameter=default_input_parameter,default_input_datatype=default_input_datatype).accept_process_graph(process_graph)
  File "/opt/openeo/lib/python3.8/site-packages/openeo/internal/process_graph_visitor.py", line 78, in accept_process_graph
    self.accept_node(graph[top_level_node])
  File "/opt/openeo/lib/python3.8/site-packages/openeo/internal/process_graph_visitor.py", line 89, in accept_node
    self._accept_process(process_id=pid, arguments=arguments, namespace=namespace)
  File "/opt/openeo/lib/python3.8/site-packages/openeo/internal/process_graph_visitor.py", line 105, in _accept_process
    self.leaveProcess(process_id=process_id, arguments=arguments, namespace=namespace)
  File "/opt/openeo/lib/python3.8/site-packages/openeogeotrellis/processgraphvisiting.py", line 35, in leaveProcess
    self.builder.expressionEnd(process_id, arguments)
  File "/usr/local/spark/python/lib/py4j-0.10.9.7-src.zip/py4j/java_gateway.py", line 1322, in __call__
    return_value = get_return_value(
  File "/usr/local/spark/python/lib/py4j-0.10.9.7-src.zip/py4j/protocol.py", line 326, in get_return_value
    raise Py4JJavaError(
py4j.protocol.Py4JJavaError: An error occurred while calling o14466.expressionEnd.
: java.lang.IllegalArgumentException: Process [lt] expects a y argument. These arguments were found: xfunction tree: lt
	at org.openeo.geotrellis.OpenEOProcessScriptBuilder.$anonfun$getProcessArg$1(OpenEOProcessScriptBuilder.scala:611)
	at scala.collection.MapLike.getOrElse(MapLike.scala:131)
	at scala.collection.MapLike.getOrElse$(MapLike.scala:129)
	at scala.collection.AbstractMap.getOrElse(Map.scala:65)
	at org.openeo.geotrellis.OpenEOProcessScriptBuilder.getProcessArg(OpenEOProcessScriptBuilder.scala:611)
	at org.openeo.geotrellis.OpenEOProcessScriptBuilder.xyFunction(OpenEOProcessScriptBuilder.scala:885)
	at org.openeo.geotrellis.OpenEOProcessScriptBuilder.expressionEnd(OpenEOProcessScriptBuilder.scala:1123)
	at jdk.internal.reflect.GeneratedMethodAccessor581.invoke(Unknown Source)

@soxofaan soxofaan added the bug Something isn't working label Oct 15, 2024
@jdries jdries changed the title failing filter_labels along time dimension support less than in filter_labels along time dimension Nov 12, 2024
@jdries
Copy link
Contributor

jdries commented Nov 12, 2024

Note: tested approach for filter labels is based on 'date_between'. Other approaches have not yet been validated, and seem to give this weird error.

@soxofaan
Copy link
Member Author

indeed, if something is not supported ("less than" on date labels in this example), we should have that more explicit in the error message.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants