Why convert http to s3 links?

[mankoff]

https://github.com/nsidc/earthdata/blob/098082519cd00c5ad68c3565bf9bdfd6d2ef85f9/earthdata/results.py#L221-L245

I'm trying to get http links with your code but it seems to convert the links returned by CMR to s3 links with the code above. Is there a reason for this? Someway to force http?

[betolink]

Hi Ken, the preference is to use the S3 links if we are in-region, the main reason behind it is performance (followed by potential egress costs). When we use the store get() method and we are in us-west-2, S3 links will be used by default but we can just skip that part and get the out of region links. For example:

from earthdata import DataGranules

query = DataGranules().parameters(
    concept_id= "C2153574670-NSIDC_CPRD",
    bounding_box=(-150.29,56.17,-128.14,62.02)
)

print("Granules fonud:", query.hits())

# For some reason this collection is super slow now
granules = query.get(10)

print("Direct Access URL:", granules[0].data_links(access="direct"), "\nOut of Region URL: ", granules[0].data_links(access="onprem"))

The output is

Direct Access URL: ['s3://nsidc-cumulus-prod-protected/ATLAS/ATL08/005/2018/10/14/ATL08_20181014001049_02350102_005_01.h5'] 
Out of Region URL:  ['https://data.nsidc.earthdatacloud.nasa.gov/nsidc-cumulus-prod-protected/ATLAS/ATL08/005/2018/10/14/ATL08_20181014001049_02350102_005_01.h5']

data_links() accepts the access parameter, if we specify direct we'll get the S3 links back, if we use onprem we ge the HTTPS links that can be used from anywhere.

[betolink]

Another thing, some cloud hosted collections from PODAAC (if I recall correctly ) used to provide HTTPS links but no S3 links (although they were accessible in S3) and that was another reason behind this helper method. Maybe this needs to be revisited and see if this is at all necessary.