Authorisation ignored when using dynamic datasets

[mcoombe]

We've currently got TDS 4.6.11 deployed with a custom Authorizer plugin installed. In addition to the custom authoriser, we've got a server-side process that allows upload of NetCDF files and automatically modifies the TDS catalog to include the uploaded file. We've disabled catalog caching in the threddsConfig.xml, to ensure that changes made by our custom upload process are picked up automatically by the TDS.

Unfortunately, using this setup we're observing some unusual behaviour regarding the authorisation of restricted datasets.

Upon a new file being uploaded, we can observe the file being written to disk and the catalog XML file being updated accordingly (including a restrictAccess property on the dataset element). Accessing the catalog via the TDS web service also shows the updated catalog with the new dataset, as expected.

However, when attempting to access the newly uploaded dataset, we find that we are able to access it fully without authorization, even though the catalog confirms restrictAccess is applied. Looking at the logs, we've determined that our custom authoriser is being ignored in such cases (the authorize method is never called). This behaviour continues until the TDS is manually restarted, at which point authorization works as expected.

Our best guess is that this is likely due to the TDS' internal caching of catalogs. Is this a known issue, or are we perhaps mis-understanding something?

[cofinoa]

@mcoombe,

Could you show what is in the threddsConfig.xml with respect to disable the TDS cache?.

[mcoombe]

@cofinoa please find our threddsConfig.xml following (I've stripped the comments for brevity and redacted the server info, but it's otherwise exactly what's on the server).

<?xml version="1.0" encoding="UTF-8"?>
<threddsConfig>
  <serverInformation>
    <!-- REDACTED -->
  </serverInformation>
  
  <htmlSetup>
    <standardCssUrl>tds.css</standardCssUrl>
    <catalogCssUrl>/thredds/data61-thredds.css</catalogCssUrl>
    <openDapCssUrl>tdsDap.css</openDapCssUrl>
  </htmlSetup>
  
  <CatalogServices>
    <allowRemote>true</allowRemote>
  </CatalogServices>
  
  <Netcdf4Clibrary>
    <libraryPath>/usr/lib</libraryPath>
    <libraryName>netcdf</libraryName>
  </Netcdf4Clibrary>
  
  <NetcdfFileCache>
    <minFiles>50</minFiles>
    <maxFiles>0</maxFiles>
    <scour>11 min</scour>
  </NetcdfFileCache>
  
  <NetcdfSubsetService>
    <allow>true</allow>
    <scour>10 min</scour>
    <maxAge>-1 min</maxAge>
  </NetcdfSubsetService>
  
  <WCS>
    <allow>true</allow>
    <allowRemote>false</allowRemote>
    <scour>15 min</scour>
    <maxAge>30 min</maxAge>
  </WCS>

  <WMS>
    <allow>true</allow>
    <allowRemote>false</allowRemote>
    <maxImageWidth>2048</maxImageWidth>
    <maxImageHeight>2048</maxImageHeight>
  </WMS>
  
  <NCISO>
    <ncmlAllow>true</ncmlAllow>
    <uddcAllow>true</uddcAllow>
    <isoAllow>true</isoAllow>
  </NCISO>
  
   <Catalog>
	  <cache>false</cache>
   </Catalog>
</threddsConfig>

[lesserwhirls]

Unfortunately, the only way to fully add new catalogs in TDS 4.6.x is to restart the server. There is an experimental feature in 5.0 (called CatalogScan) which allows for catalogs to be added and removed without restarting, but my guess is it is not tested to work with restricted catalogs, so that may or may not be working at this point.