[mapserver-users] Potential bug when using Layer-Tile-Index where indexed files are GeoTiff files

[Sommer, Ashley (L&W, Dutton Park)]

Hi Everyone,
I'm trying to get WCS service with TIME parameter support working on latest Mapserver.
My data is a timeseries of large geotiff files, in a directory, with the date encoded in the filename.
Eg:
 - /datasets/mydataset/v1/layer1/raster_2020-08-01.tif
 - /datasets/mydataset/v1/layer1/raster_2020-08-02.tif
 - etc.
I'm had it working on the WMS side of things, simply using Runtime Substitution, and embedding the TIME parameter into the DATA directive.
eg:
  DATA "/datasets/mydataset/v1/layer1/raster_%time%.tif"
That worked for WMS, but not for WCS. When Mapserver's WCS service encounters a TIME parameter, it assumes I'm using a tile index. If there is no tile index, it errors out.

So I create a tile index for this dataset. Using gdaltindex for a record of "location" for each raster, then adding a temporal attribute to the dbf file, and assigning dates to each of the records.
I then tried to use that in place of DATA:
LAYER
  NAME "mylayer"
  TILEINDEX "/datasets/mydataset/v1/layer1/index.shp"
  TILEITEM "location"
END
Unfortunately, that doesn't work, because Mapserver now requires the use of new-style Layer-Tile-Index directives in order for WCS to work properly.
So I changed it to:
LAYER
  NAME "MyIndex"
  TILEINDEX "/datasets/mydataset/v1/layer1/index.shp"
  TILEITEM "location"
END
LAYER
  TILEINDEX "MyIndex"
  TILEITEM "location"
END

This now works to some extent, but then I get an error:
msTiledSHPTryOpen(): Unable to access file. Unable to open shapefile 'raster_2020-08-02.tif' for layer 'MyIndex' ... fatal error. msShapefileOpen(): Unable to access file. (/tmp/raster_2020-08-02.tif) msShapefileOpen(): Unable to access file. (/datasets/mydataset/v1/layer1/raster_2020-08-01.tif)

I believe I have tracked down why this is happening, and I believe it is a bug.
1) Mapserver _can_ open the .shp and .dbf file at the location. It must be able to to get the file location from the dbf index.
2) msTiledSHPTryOpen() is a helper fn that runs _after_ opening a tile index, to test if we can open the first-indexed file (to template some parts of the layer).
See comment here: https://github.com/MapServer/MapServer/blob/0d156a72893b1df712cb49c1b3ddaeed5365a68b/mapshape.c#L2028
See also msTileSHPTryOpen fn here: https://github.com/MapServer/MapServer/blob/0d156a72893b1df712cb49c1b3ddaeed5365a68b/mapshape.c#L1868
3) So msTiledSHPTryOpen() tries to open the first-indexed file: "/datasets/mydataset/v1/layer1/raster_2020-08-02.tif" but returns an error, even though it _can_ open that file.
The problem is, it is using msShapefileOpen() to do this, which in-turn uses msSHPOpen() to open the tif file:
See msShapefileOpen fn: https://github.com/MapServer/MapServer/blob/0d156a72893b1df712cb49c1b3ddaeed5365a68b/mapshape.c#L1665
See opener: https://github.com/MapServer/MapServer/blob/0d156a72893b1df712cb49c1b3ddaeed5365a68b/mapshape.c#L1686
4) msSHPOpen() assumes whatever file its opening will have a .shp file and a .shx file.
In this case, it's trying to open a .tif file. The function tries to open a non-existent /datasets/mydataset/v1/layer1/raster_2020-08-02.shp file.
See the implementation here: https://github.com/MapServer/MapServer/blob/0d156a72893b1df712cb49c1b3ddaeed5365a68b/mapshape.c#L194
And where it opens a file here: https://github.com/MapServer/MapServer/blob/0d156a72893b1df712cb49c1b3ddaeed5365a68b/mapshape.c#L252

So I believe it is a bug to use msShapefileOpen() to test if a tileindex can open a file at its first-indexed location, because that file may not be a .shp file.

I might be able to put in a PR to fix the bug, but I don't know the codebase well enough to not potentially introduce regressions.
Let me know what you think?

- Ashley Sommer
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/mapserver-users/attachments/20210906/415eabc1/attachment.html>