[mapserver-users] Potential bug when using Layer-Tile-Index where indexed files are GeoTiff files

[Steve Lime]

I'd agree this is a bug - just not sure what the fix should be.

On Mon, Sep 6, 2021 at 4:21 AM Sommer, Ashley (L&W, Dutton Park)
<Ashley.Sommer at csiro.au> wrote:

> Hi Everyone,
> I'm trying to get WCS service with TIME parameter support working on
> latest Mapserver.
> My data is a timeseries of large geotiff files, in a directory, with the
> date encoded in the filename.
> Eg:
>  - /datasets/mydataset/v1/layer1/raster_2020-08-01.tif
>  - /datasets/mydataset/v1/layer1/raster_2020-08-02.tif
>  - etc.
> I'm had it working on the WMS side of things, simply using Runtime
> Substitution, and embedding the TIME parameter into the DATA directive.
> eg:
>   DATA "/datasets/mydataset/v1/layer1/raster_%time%.tif"
> That worked for WMS, but not for WCS. When Mapserver's WCS service
> encounters a TIME parameter, it assumes I'm using a tile index. If there is
> no tile index, it errors out.
>
> So I create a tile index for this dataset. Using gdaltindex for a record
> of "location" for each raster, then adding a temporal attribute to the dbf
> file, and assigning dates to each of the records.
> I then tried to use that in place of DATA:
> LAYER
>   NAME "mylayer"
>   TILEINDEX "/datasets/mydataset/v1/layer1/index.shp"
>   TILEITEM "location"
> END
> Unfortunately, that doesn't work, because Mapserver now requires the use
> of new-style Layer-Tile-Index directives in order for WCS to work properly.
> So I changed it to:
> LAYER
>   NAME "MyIndex"
>   TILEINDEX "/datasets/mydataset/v1/layer1/index.shp"
>   TILEITEM "location"
> END
> LAYER
>   TILEINDEX "MyIndex"
>   TILEITEM "location"
> END
>
> This now works to some extent, but then I get an error:
> msTiledSHPTryOpen(): Unable to access file. Unable to open shapefile
> 'raster_2020-08-02.tif' for layer 'MyIndex' ... fatal error.
> msShapefileOpen(): Unable to access file. (/tmp/raster_2020-08-02.tif)
> msShapefileOpen(): Unable to access file.
> (/datasets/mydataset/v1/layer1/raster_2020-08-01.tif)
>
> I believe I have tracked down why this is happening, and I believe it is a
> bug.
> 1) Mapserver _can_ open the .shp and .dbf file at the location. It must be
> able to to get the file location from the dbf index.
> 2) msTiledSHPTryOpen() is a helper fn that runs _after_ opening a tile
> index, to test if we can open the first-indexed file (to template some
> parts of the layer).
> See comment here:
> https://github.com/MapServer/MapServer/blob/0d156a72893b1df712cb49c1b3ddaeed5365a68b/mapshape.c#L2028
> See also msTileSHPTryOpen fn here:
> https://github.com/MapServer/MapServer/blob/0d156a72893b1df712cb49c1b3ddaeed5365a68b/mapshape.c#L1868
> 3) So msTiledSHPTryOpen() tries to open the first-indexed file:
> "/datasets/mydataset/v1/layer1/raster_2020-08-02.tif" but returns an error,
> even though it _can_ open that file.
> The problem is, it is using msShapefileOpen() to do this, which in-turn
> uses msSHPOpen() to open the tif file:
> See msShapefileOpen fn:
> https://github.com/MapServer/MapServer/blob/0d156a72893b1df712cb49c1b3ddaeed5365a68b/mapshape.c#L1665
> See opener:
> https://github.com/MapServer/MapServer/blob/0d156a72893b1df712cb49c1b3ddaeed5365a68b/mapshape.c#L1686
> 4) msSHPOpen() assumes whatever file its opening will have a .shp file and
> a .shx file.
> In this case, it's trying to open a .tif file. The function tries to open
> a non-existent /datasets/mydataset/v1/layer1/raster_2020-08-02.shp file.
> See the implementation here:
> https://github.com/MapServer/MapServer/blob/0d156a72893b1df712cb49c1b3ddaeed5365a68b/mapshape.c#L194
> And where it opens a file here:
> https://github.com/MapServer/MapServer/blob/0d156a72893b1df712cb49c1b3ddaeed5365a68b/mapshape.c#L252
>
> So I believe it is a bug to use msShapefileOpen() to test if a tileindex
> can open a file at its first-indexed location, because that file may not be
> a .shp file.
>
> I might be able to put in a PR to fix the bug, but I don't know the
> codebase well enough to not potentially introduce regressions.
> Let me know what you think?
>
> - Ashley Sommer
> _______________________________________________
> MapServer-users mailing list
> MapServer-users at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/mapserver-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/mapserver-users/attachments/20210914/bf565917/attachment.html>