[gdal-dev] Cascaded VRTs
Even Rouault
even.rouault at spatialys.com
Tue Aug 11 10:08:26 PDT 2015
Le mardi 11 août 2015 16:35:48, Fabian Schindler a écrit :
> Hi list,
>
> Premise: I’m tinkering to add support for datasets from multiple sources
> (like different subdatasets) in EOxServer (Python, MapScript, …)
>
> Versions: GDAL 1.9.2, MapServer 6.2.2
>
> The sample dataset I try visualize is a MODIS/HDF 4 file with 7+
> subdatasets (see [1] for a description for the whole file HDF 4 and [2]
> for a description of one of the subdatasets).
>
> The subdatasets are not georeferenced, but are supplied with GCPs (and
> geolocation arrays, which I have not yet tried).
>
>
> The bands from the dataset are coming from different subdatasets, so I
> create a VRT where I collect all the subdatasets and add them as bands to
> the VRT [3]. Since MapServer is not capable of dealing with GCPs I need to
> create a rectified VRT (created with the CreateWarpedVRT function) [4].
>
>
> This works quite well, unless I’m in a multithreaded environment (I tested
> with both apache mod_wsgi with 1 process and 10 threads and Djangos dev
> server). In this case, when I request more than one WMS request at once I
> get the error message:
>
> `HDF4_EOS:EOS_SWATH:"/var/eoxserver/autotest/MOD11_L2.A2015001.0015.005.201
> 5005162430.hdf":MOD_Swath_LST:Emis_32' does not exist in the file system,
> and is not recognised as a supported dataset name.
>
> Once I get this error message, ALL further requests are also running into
> this error and I don’t get any reasonable response unless I restart the
> server.
>
> Further details: We used the WarpedVRT method earlier to great success when
> all the sources come from a single dataset (TIFF with GCPs) and I never
> ran into this problem.
>
> My questions:
>
> * Is this approach feasible?
> * Is it possible to just have a single VRT for both collecting the
> datasets into a single VRT and having the warping instructions? * Any
> ideas why I get the error message? Any chance to get around this?
Hi Fabian,
recently I discovered an issue with how VRT and multi-threading played
together. A first fix was pushed in GDAL 2.0.0. See following news entry :
"""
* add handling of a shared='0' attribute on <SourceFilename> to open sources
in non-shared mode, and VRT_SHARED_SOURCE config option that can be set to 0 in
case the shared attribute isn't there (#5992)
"""
I pushed although a related completementary fix in the 2.0 branch after 2.0.0
release. So I'd recommend you to try the 2.0 branch and define
VRT_SHARED_SOURCE=0 as configuration option (or add a shared='0' attribute on
all SourceFilename elements of your VRT)
I'd note that the HDF4 libraries aren't thread-safe, so in recent GDAL
versions (1.11 for sure), there's a global GDAL mutex in the HDF4 driver to
hopefully avoid issues (this hasn't been strongly checked, so I wouldn't
exclude there are remaining issues).
Regarding how to have a single VRT for both collecting the datasets into a
single VRT and having the warping instructions, you could try to put as text
of the SourceDataset element of the warped VRT the XML escaped text of the
"collecting VRT". Something like
<SourceDataset><VRTDataset rasterXSize=1354"
rasterYSize="2030">.....</VRTDataset></SourceDataset>
![CDATA[...]]> syntax also works.
Even
>
> Regards,
> Fabian
>
>
> [1]
>
> $ gdalinfo
> /var/eoxserver/autotest/MOD11_L2.A2015001.0015.005.2015005162430.hdf
> Driver: HDF4/Hierarchical Data Format Release 4
> Files: /var/eoxserver/autotest/MOD11_L2.A2015001.0015.005.2015005162430.hdf
> Size is 512, 512
> Coordinate System is `'
> Metadata:
> ...
> Subdatasets:
>
> SUBDATASET_1_NAME=HDF4_EOS:EOS_SWATH:"/var/eoxserver/autotest/MOD11_L2.A20
> 15001.0015.005.2015005162430.hdf":MOD_Swath_LST:LST
> SUBDATASET_1_DESC=[2030x1354] LST MOD_Swath_LST (16-bit unsigned integer)
> SUBDATASET_2_NAME=HDF4_EOS:EOS_SWATH:"/var/eoxserver/autotest/MOD11_L2.A20
> 15001.0015.005.2015005162430.hdf":MOD_Swath_LST:QC
> SUBDATASET_2_DESC=[2030x1354] QC MOD_Swath_LST (16-bit unsigned integer)
> SUBDATASET_3_NAME=HDF4_EOS:EOS_SWATH:"/var/eoxserver/autotest/MOD11_L2.A20
> 15001.0015.005.2015005162430.hdf":MOD_Swath_LST:Error_LST
> SUBDATASET_3_DESC=[2030x1354] Error_LST MOD_Swath_LST (8-bit unsigned
> integer)
> SUBDATASET_4_NAME=HDF4_EOS:EOS_SWATH:"/var/eoxserver/autotest/MOD11_L2.A20
> 15001.0015.005.2015005162430.hdf":MOD_Swath_LST:Emis_31
> SUBDATASET_4_DESC=[2030x1354] Emis_31 MOD_Swath_LST (8-bit unsigned
> integer)
> SUBDATASET_5_NAME=HDF4_EOS:EOS_SWATH:"/var/eoxserver/autotest/MOD11_L2.A20
> 15001.0015.005.2015005162430.hdf":MOD_Swath_LST:Emis_32
> SUBDATASET_5_DESC=[2030x1354] Emis_32 MOD_Swath_LST (8-bit unsigned
> integer)
> SUBDATASET_6_NAME=HDF4_EOS:EOS_SWATH:"/var/eoxserver/autotest/MOD11_L2.A20
> 15001.0015.005.2015005162430.hdf":MOD_Swath_LST:View_angle
> SUBDATASET_6_DESC=[2030x1354] View_angle MOD_Swath_LST (8-bit unsigned
> integer)
> SUBDATASET_7_NAME=HDF4_EOS:EOS_SWATH:"/var/eoxserver/autotest/MOD11_L2.A20
> 15001.0015.005.2015005162430.hdf":MOD_Swath_LST:View_time
> SUBDATASET_7_DESC=[2030x1354] View_time MOD_Swath_LST (8-bit unsigned
> integer)
> SUBDATASET_8_NAME=HDF4_SDS:UNKNOWN:"/var/eoxserver/autotest/MOD11_L2.A2015
> 001.0015.005.2015005162430.hdf":0 SUBDATASET_8_DESC=[406x271] Latitude
> (32-bit floating-point)
>
> SUBDATASET_9_NAME=HDF4_SDS:UNKNOWN:"/var/eoxserver/autotest/MOD11_L2.A2015
> 001.0015.005.2015005162430.hdf":1 SUBDATASET_9_DESC=[406x271] Longitude
> (32-bit floating-point)
>
> SUBDATASET_10_NAME=HDF4_SDS:UNKNOWN:"/var/eoxserver/autotest/MOD11_L2.A201
> 5001.0015.005.2015005162430.hdf":2 SUBDATASET_10_DESC=[2030x1354] LST
> (16-bit unsigned integer)
> ...
> Corner Coordinates:
> Upper Left ( 0.0, 0.0)
> Lower Left ( 0.0, 512.0)
> Upper Right ( 512.0, 0.0)
> Lower Right ( 512.0, 512.0)
> Center ( 256.0, 256.0)
>
> [2]
>
> $ gdalinfo
> HDF4_EOS:EOS_SWATH:"/var/eoxserver/autotest/MOD11_L2.A2015001.0015.005.201
> 5005162430.hdf":MOD_Swath_LST:Emis_32 Driver: HDF4Image/HDF4 Dataset
> Files: /var/eoxserver/autotest/MOD11_L2.A2015001.0015.005.2015005162430.hdf
>
> /var/eoxserver/autotest/MOD11_L2.A2015001.0015.005.2015005162430.hdf.aux.x
> ml Size is 1354, 2030
> Coordinate System is `'
> GCP[ 0]: Id=, Info=
> (2.5,2.5) -> (-17.7197341918945,16.5706329345703,0)
> GCP[ 1]: Id=, Info=
> (122.5,2.5) -> (-21.4628753662109,16.2001628875732,0)
> GCP[ 2]: Id=, Info=
> (242.5,2.5) -> (-23.6760387420654,15.9492874145508,0)
> ...
> GCP[143]: Id=, Info=
> (1322.5,1982.5) -> (-43.3567504882812,30.9182910919189,0)
> Metadata:
> ...
> Geolocation:
> LINE_OFFSET=2
> LINE_STEP=5
> PIXEL_OFFSET=2
> PIXEL_STEP=5
> X_BAND=1
>
> X_DATASET=HDF4_EOS:EOS_SWATH_GEOL:"/var/eoxserver/autotest/MOD11_L2.A20150
> 01.0015.005.2015005162430.hdf":MOD_Swath_LST:Longitude Y_BAND=1
>
> Y_DATASET=HDF4_EOS:EOS_SWATH_GEOL:"/var/eoxserver/autotest/MOD11_L2.A20150
> 01.0015.005.2015005162430.hdf":MOD_Swath_LST:Latitude Corner Coordinates:
> Upper Left ( 0.0, 0.0)
> Lower Left ( 0.0, 2030.0)
> Upper Right ( 1354.0, 0.0)
> Lower Right ( 1354.0, 2030.0)
> Center ( 677.0, 1015.0)
> Band 1 Block=1354x738 Type=Byte, ColorInterp=Gray
> Description = Band 32 emissivity
> NoData Value=0
> Offset: 0.49, Scale:0.002
>
> [3]
>
> <VRTDataset rasterXSize="1354" rasterYSize="2030">
> <GCPList>
> <GCP Id="" Pixel="2.5000" Line="2.5000" X="-1.771973419189E+01"
> Y="1.657063293457E+01" /> <GCP Id="" Pixel="122.5000" Line="2.5000"
> X="-2.146287536621E+01" Y="1.620016288757E+01" /> ...
> <GCP Id="" Pixel="1322.5000" Line="1982.5000" X="-4.335675048828E+01"
> Y="3.091829109192E+01" /> </GCPList>
> <VRTRasterBand dataType="UInt16" band="1">
> <SimpleSource>
> <SourceFilename
> relativeToVRT="0">HDF4_EOS:EOS_SWATH:"/var/eoxserver/autotest/MOD11_L
> 2.A2015001.0015.005.2015005162430.hdf":MOD_Swath_LST:LST</SourceFilena
> me> <SourceBand>1</SourceBand>
> <SourceProperties RasterXSize="1354" RasterYSize="2030"
> DataType="UInt16" BlockXSize="1354" BlockYSize="738" /> </SimpleSource>
> </VRTRasterBand>
> <VRTRasterBand dataType="UInt16" band="2">
> <SimpleSource>
> <SourceFilename
> relativeToVRT="0">HDF4_EOS:EOS_SWATH:"/var/eoxserver/autotest/MOD11_L
> 2.A2015001.0015.005.2015005162430.hdf":MOD_Swath_LST:QC</SourceFilenam
> e> <SourceBand>1</SourceBand>
> <SourceProperties RasterXSize="1354" RasterYSize="2030"
> DataType="UInt16" BlockXSize="1354" BlockYSize="738" /> </SimpleSource>
> </VRTRasterBand>
> <VRTRasterBand dataType="Byte" band="3">
> <SimpleSource>
> <SourceFilename
> relativeToVRT="0">HDF4_EOS:EOS_SWATH:"/var/eoxserver/autotest/MOD11_L
> 2.A2015001.0015.005.2015005162430.hdf":MOD_Swath_LST:Error_LST</Source
> Filename> <SourceBand>1</SourceBand>
> <SourceProperties RasterXSize="1354" RasterYSize="2030"
> DataType="Byte" BlockXSize="1354" BlockYSize="738" /> </SimpleSource>
> </VRTRasterBand>
> <VRTRasterBand dataType="Byte" band="4">
> <SimpleSource>
> <SourceFilename
> relativeToVRT="0">HDF4_EOS:EOS_SWATH:"/var/eoxserver/autotest/MOD11_L
> 2.A2015001.0015.005.2015005162430.hdf":MOD_Swath_LST:Emis_31</SourceFi
> lename> <SourceBand>1</SourceBand>
> <SourceProperties RasterXSize="1354" RasterYSize="2030"
> DataType="Byte" BlockXSize="1354" BlockYSize="738" /> </SimpleSource>
> </VRTRasterBand>
> <VRTRasterBand dataType="Byte" band="5">
> <SimpleSource>
> <SourceFilename
> relativeToVRT="0">HDF4_EOS:EOS_SWATH:"/var/eoxserver/autotest/MOD11_L
> 2.A2015001.0015.005.2015005162430.hdf":MOD_Swath_LST:Emis_32</SourceFi
> lename> <SourceBand>1</SourceBand>
> <SourceProperties RasterXSize="1354" RasterYSize="2030"
> DataType="Byte" BlockXSize="1354" BlockYSize="738" /> </SimpleSource>
> </VRTRasterBand>
> <VRTRasterBand dataType="Byte" band="6">
> <SimpleSource>
> <SourceFilename
> relativeToVRT="0">HDF4_EOS:EOS_SWATH:"/var/eoxserver/autotest/MOD11_L
> 2.A2015001.0015.005.2015005162430.hdf":MOD_Swath_LST:View_angle</Sourc
> eFilename> <SourceBand>1</SourceBand>
> <SourceProperties RasterXSize="1354" RasterYSize="2030"
> DataType="Byte" BlockXSize="1354" BlockYSize="738" /> </SimpleSource>
> </VRTRasterBand>
> <VRTRasterBand dataType="Byte" band="7">
> <SimpleSource>
> <SourceFilename
> relativeToVRT="0">HDF4_EOS:EOS_SWATH:"/var/eoxserver/autotest/MOD11_L
> 2.A2015001.0015.005.2015005162430.hdf":MOD_Swath_LST:View_time</Source
> Filename> <SourceBand>1</SourceBand>
> <SourceProperties RasterXSize="1354" RasterYSize="2030"
> DataType="Byte" BlockXSize="1354" BlockYSize="738" /> </SimpleSource>
> </VRTRasterBand>
> </VRTDataset>
>
> [4]
>
> <VRTDataset rasterXSize="2029" rasterYSize="1820"
> subClass="VRTWarpedDataset"> <GeoTransform> -4.2266212922257814e+01,
> 1.1322693568940058e-02, 0.0000000000000000e+00, 3.4413385891072409e+01,
> 0.0000000000000000e+00, -1.1322693568940058e-02</GeoTransform>
> <VRTRasterBand dataType="UInt16" band="1" subClass="VRTWarpedRasterBand">
> <Metadata />
> </VRTRasterBand>
> <VRTRasterBand dataType="UInt16" band="2" subClass="VRTWarpedRasterBand">
> <Metadata />
> </VRTRasterBand>
> <VRTRasterBand dataType="Byte" band="3" subClass="VRTWarpedRasterBand">
> <Metadata />
> </VRTRasterBand>
> <VRTRasterBand dataType="Byte" band="4" subClass="VRTWarpedRasterBand">
> <Metadata />
> </VRTRasterBand>
> <VRTRasterBand dataType="Byte" band="5" subClass="VRTWarpedRasterBand">
> <Metadata />
> </VRTRasterBand>
> <VRTRasterBand dataType="Byte" band="6" subClass="VRTWarpedRasterBand">
> <Metadata />
> </VRTRasterBand>
> <VRTRasterBand dataType="Byte" band="7" subClass="VRTWarpedRasterBand">
> <Metadata />
> </VRTRasterBand>
> <BlockXSize>512</BlockXSize>
> <BlockYSize>128</BlockYSize>
> <GDALWarpOptions>
> <WarpMemoryLimit>6.71089e+07</WarpMemoryLimit>
> <ResampleAlg>NearestNeighbour</ResampleAlg>
> <WorkingDataType>UInt16</WorkingDataType>
> <SourceDataset
> relativeToVRT="1">ff71b2c70ab940d6b35ecaa6c4b100e6</SourceDataset>
> <Transformer>
> <ApproxTransformer>
> <MaxError>0.125</MaxError>
> <BaseTransformer>
> <GenImgProjTransformer>
> <SrcGCPTransformer>
> <GCPTransformer>
> <Order>2</Order>
> <Reversed>0</Reversed>
> <GCPList>
> <GCP Id="" Pixel="2.5000" Line="2.5000"
> X="-1.771973419189E+01" Y="1.657063293457E+01" /> <GCP Id=""
> Pixel="122.5000" Line="2.5000" X="-2.146287536621E+01"
> Y="1.620016288757E+01" /> ...
> <GCP Id="" Pixel="1322.5000" Line="1982.5000"
> X="-4.335675048828E+01" Y="3.091829109192E+01" /> </GCPList>
> </GCPTransformer>
> </SrcGCPTransformer>
>
> <DstGeoTransform>-42.26621292225781,0.01132269356894006,0,34.4133858910724
> 1,0,-0.01132269356894006</DstGeoTransform>
> <DstInvGeoTransform>3732.876162806413,88.31820749288477,0,3039.32855566044
> 6,0,-88.31820749288477</DstInvGeoTransform> </GenImgProjTransformer>
> </BaseTransformer>
> </ApproxTransformer>
> </Transformer>
> <BandList>
> <BandMapping src="1" dst="1" />
> <BandMapping src="2" dst="2" />
> <BandMapping src="3" dst="3" />
> <BandMapping src="4" dst="4" />
> <BandMapping src="5" dst="5" />
> <BandMapping src="6" dst="6" />
> <BandMapping src="7" dst="7" />
> </BandList>
> </GDALWarpOptions>
> </VRTDataset>
--
Spatialys - Geospatial professional services
http://www.spatialys.com
More information about the gdal-dev
mailing list