[gdal-dev] Cascaded VRTs

Fabian Schindler fabian.schindler at eox.at
Tue Aug 11 07:35:48 PDT 2015 

Hi list,

Premise: I’m tinkering to add support for datasets from multiple sources (like different subdatasets) in EOxServer (Python, MapScript, …)

Versions: GDAL 1.9.2, MapServer 6.2.2

The sample dataset I try visualize is a MODIS/HDF 4 file with 7+ subdatasets (see [1] for a description for the whole file HDF 4 and [2] for a description of one of the subdatasets).

The subdatasets are not georeferenced, but are supplied with GCPs (and geolocation arrays, which I have not yet tried).


The bands from the dataset are coming from different subdatasets, so I create a VRT where I collect all the subdatasets and add them as bands to the VRT [3]. Since MapServer is not capable of dealing with GCPs I need to create a rectified VRT (created with the CreateWarpedVRT function) [4].


This works quite well, unless I’m in a multithreaded environment (I tested with both apache mod_wsgi with 1 process and 10 threads and Djangos dev server). In this case, when I request more than one WMS request at once I get the error message:

`HDF4_EOS:EOS_SWATH:"/var/eoxserver/autotest/MOD11_L2.A2015001.0015.005.2015005162430.hdf":MOD_Swath_LST:Emis_32' does not exist in the file system, and is not recognised as a supported dataset name.

Once I get this error message, ALL further requests are also running into this error and I don’t get any reasonable response unless I restart the server.

Further details: We used the WarpedVRT method earlier to great success when all the sources come from a single dataset (TIFF with GCPs) and I never ran into this problem.

My questions:

	* Is this approach feasible?
	* Is it possible to just have a single VRT for both collecting the datasets into a single VRT and having the warping instructions?
	* Any ideas why I get the error message? Any chance to get around this?

Regards,
Fabian


[1]

$ gdalinfo /var/eoxserver/autotest/MOD11_L2.A2015001.0015.005.2015005162430.hdf
Driver: HDF4/Hierarchical Data Format Release 4
Files: /var/eoxserver/autotest/MOD11_L2.A2015001.0015.005.2015005162430.hdf
Size is 512, 512
Coordinate System is `'
Metadata:
 ...
Subdatasets:
  SUBDATASET_1_NAME=HDF4_EOS:EOS_SWATH:"/var/eoxserver/autotest/MOD11_L2.A2015001.0015.005.2015005162430.hdf":MOD_Swath_LST:LST
  SUBDATASET_1_DESC=[2030x1354] LST MOD_Swath_LST (16-bit unsigned integer)
  SUBDATASET_2_NAME=HDF4_EOS:EOS_SWATH:"/var/eoxserver/autotest/MOD11_L2.A2015001.0015.005.2015005162430.hdf":MOD_Swath_LST:QC
  SUBDATASET_2_DESC=[2030x1354] QC MOD_Swath_LST (16-bit unsigned integer)
  SUBDATASET_3_NAME=HDF4_EOS:EOS_SWATH:"/var/eoxserver/autotest/MOD11_L2.A2015001.0015.005.2015005162430.hdf":MOD_Swath_LST:Error_LST
  SUBDATASET_3_DESC=[2030x1354] Error_LST MOD_Swath_LST (8-bit unsigned integer)
  SUBDATASET_4_NAME=HDF4_EOS:EOS_SWATH:"/var/eoxserver/autotest/MOD11_L2.A2015001.0015.005.2015005162430.hdf":MOD_Swath_LST:Emis_31
  SUBDATASET_4_DESC=[2030x1354] Emis_31 MOD_Swath_LST (8-bit unsigned integer)
  SUBDATASET_5_NAME=HDF4_EOS:EOS_SWATH:"/var/eoxserver/autotest/MOD11_L2.A2015001.0015.005.2015005162430.hdf":MOD_Swath_LST:Emis_32
  SUBDATASET_5_DESC=[2030x1354] Emis_32 MOD_Swath_LST (8-bit unsigned integer)
  SUBDATASET_6_NAME=HDF4_EOS:EOS_SWATH:"/var/eoxserver/autotest/MOD11_L2.A2015001.0015.005.2015005162430.hdf":MOD_Swath_LST:View_angle
  SUBDATASET_6_DESC=[2030x1354] View_angle MOD_Swath_LST (8-bit unsigned integer)
  SUBDATASET_7_NAME=HDF4_EOS:EOS_SWATH:"/var/eoxserver/autotest/MOD11_L2.A2015001.0015.005.2015005162430.hdf":MOD_Swath_LST:View_time
  SUBDATASET_7_DESC=[2030x1354] View_time MOD_Swath_LST (8-bit unsigned integer)
  SUBDATASET_8_NAME=HDF4_SDS:UNKNOWN:"/var/eoxserver/autotest/MOD11_L2.A2015001.0015.005.2015005162430.hdf":0
  SUBDATASET_8_DESC=[406x271] Latitude (32-bit floating-point)
  SUBDATASET_9_NAME=HDF4_SDS:UNKNOWN:"/var/eoxserver/autotest/MOD11_L2.A2015001.0015.005.2015005162430.hdf":1
  SUBDATASET_9_DESC=[406x271] Longitude (32-bit floating-point)
  SUBDATASET_10_NAME=HDF4_SDS:UNKNOWN:"/var/eoxserver/autotest/MOD11_L2.A2015001.0015.005.2015005162430.hdf":2
  SUBDATASET_10_DESC=[2030x1354] LST (16-bit unsigned integer)
...
Corner Coordinates:
Upper Left  (    0.0,    0.0)
Lower Left  (    0.0,  512.0)
Upper Right (  512.0,    0.0)
Lower Right (  512.0,  512.0)
Center      (  256.0,  256.0)

[2]

$ gdalinfo HDF4_EOS:EOS_SWATH:"/var/eoxserver/autotest/MOD11_L2.A2015001.0015.005.2015005162430.hdf":MOD_Swath_LST:Emis_32
Driver: HDF4Image/HDF4 Dataset
Files: /var/eoxserver/autotest/MOD11_L2.A2015001.0015.005.2015005162430.hdf
       /var/eoxserver/autotest/MOD11_L2.A2015001.0015.005.2015005162430.hdf.aux.xml
Size is 1354, 2030
Coordinate System is `'
GCP[  0]: Id=, Info=
          (2.5,2.5) -> (-17.7197341918945,16.5706329345703,0)
GCP[  1]: Id=, Info=
          (122.5,2.5) -> (-21.4628753662109,16.2001628875732,0)
GCP[  2]: Id=, Info=
          (242.5,2.5) -> (-23.6760387420654,15.9492874145508,0)
...
GCP[143]: Id=, Info=
          (1322.5,1982.5) -> (-43.3567504882812,30.9182910919189,0)
Metadata:
  ...
Geolocation:
  LINE_OFFSET=2
  LINE_STEP=5
  PIXEL_OFFSET=2
  PIXEL_STEP=5
  X_BAND=1
  X_DATASET=HDF4_EOS:EOS_SWATH_GEOL:"/var/eoxserver/autotest/MOD11_L2.A2015001.0015.005.2015005162430.hdf":MOD_Swath_LST:Longitude
  Y_BAND=1
  Y_DATASET=HDF4_EOS:EOS_SWATH_GEOL:"/var/eoxserver/autotest/MOD11_L2.A2015001.0015.005.2015005162430.hdf":MOD_Swath_LST:Latitude
Corner Coordinates:
Upper Left  (    0.0,    0.0)
Lower Left  (    0.0, 2030.0)
Upper Right ( 1354.0,    0.0)
Lower Right ( 1354.0, 2030.0)
Center      (  677.0, 1015.0)
Band 1 Block=1354x738 Type=Byte, ColorInterp=Gray
  Description = Band 32 emissivity
  NoData Value=0
  Offset: 0.49,   Scale:0.002

[3]

<VRTDataset rasterXSize="1354" rasterYSize="2030">
  <GCPList>
    <GCP Id="" Pixel="2.5000" Line="2.5000" X="-1.771973419189E+01" Y="1.657063293457E+01" />
    <GCP Id="" Pixel="122.5000" Line="2.5000" X="-2.146287536621E+01" Y="1.620016288757E+01" />
...
    <GCP Id="" Pixel="1322.5000" Line="1982.5000" X="-4.335675048828E+01" Y="3.091829109192E+01" />
  </GCPList>
  <VRTRasterBand dataType="UInt16" band="1">
    <SimpleSource>
      <SourceFilename relativeToVRT="0">HDF4_EOS:EOS_SWATH:"/var/eoxserver/autotest/MOD11_L2.A2015001.0015.005.2015005162430.hdf":MOD_Swath_LST:LST</SourceFilename>
      <SourceBand>1</SourceBand>
      <SourceProperties RasterXSize="1354" RasterYSize="2030" DataType="UInt16" BlockXSize="1354" BlockYSize="738" />
    </SimpleSource>
  </VRTRasterBand>
  <VRTRasterBand dataType="UInt16" band="2">
    <SimpleSource>
      <SourceFilename relativeToVRT="0">HDF4_EOS:EOS_SWATH:"/var/eoxserver/autotest/MOD11_L2.A2015001.0015.005.2015005162430.hdf":MOD_Swath_LST:QC</SourceFilename>
      <SourceBand>1</SourceBand>
      <SourceProperties RasterXSize="1354" RasterYSize="2030" DataType="UInt16" BlockXSize="1354" BlockYSize="738" />
    </SimpleSource>
  </VRTRasterBand>
  <VRTRasterBand dataType="Byte" band="3">
    <SimpleSource>
      <SourceFilename relativeToVRT="0">HDF4_EOS:EOS_SWATH:"/var/eoxserver/autotest/MOD11_L2.A2015001.0015.005.2015005162430.hdf":MOD_Swath_LST:Error_LST</SourceFilename>
      <SourceBand>1</SourceBand>
      <SourceProperties RasterXSize="1354" RasterYSize="2030" DataType="Byte" BlockXSize="1354" BlockYSize="738" />
    </SimpleSource>
  </VRTRasterBand>
  <VRTRasterBand dataType="Byte" band="4">
    <SimpleSource>
      <SourceFilename relativeToVRT="0">HDF4_EOS:EOS_SWATH:"/var/eoxserver/autotest/MOD11_L2.A2015001.0015.005.2015005162430.hdf":MOD_Swath_LST:Emis_31</SourceFilename>
      <SourceBand>1</SourceBand>
      <SourceProperties RasterXSize="1354" RasterYSize="2030" DataType="Byte" BlockXSize="1354" BlockYSize="738" />
    </SimpleSource>
  </VRTRasterBand>
  <VRTRasterBand dataType="Byte" band="5">
    <SimpleSource>
      <SourceFilename relativeToVRT="0">HDF4_EOS:EOS_SWATH:"/var/eoxserver/autotest/MOD11_L2.A2015001.0015.005.2015005162430.hdf":MOD_Swath_LST:Emis_32</SourceFilename>
      <SourceBand>1</SourceBand>
      <SourceProperties RasterXSize="1354" RasterYSize="2030" DataType="Byte" BlockXSize="1354" BlockYSize="738" />
    </SimpleSource>
  </VRTRasterBand>
  <VRTRasterBand dataType="Byte" band="6">
    <SimpleSource>
      <SourceFilename relativeToVRT="0">HDF4_EOS:EOS_SWATH:"/var/eoxserver/autotest/MOD11_L2.A2015001.0015.005.2015005162430.hdf":MOD_Swath_LST:View_angle</SourceFilename>
      <SourceBand>1</SourceBand>
      <SourceProperties RasterXSize="1354" RasterYSize="2030" DataType="Byte" BlockXSize="1354" BlockYSize="738" />
    </SimpleSource>
  </VRTRasterBand>
  <VRTRasterBand dataType="Byte" band="7">
    <SimpleSource>
      <SourceFilename relativeToVRT="0">HDF4_EOS:EOS_SWATH:"/var/eoxserver/autotest/MOD11_L2.A2015001.0015.005.2015005162430.hdf":MOD_Swath_LST:View_time</SourceFilename>
      <SourceBand>1</SourceBand>
      <SourceProperties RasterXSize="1354" RasterYSize="2030" DataType="Byte" BlockXSize="1354" BlockYSize="738" />
    </SimpleSource>
  </VRTRasterBand>
</VRTDataset>

[4]

<VRTDataset rasterXSize="2029" rasterYSize="1820" subClass="VRTWarpedDataset">
  <GeoTransform> -4.2266212922257814e+01,  1.1322693568940058e-02,  0.0000000000000000e+00,  3.4413385891072409e+01,  0.0000000000000000e+00, -1.1322693568940058e-02</GeoTransform>
  <VRTRasterBand dataType="UInt16" band="1" subClass="VRTWarpedRasterBand">
    <Metadata />
  </VRTRasterBand>
  <VRTRasterBand dataType="UInt16" band="2" subClass="VRTWarpedRasterBand">
    <Metadata />
  </VRTRasterBand>
  <VRTRasterBand dataType="Byte" band="3" subClass="VRTWarpedRasterBand">
    <Metadata />
  </VRTRasterBand>
  <VRTRasterBand dataType="Byte" band="4" subClass="VRTWarpedRasterBand">
    <Metadata />
  </VRTRasterBand>
  <VRTRasterBand dataType="Byte" band="5" subClass="VRTWarpedRasterBand">
    <Metadata />
  </VRTRasterBand>
  <VRTRasterBand dataType="Byte" band="6" subClass="VRTWarpedRasterBand">
    <Metadata />
  </VRTRasterBand>
  <VRTRasterBand dataType="Byte" band="7" subClass="VRTWarpedRasterBand">
    <Metadata />
  </VRTRasterBand>
  <BlockXSize>512</BlockXSize>
  <BlockYSize>128</BlockYSize>
  <GDALWarpOptions>
    <WarpMemoryLimit>6.71089e+07</WarpMemoryLimit>
    <ResampleAlg>NearestNeighbour</ResampleAlg>
    <WorkingDataType>UInt16</WorkingDataType>
    <SourceDataset relativeToVRT="1">ff71b2c70ab940d6b35ecaa6c4b100e6</SourceDataset>
    <Transformer>
      <ApproxTransformer>
        <MaxError>0.125</MaxError>
        <BaseTransformer>
          <GenImgProjTransformer>
            <SrcGCPTransformer>
              <GCPTransformer>
                <Order>2</Order>
                <Reversed>0</Reversed>
                <GCPList>
                  <GCP Id="" Pixel="2.5000" Line="2.5000" X="-1.771973419189E+01" Y="1.657063293457E+01" />
                  <GCP Id="" Pixel="122.5000" Line="2.5000" X="-2.146287536621E+01" Y="1.620016288757E+01" />
                  ...
                  <GCP Id="" Pixel="1322.5000" Line="1982.5000" X="-4.335675048828E+01" Y="3.091829109192E+01" />
                </GCPList>
              </GCPTransformer>
            </SrcGCPTransformer>
            <DstGeoTransform>-42.26621292225781,0.01132269356894006,0,34.41338589107241,0,-0.01132269356894006</DstGeoTransform>
            <DstInvGeoTransform>3732.876162806413,88.31820749288477,0,3039.328555660446,0,-88.31820749288477</DstInvGeoTransform>
          </GenImgProjTransformer>
        </BaseTransformer>
      </ApproxTransformer>
    </Transformer>
    <BandList>
      <BandMapping src="1" dst="1" />
      <BandMapping src="2" dst="2" />
      <BandMapping src="3" dst="3" />
      <BandMapping src="4" dst="4" />
      <BandMapping src="5" dst="5" />
      <BandMapping src="6" dst="6" />
      <BandMapping src="7" dst="7" />
    </BandList>
  </GDALWarpOptions>
</VRTDataset>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20150811/3acd2b63/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 455 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20150811/3acd2b63/attachment-0001.pgp>

More information about the gdal-dev mailing list