<div dir="ltr">Oh wow, that's fantastic - thanks so much. Lots for me to learn from there too, with the 

HDF5EOS GRID details. <div><br></div><div>Cheers, Mike</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Apr 30, 2024 at 6:08 AM Even Rouault <<a href="mailto:even.rouault@spatialys.com">even.rouault@spatialys.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><u></u>

  
    
  
  <div>
    <p>Michael,</p>
    <p>actually support for those products was already there at 95%
      since they use the HDF5EOS GRID formalism which we support since
      3.7. But that particular type of products has an oddity. They have
      an empty <u></u>GROUP=Dimension<u></u>
      within the GROUP=GridStructure in the HDF5EOS metadata, which our
      parser didn't like as we use the information in this group to map
      dimension names to dimension index and size. Fixed/worked around
      in <a href="https://github.com/OSGeo/gdal/pull/9807" target="_blank">https://github.com/OSGeo/gdal/pull/9807</a> . Hard to tell if those
      products are in fault given that the HDF5EOS specification at
<a href="https://www.earthdata.nasa.gov/s3fs-public/imported/ESDS-RFC-008-v1.1.pdf" target="_blank">https://www.earthdata.nasa.gov/s3fs-public/imported/ESDS-RFC-008-v1.1.pdf</a>
      , appendix A, is extremely minimum regarding the HDF5EOS metadata
      itself (for once, I complain for a spec to be too minimal !), and
      the example they give for grids doesn't make sense to me (the
      declared dimension names and sizes have nothing to do with the
      ones use in the grid)</p>
    <p>I now get:</p>
    <p>$ gdalinfo
HDF5:"AMSR_U2_L3_SeaIce12km_B04_20120702.he5"://HDFEOS/GRIDS/SpPolarGrid12km/Data_Fields/SI_12km_SH_18H_ASC<br>
      Driver: HDF5Image/HDF5 Dataset<br>
      Files: AMSR_U2_L3_SeaIce12km_B04_20120702.he5<br>
      Size is 632, 664<br>
      Coordinate System is:<br>
      PROJCRS["unnamed",<br>
          BASEGEOGCRS["Unknown datum based upon the custom spheroid",<br>
              DATUM["Not specified (based on custom spheroid)",<br>
                  ELLIPSOID["Custom spheroid",6378273,0,<br>
                      LENGTHUNIT["metre",1,<br>
                          ID["EPSG",9001]]]],<br>
              PRIMEM["Greenwich",0,<br>
                  ANGLEUNIT["degree",0.0174532925199433,<br>
                      ID["EPSG",9122]]]],<br>
          CONVERSION["Polar Stereographic (variant B)",<br>
              METHOD["Polar Stereographic (variant B)",<br>
                  ID["EPSG",9829]],<br>
              PARAMETER["Latitude of standard parallel",-70,<br>
                  ANGLEUNIT["degree",0.0174532925199433],<br>
                  ID["EPSG",8832]],<br>
              PARAMETER["Longitude of origin",0,<br>
                  ANGLEUNIT["degree",0.0174532925199433],<br>
                  ID["EPSG",8833]],<br>
              PARAMETER["False easting",0,<br>
                  LENGTHUNIT["metre",1],<br>
                  ID["EPSG",8806]],<br>
              PARAMETER["False northing",0,<br>
                  LENGTHUNIT["metre",1],<br>
                  ID["EPSG",8807]]],<br>
          CS[Cartesian,2],<br>
              AXIS["(E)",north,<br>
                  MERIDIAN[90,<br>
                      ANGLEUNIT["degree",0.0174532925199433,<br>
                          ID["EPSG",9122]]],<br>
                  ORDER[1],<br>
                  LENGTHUNIT["Meter",1]],<br>
              AXIS["(N)",north,<br>
                  MERIDIAN[0,<br>
                      ANGLEUNIT["degree",0.0174532925199433,<br>
                          ID["EPSG",9122]]],<br>
                  ORDER[2],<br>
                  LENGTHUNIT["Meter",1]]]<br>
      Data axis to CRS axis mapping: 1,2<br>
      Origin = (-3950000.000000000000000,4350000.000000000000000)<br>
      Pixel Size = (12500.000000000000000,-12500.000000000000000)<br>
      Metadata:<br>
        Conventions=CF-1.6<br>
        HDFEOS_INFORMATION_HDFEOSVersion=HDFEOS_5.1.15<br>
        history=This version of the Sea Ice processing code contains
      updates provided by the science team on September 16, 2019. For
      details on these updates, see the release notes provided in the
      DAP.<br>
        institution=NASA's AMSR Science Investigator-led Processing
      System (SIPS)<br>
        references=Please cite these data as: Markus, T., J. C. Comiso,
      and W. N. Meier. 2018. AMSR-E/AMSR2 Unified L3 Daily 12.5 km
      Brightness Temperatures, Sea Ice Concentration, Motion & Snow
      Depth Polar Grids, Version 1. [Indicate subset used]. Boulder,
      Colorado USA. NASA National Snow and Ice Data Center Distributed
      Active Archive Center. doi: <a href="https://doi.org/10.5067/RA1MIJOYPK3P" target="_blank">https://doi.org/10.5067/RA1MIJOYPK3P</a>.<br>
        source=satellite observation<br>
        title=AMSR-E/AMSR2 Unified L3 Daily 12.5 km Brightness
      Temperatures, Sea Ice Concentration, Motion & Snow Depth Polar
      Grids<br>
      Corner Coordinates:<br>
      Upper Left  (-3950000.000, 4350000.000) ( 42d14'27.21"W,
      39d11'27.54"S)<br>
      Lower Left  (-3950000.000,-3950000.000) (135d 0' 0.00"W,
      41d23'59.41"S)<br>
      Upper Right ( 3950000.000, 4350000.000) ( 42d14'27.21"E,
      39d11'27.54"S)<br>
      Lower Right ( 3950000.000,-3950000.000) (135d 0' 0.00"E,
      41d23'59.41"S)<br>
      Center      (       0.000,  200000.000) (  0d 0' 0.01"E, 88d
      8'51.76"S)<br>
      Band 1 Block=632x1 Type=Int32, ColorInterp=Undefined<br>
        NoData Value=0<br>
        Metadata:<br>
          add_offset=0<br>
          coordinates=lon lat<br>
          long_name=18.7 GHz horizontal daily average ascending Tbs<br>
          packing_convention=netCDF<br>
          packing_convention_description=unpacked = scale_factor x
      packed + add_offset<br>
          scale_factor=0.1<br>
          standard_name=brightness_temperature<br>
          units=degree_kelvin<br>
          _FillValue=0<br>
      <br>
    </p>
    <p>Even<br>
    </p>
    <div>Le 29/04/2024 à 21:10, Michael Sumner
      via gdal-dev a écrit :<br>
    </div>
    <blockquote type="cite">
      
      <div dir="ltr">This HDF5 (requires earthdata credentials your
        "Authorization: Bearer <token>" in GDAL_HTTP_HEADERS, or
        equiv) presents without geolocation arrays. 
        <div><br>
        </div>
        <div>gdalinfo "/vsicurl/<a href="https://n5eil01u.ecs.nsidc.org/AMSA/AU_SI12.001/2012.07.02/AMSR_U2_L3_SeaIce12km_B04_20120702.he5" target="_blank">https://n5eil01u.ecs.nsidc.org/AMSA/AU_SI12.001/2012.07.02/AMSR_U2_L3_SeaIce12km_B04_20120702.he5</a>"
          -sd 26<br>
          Driver: HDF5Image/HDF5 Dataset<br>
          Files: /vsicurl/<a href="https://n5eil01u.ecs.nsidc.org/AMSA/AU_SI12.001/2012.07.02/AMSR_U2_L3_SeaIce12km_B04_20120702.he5" target="_blank">https://n5eil01u.ecs.nsidc.org/AMSA/AU_SI12.001/2012.07.02/AMSR_U2_L3_SeaIce12km_B04_20120702.he5</a><br>
          Size is 608, 896<br>
          Metadata:<br>
            Conventions=CF-1.6<br>
            HDFEOS_INFORMATION_HDFEOSVersion=HDFEOS_5.1.15<br>
            history=This version of the Sea Ice processing code contains
          updates provided by the science team on September 16, 2019.
          For details on these updates, see the release notes provided
          in the DAP.<br>
            institution=NASA's AMSR Science Investigator-led Processing
          System (SIPS)<br>
            references=Please cite these data as: Markus, T., J. C.
          Comiso, and W. N. Meier. 2018. AMSR-E/AMSR2 Unified L3 Daily
          12.5 km Brightness Temperatures, Sea Ice Concentration, Motion
          & Snow Depth Polar Grids, Version 1. [Indicate subset
          used]. Boulder, Colorado USA. NASA National Snow and Ice Data
          Center Distributed Active Archive Center. doi: <a href="https://doi.org/10.5067/RA1MIJOYPK3P" target="_blank">https://doi.org/10.5067/RA1MIJOYPK3P</a>.<br>
            source=satellite observation<br>
            title=AMSR-E/AMSR2 Unified L3 Daily 12.5 km Brightness
          Temperatures, Sea Ice Concentration, Motion & Snow Depth
          Polar Grids<br>
          Corner Coordinates:<br>
          Upper Left  (    0.0,    0.0)<br>
          Lower Left  (    0.0,  896.0)<br>
          Upper Right (  608.0,    0.0)<br>
          Lower Right (  608.0,  896.0)<br>
          Center      (  304.0,  448.0)<br>
          Band 1 Block=608x1 Type=Int32, ColorInterp=Undefined<br>
            Metadata:<br>
              comment=data value meaning: 0 -- Open Water, 110 --
          missing/not calculated, 120 -- Land<br>
              coordinates=lon lat<br>
              long_name=Sea ice concentration daily average<br>
              units=percent<br>
          <div><br>
          </div>
          <div><br>
          </div>
          <div><br>
          </div>
          <div>gdalinfo --version<br>
            GDAL 3.9.0dev-cb4d30f56d, released 2024/04/15 <br>
          </div>
          <div><br>
          </div>
          <div>The geolocation arrays are sds 33 and 32 respectively: </div>
          <div><br>
          </div>
          <div>HDF5:"/vsicurl/<a href="https://n5eil01u.ecs.nsidc.org/AMSA/AU_SI12.001/2012.07.02/AMSR_U2_L3_SeaIce12km_B04_20120702.he5" target="_blank">https://n5eil01u.ecs.nsidc.org/AMSA/AU_SI12.001/2012.07.02/AMSR_U2_L3_SeaIce12km_B04_20120702.he5</a>"://HDFEOS/GRIDS/NpPolarGrid12km/lon<br>
          </div>
          <div><br>
          </div>
          <div>HDF5:"/vsicurl/<a href="https://n5eil01u.ecs.nsidc.org/AMSA/AU_SI12.001/2012.07.02/AMSR_U2_L3_SeaIce12km_B04_20120702.he5" target="_blank">https://n5eil01u.ecs.nsidc.org/AMSA/AU_SI12.001/2012.07.02/AMSR_U2_L3_SeaIce12km_B04_20120702.he5</a>"://HDFEOS/GRIDS/NpPolarGrid12km/lat<br>
          </div>
          <div><br>
          </div>
          <div>And things work when lining those up in VRT with warp.
            Can the HDF5 driver be made to auto-detect these geolocation
            arrays? </div>
          <div><br>
          </div>
          <div>I see that the NETCDF driver actually does: </div>
          <div><br>
          </div>
          <div>gdalinfo "NetCDF:/vsicurl/<a href="https://n5eil01u.ecs.nsidc.org/AMSA/AU_SI12.001/2012.07.02/AMSR_U2_L3_SeaIce12km_B04_20120702.he5" target="_blank">https://n5eil01u.ecs.nsidc.org/AMSA/AU_SI12.001/2012.07.02/AMSR_U2_L3_SeaIce12km_B04_20120702.he5</a>"
            -sd 26<br>
          </div>
          <div><br>
          </div>
          <div>I'm asking as an email rather than pursuing the fix
            because, these data are actually a regular grid on the north
            and south poles, and so geolocation by arrays is
            sub-optimal  the specification is listed in </div>
          <div><br>
          </div>
          <div><a href="https://nsidc.org/sites/default/files/au_si12-v001-userguide_1.pdf" target="_blank">https://nsidc.org/sites/default/files/au_si12-v001-userguide_1.pdf</a><br>
          </div>
          <div><br>
          </div>
          <div>and the two parameter sets are</div>
          <div><br>
          </div>
          <div>Np-north: -te -3850000,  -5350000, 3750000, 5850000
            -t_srs EPSG:3411</div>
          <div>Sp-south: -te -3950000,  -3950000, 3950000, 4350000
            -t_srs EPSG:3412</div>
          <div><br>
          </div>
          <div>Is this generally something we should pursue within GDAL?
            It seems like an endless task to detect-on-open exactly this
            situation and assign the easy fix, but this is a pretty
            fundamental data stream and it's very common so the
            longlat/arrays might be numerically detectable with other
            heuristics hinting that it's polar (??) and there are plenty
            of other sources that present equivalents in the right way
            e.g. this one: </div>
          <div><br>
          </div>
          <div>"/vsicurl/<a href="https://noaadata.apps.nsidc.org/NOAA/G02135/north/daily/geotiff/2012/07_Jul/N_20120702_concentration_v3.0.tif" target="_blank">https://noaadata.apps.nsidc.org/NOAA/G02135/north/daily/geotiff/2012/07_Jul/N_20120702_concentration_v3.0.tif</a>"<br>
          </div>
          <div><br>
          </div>
          <div>The right approach is probably to inform the providers
            and get the right metadata baked in ... but there's pros and
            cons to either. I'm not sure there's even enough information
            in these files to clearly detect the situation, it would be
            a bit like the NSIDCbin driver with its very strict
            requirements. </div>
          <div><br>
          </div>
          <div>Cheers, Mike</div>
          <div><br>
          </div>
          <div><br>
          </div>
          <span class="gmail_signature_prefix">--</span></div>
        <div><span class="gmail_signature_prefix"></span><br>
          <div dir="ltr" class="gmail_signature">Michael Sumner<br>
            Software and Database Engineer<br>
            Australian Antarctic Division<br>
            Hobart, Australia<br>
            e-mail: <a href="mailto:mdsumner@gmail.com" target="_blank">mdsumner@gmail.com</a></div>
        </div>
      </div>
      <br>
      <fieldset></fieldset>
      <pre>_______________________________________________
gdal-dev mailing list
<a href="mailto:gdal-dev@lists.osgeo.org" target="_blank">gdal-dev@lists.osgeo.org</a>
<a href="https://lists.osgeo.org/mailman/listinfo/gdal-dev" target="_blank">https://lists.osgeo.org/mailman/listinfo/gdal-dev</a>
</pre>
    </blockquote>
    <pre cols="72">-- 
<a href="http://www.spatialys.com" target="_blank">http://www.spatialys.com</a>
My software is free, but my time generally not.</pre>
  </div>

</blockquote></div><br clear="all"><div><br></div><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature">Michael Sumner<br>Software and Database Engineer<br>Australian Antarctic Division<br>Hobart, Australia<br>e-mail: <a href="mailto:mdsumner@gmail.com" target="_blank">mdsumner@gmail.com</a></div>