<div dir="ltr">Oh wow, that's fantastic - thanks so much. Lots for me to learn from there too, with the
HDF5EOS GRID details. <div><br></div><div>Cheers, Mike</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Apr 30, 2024 at 6:08 AM Even Rouault <<a href="mailto:even.rouault@spatialys.com">even.rouault@spatialys.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><u></u>
<div>
<p>Michael,</p>
<p>actually support for those products was already there at 95%
since they use the HDF5EOS GRID formalism which we support since
3.7. But that particular type of products has an oddity. They have
an empty <u></u>GROUP=Dimension<u></u>
within the GROUP=GridStructure in the HDF5EOS metadata, which our
parser didn't like as we use the information in this group to map
dimension names to dimension index and size. Fixed/worked around
in <a href="https://github.com/OSGeo/gdal/pull/9807" target="_blank">https://github.com/OSGeo/gdal/pull/9807</a> . Hard to tell if those
products are in fault given that the HDF5EOS specification at
<a href="https://www.earthdata.nasa.gov/s3fs-public/imported/ESDS-RFC-008-v1.1.pdf" target="_blank">https://www.earthdata.nasa.gov/s3fs-public/imported/ESDS-RFC-008-v1.1.pdf</a>
, appendix A, is extremely minimum regarding the HDF5EOS metadata
itself (for once, I complain for a spec to be too minimal !), and
the example they give for grids doesn't make sense to me (the
declared dimension names and sizes have nothing to do with the
ones use in the grid)</p>
<p>I now get:</p>
<p>$ gdalinfo
HDF5:"AMSR_U2_L3_SeaIce12km_B04_20120702.he5"://HDFEOS/GRIDS/SpPolarGrid12km/Data_Fields/SI_12km_SH_18H_ASC<br>
Driver: HDF5Image/HDF5 Dataset<br>
Files: AMSR_U2_L3_SeaIce12km_B04_20120702.he5<br>
Size is 632, 664<br>
Coordinate System is:<br>
PROJCRS["unnamed",<br>
BASEGEOGCRS["Unknown datum based upon the custom spheroid",<br>
DATUM["Not specified (based on custom spheroid)",<br>
ELLIPSOID["Custom spheroid",6378273,0,<br>
LENGTHUNIT["metre",1,<br>
ID["EPSG",9001]]]],<br>
PRIMEM["Greenwich",0,<br>
ANGLEUNIT["degree",0.0174532925199433,<br>
ID["EPSG",9122]]]],<br>
CONVERSION["Polar Stereographic (variant B)",<br>
METHOD["Polar Stereographic (variant B)",<br>
ID["EPSG",9829]],<br>
PARAMETER["Latitude of standard parallel",-70,<br>
ANGLEUNIT["degree",0.0174532925199433],<br>
ID["EPSG",8832]],<br>
PARAMETER["Longitude of origin",0,<br>
ANGLEUNIT["degree",0.0174532925199433],<br>
ID["EPSG",8833]],<br>
PARAMETER["False easting",0,<br>
LENGTHUNIT["metre",1],<br>
ID["EPSG",8806]],<br>
PARAMETER["False northing",0,<br>
LENGTHUNIT["metre",1],<br>
ID["EPSG",8807]]],<br>
CS[Cartesian,2],<br>
AXIS["(E)",north,<br>
MERIDIAN[90,<br>
ANGLEUNIT["degree",0.0174532925199433,<br>
ID["EPSG",9122]]],<br>
ORDER[1],<br>
LENGTHUNIT["Meter",1]],<br>
AXIS["(N)",north,<br>
MERIDIAN[0,<br>
ANGLEUNIT["degree",0.0174532925199433,<br>
ID["EPSG",9122]]],<br>
ORDER[2],<br>
LENGTHUNIT["Meter",1]]]<br>
Data axis to CRS axis mapping: 1,2<br>
Origin = (-3950000.000000000000000,4350000.000000000000000)<br>
Pixel Size = (12500.000000000000000,-12500.000000000000000)<br>
Metadata:<br>
Conventions=CF-1.6<br>
HDFEOS_INFORMATION_HDFEOSVersion=HDFEOS_5.1.15<br>
history=This version of the Sea Ice processing code contains
updates provided by the science team on September 16, 2019. For
details on these updates, see the release notes provided in the
DAP.<br>
institution=NASA's AMSR Science Investigator-led Processing
System (SIPS)<br>
references=Please cite these data as: Markus, T., J. C. Comiso,
and W. N. Meier. 2018. AMSR-E/AMSR2 Unified L3 Daily 12.5 km
Brightness Temperatures, Sea Ice Concentration, Motion & Snow
Depth Polar Grids, Version 1. [Indicate subset used]. Boulder,
Colorado USA. NASA National Snow and Ice Data Center Distributed
Active Archive Center. doi: <a href="https://doi.org/10.5067/RA1MIJOYPK3P" target="_blank">https://doi.org/10.5067/RA1MIJOYPK3P</a>.<br>
source=satellite observation<br>
title=AMSR-E/AMSR2 Unified L3 Daily 12.5 km Brightness
Temperatures, Sea Ice Concentration, Motion & Snow Depth Polar
Grids<br>
Corner Coordinates:<br>
Upper Left (-3950000.000, 4350000.000) ( 42d14'27.21"W,
39d11'27.54"S)<br>
Lower Left (-3950000.000,-3950000.000) (135d 0' 0.00"W,
41d23'59.41"S)<br>
Upper Right ( 3950000.000, 4350000.000) ( 42d14'27.21"E,
39d11'27.54"S)<br>
Lower Right ( 3950000.000,-3950000.000) (135d 0' 0.00"E,
41d23'59.41"S)<br>
Center ( 0.000, 200000.000) ( 0d 0' 0.01"E, 88d
8'51.76"S)<br>
Band 1 Block=632x1 Type=Int32, ColorInterp=Undefined<br>
NoData Value=0<br>
Metadata:<br>
add_offset=0<br>
coordinates=lon lat<br>
long_name=18.7 GHz horizontal daily average ascending Tbs<br>
packing_convention=netCDF<br>
packing_convention_description=unpacked = scale_factor x
packed + add_offset<br>
scale_factor=0.1<br>
standard_name=brightness_temperature<br>
units=degree_kelvin<br>
_FillValue=0<br>
<br>
</p>
<p>Even<br>
</p>
<div>Le 29/04/2024 à 21:10, Michael Sumner
via gdal-dev a écrit :<br>
</div>
<blockquote type="cite">
<div dir="ltr">This HDF5 (requires earthdata credentials your
"Authorization: Bearer <token>" in GDAL_HTTP_HEADERS, or
equiv) presents without geolocation arrays.
<div><br>
</div>
<div>gdalinfo "/vsicurl/<a href="https://n5eil01u.ecs.nsidc.org/AMSA/AU_SI12.001/2012.07.02/AMSR_U2_L3_SeaIce12km_B04_20120702.he5" target="_blank">https://n5eil01u.ecs.nsidc.org/AMSA/AU_SI12.001/2012.07.02/AMSR_U2_L3_SeaIce12km_B04_20120702.he5</a>"
-sd 26<br>
Driver: HDF5Image/HDF5 Dataset<br>
Files: /vsicurl/<a href="https://n5eil01u.ecs.nsidc.org/AMSA/AU_SI12.001/2012.07.02/AMSR_U2_L3_SeaIce12km_B04_20120702.he5" target="_blank">https://n5eil01u.ecs.nsidc.org/AMSA/AU_SI12.001/2012.07.02/AMSR_U2_L3_SeaIce12km_B04_20120702.he5</a><br>
Size is 608, 896<br>
Metadata:<br>
Conventions=CF-1.6<br>
HDFEOS_INFORMATION_HDFEOSVersion=HDFEOS_5.1.15<br>
history=This version of the Sea Ice processing code contains
updates provided by the science team on September 16, 2019.
For details on these updates, see the release notes provided
in the DAP.<br>
institution=NASA's AMSR Science Investigator-led Processing
System (SIPS)<br>
references=Please cite these data as: Markus, T., J. C.
Comiso, and W. N. Meier. 2018. AMSR-E/AMSR2 Unified L3 Daily
12.5 km Brightness Temperatures, Sea Ice Concentration, Motion
& Snow Depth Polar Grids, Version 1. [Indicate subset
used]. Boulder, Colorado USA. NASA National Snow and Ice Data
Center Distributed Active Archive Center. doi: <a href="https://doi.org/10.5067/RA1MIJOYPK3P" target="_blank">https://doi.org/10.5067/RA1MIJOYPK3P</a>.<br>
source=satellite observation<br>
title=AMSR-E/AMSR2 Unified L3 Daily 12.5 km Brightness
Temperatures, Sea Ice Concentration, Motion & Snow Depth
Polar Grids<br>
Corner Coordinates:<br>
Upper Left ( 0.0, 0.0)<br>
Lower Left ( 0.0, 896.0)<br>
Upper Right ( 608.0, 0.0)<br>
Lower Right ( 608.0, 896.0)<br>
Center ( 304.0, 448.0)<br>
Band 1 Block=608x1 Type=Int32, ColorInterp=Undefined<br>
Metadata:<br>
comment=data value meaning: 0 -- Open Water, 110 --
missing/not calculated, 120 -- Land<br>
coordinates=lon lat<br>
long_name=Sea ice concentration daily average<br>
units=percent<br>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div>gdalinfo --version<br>
GDAL 3.9.0dev-cb4d30f56d, released 2024/04/15 <br>
</div>
<div><br>
</div>
<div>The geolocation arrays are sds 33 and 32 respectively: </div>
<div><br>
</div>
<div>HDF5:"/vsicurl/<a href="https://n5eil01u.ecs.nsidc.org/AMSA/AU_SI12.001/2012.07.02/AMSR_U2_L3_SeaIce12km_B04_20120702.he5" target="_blank">https://n5eil01u.ecs.nsidc.org/AMSA/AU_SI12.001/2012.07.02/AMSR_U2_L3_SeaIce12km_B04_20120702.he5</a>"://HDFEOS/GRIDS/NpPolarGrid12km/lon<br>
</div>
<div><br>
</div>
<div>HDF5:"/vsicurl/<a href="https://n5eil01u.ecs.nsidc.org/AMSA/AU_SI12.001/2012.07.02/AMSR_U2_L3_SeaIce12km_B04_20120702.he5" target="_blank">https://n5eil01u.ecs.nsidc.org/AMSA/AU_SI12.001/2012.07.02/AMSR_U2_L3_SeaIce12km_B04_20120702.he5</a>"://HDFEOS/GRIDS/NpPolarGrid12km/lat<br>
</div>
<div><br>
</div>
<div>And things work when lining those up in VRT with warp.
Can the HDF5 driver be made to auto-detect these geolocation
arrays? </div>
<div><br>
</div>
<div>I see that the NETCDF driver actually does: </div>
<div><br>
</div>
<div>gdalinfo "NetCDF:/vsicurl/<a href="https://n5eil01u.ecs.nsidc.org/AMSA/AU_SI12.001/2012.07.02/AMSR_U2_L3_SeaIce12km_B04_20120702.he5" target="_blank">https://n5eil01u.ecs.nsidc.org/AMSA/AU_SI12.001/2012.07.02/AMSR_U2_L3_SeaIce12km_B04_20120702.he5</a>"
-sd 26<br>
</div>
<div><br>
</div>
<div>I'm asking as an email rather than pursuing the fix
because, these data are actually a regular grid on the north
and south poles, and so geolocation by arrays is
sub-optimal the specification is listed in </div>
<div><br>
</div>
<div><a href="https://nsidc.org/sites/default/files/au_si12-v001-userguide_1.pdf" target="_blank">https://nsidc.org/sites/default/files/au_si12-v001-userguide_1.pdf</a><br>
</div>
<div><br>
</div>
<div>and the two parameter sets are</div>
<div><br>
</div>
<div>Np-north: -te -3850000, -5350000, 3750000, 5850000
-t_srs EPSG:3411</div>
<div>Sp-south: -te -3950000, -3950000, 3950000, 4350000
-t_srs EPSG:3412</div>
<div><br>
</div>
<div>Is this generally something we should pursue within GDAL?
It seems like an endless task to detect-on-open exactly this
situation and assign the easy fix, but this is a pretty
fundamental data stream and it's very common so the
longlat/arrays might be numerically detectable with other
heuristics hinting that it's polar (??) and there are plenty
of other sources that present equivalents in the right way
e.g. this one: </div>
<div><br>
</div>
<div>"/vsicurl/<a href="https://noaadata.apps.nsidc.org/NOAA/G02135/north/daily/geotiff/2012/07_Jul/N_20120702_concentration_v3.0.tif" target="_blank">https://noaadata.apps.nsidc.org/NOAA/G02135/north/daily/geotiff/2012/07_Jul/N_20120702_concentration_v3.0.tif</a>"<br>
</div>
<div><br>
</div>
<div>The right approach is probably to inform the providers
and get the right metadata baked in ... but there's pros and
cons to either. I'm not sure there's even enough information
in these files to clearly detect the situation, it would be
a bit like the NSIDCbin driver with its very strict
requirements. </div>
<div><br>
</div>
<div>Cheers, Mike</div>
<div><br>
</div>
<div><br>
</div>
<span class="gmail_signature_prefix">--</span></div>
<div><span class="gmail_signature_prefix"></span><br>
<div dir="ltr" class="gmail_signature">Michael Sumner<br>
Software and Database Engineer<br>
Australian Antarctic Division<br>
Hobart, Australia<br>
e-mail: <a href="mailto:mdsumner@gmail.com" target="_blank">mdsumner@gmail.com</a></div>
</div>
</div>
<br>
<fieldset></fieldset>
<pre>_______________________________________________
gdal-dev mailing list
<a href="mailto:gdal-dev@lists.osgeo.org" target="_blank">gdal-dev@lists.osgeo.org</a>
<a href="https://lists.osgeo.org/mailman/listinfo/gdal-dev" target="_blank">https://lists.osgeo.org/mailman/listinfo/gdal-dev</a>
</pre>
</blockquote>
<pre cols="72">--
<a href="http://www.spatialys.com" target="_blank">http://www.spatialys.com</a>
My software is free, but my time generally not.</pre>
</div>
</blockquote></div><br clear="all"><div><br></div><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature">Michael Sumner<br>Software and Database Engineer<br>Australian Antarctic Division<br>Hobart, Australia<br>e-mail: <a href="mailto:mdsumner@gmail.com" target="_blank">mdsumner@gmail.com</a></div>